A Robust Spectral Three-Term Conjugate Gradient Algorithm for Solving Unconstrained Minimization Problems

In this paper, we investigated a new improved Conjugate Gradient (CG) algorithm of a Three-Term type (TTCG) based on Dai and Liao procedure to improve the CG algorithm of (Hamoda, Rivaie, and Mamat / HRM). The new CG-algorithm satisfies both the conjugacy condition and the sufficient descent condition. The step-size of this TTCG-algorithm would be computed by accelerating the Wolfe-Powell line search technique. The proposed new TTCG algorithms have demonstrated their global affinity in certain specific circumstances given in this paper.


. Introduction
To start by giving any issue of unconstrained optimization we need to know this issue as follows: min {f(x): x ∈ R n } (1) Let us define the function as f from R n to R is continuously differentiable function, and its gradient is denoted by g(x) = ∇f(x), these CG methods are known to be designed to solve the problem of type (1), specifically when the n dimension is very large due to the simplicity of repetition of the search, memory requirements are very low. The iterative formula for standard TTCG methods is given by: x k+1 = x k + α k d k , k = 0,1,2 … (2) The α k amount of the step is always evident as a positive scalar and d k is the search direction specified by: Where β k is scalar. The differences highlight CG methods in their speed and performance by specifying the numerical parameter . This is a set of the famous versions of as contained in their sources: ( +1 − ) CG methods converge with the FR format worldwide but have poor numerical performance because of the behavior of the jamming along the repetition process. The PR and HS methods have good numerical performance but are not always close [11]. To clarify all of the = +1 − and = +1 − ; ‖. ‖ indicates the Euclidean norm. In the order given to those parameters, we will list their sources severally, HS (Hestenes Stiefel [17]), FR (Fletcher Reeves [25]), PR (Polak Ribière [2,4]), CD (Conjugate Descent [26]), LS (Liu Storey [28]) and DY (Dai Yuan [30]) CG ways. Any methodology of gradient has to update the point by the line search used. The Wolfe-Powell (WWP) customary search terms square measure usually employed in CG ways. The search terms within the Weak Wolfe-Powell (WWP) line square measure as follows: ( +1 ) − ( ) ≤ (4) +1 ≥ (5) By this condition, is a descent search direction such that 0< < < 1. The Strong Wolfe-Powell (SWP) conditions defined in (4) and satisfies: | +1 | ≤ (6) As a generalization of Strong Wolfe conditions, the absolute value opens in (6) with two disparities of inequality so that: − ≤ +1 ≤ Moreover, from all of these previous conditions we obtain the characteristic of sufficient descent, namely: In the last year, Wei and et. al. [32] introduced an odd plan of its predecessors from the formulas of the PR-CG methodology referred to as the WYL-CG methodology. Zhang recommended through his study a lot of and has improved WYL-CG normal methodology victimization the new CG, additionally it well-tried that the strategy of NPR-CG met the traditional demand ratios beneath conditions of line search SWP [15]. Finally, Zhang et al. planned another changed CG-method referred to as the MPR-CG methodology, wherever Dai and Wen [31] planned another changed version of NPR-CG methodology referred to as the DPR-CG methodology.

Researchers
Formulas The researchers (Hamoda, Rivaie and Mamat) in [15] have developed another new CG-type formula for the formulas mentioned in the above table by changing the compound based on the convex structure method to obtain better results compared to the previous formulas by suggesting the following formula: Then from the results that obtained by [15], the quantity of the parameter u=0.9 i.e. ∈ (0,1) and from given in equation (8), the search direction of this formula was given in the following gradient:

Spectral CG-Methods.
As a necessary and important type of gradient methods to solve the problem (1) is the method of spectral gradient (SCG), which was originally developed by Barzilai and Borwein [9]. In 1988, Raydan [19] added SCG method to the problems of unrestricted improvement on a large scale. The idea of this method depends mainly on the use of gradient trends only in each line search in order to ensure the regression strategy by multiplying the first limit by a parameter derived from one of the known derivation methods or by the parameter as in the first idea of this method, Good global. So this method is superior to the CG method developed in many problems but when you devise the exact line of search with both methods you will revert to the traditional method. The first to give the first idea of this method was Birgin and Martinez [6] proposed three types of SCG methods. So that the direction k d of this method is defined as follows: Variable parameter k  formulas vary in these ways: The gradient formula varies according to the parameter k  calculated by the following: (12) Note in the numerical results that these methods give good effectiveness, but they may not be able to guarantee the generation of k d descent directions if the use of inexact-line searches as when using the search Armijo or Wolfe as a type of inexact-line searches, in the case of general functions is not guaranteed so each algorithm is proven to be effective in these properties theoretically and practically through numerical results. To ensure descent property, Dixon [14] and Al-Baali [18] suggested to use the steepest Recently noted, there are many modified CG methods studied. Liu et al. [10] make an adjustment to the CD-method so that the direction that is always generated is the descent direction and k d is determined by the following: They demonstrate that this method can ensure that descent directions are generated and are globally convergent. This paper is divided as follows: In Section 2, we evaluate the new form of the CG-three-term method using the spectral gradient method with derivation of , = 1,2,3. In Section 3, we give some proof to prove sufficient properties of global proportions and convergence using standard derivation of Section 2. Finally, in Section 4, the task of this part is the good numerical results of the new algorithm. The new CG-algorithms under standard conditions gave new search directions which have better performance than the two-dimensional algorithms for solving problems of large scale dimensions, i.e. n = 1000, .., 10000, for total of thirty-eight standard non-linear test functions.

New CG-Algorithm (Modified HRM).
Through what researchers (Zhang, Zhou and Li [12,13]) have prompt in recent years, three CG strategies prompt a way that forever satisfies the case of sufficient descent condition (7), freelance of the techniques of line search. Here, during this work, we've superimposed another extra term for the search direction (9) to become a scaled three-term search direction to enhance the number of iterations calculated and therefore the number of conniving the objective functions and their gradients. Will be rewritten by a three-term form: The parameter k  can be calculated, normally, from (12) and the new proposed search direction defined in (16), gives more efficiency and more stable CG-algorithm to reach the optimal minimum point compared with the original two-term search direction defined in (9). Also, the new proposed parameter used in (16) has good theoretical background compared to the theories derived from the both PR-CG and HRM -CG methods.

Derivation of The New Parameters
. To derive a formula θ k SBi , depending on the classic conjugacy property given in the following equation: y k T d k+1 = 0 (17) The first conditional condition, in this approach, gives a method of approximation of the search direction by deleting one term through derivation and as a generalization of this formula by placing the inexact line search (ILS) condition of the formula. We obtain the conditional condition given by Perry [1] as in the following formula: From (16), we use the direction Multiplying (19) : (20) Using (17) and substituting from the formula: = (21) Then we get: Hence, B. Derivation of The New Parameter .
From (18), (21) and (16) we use the direction: C. Derivation of The New Parameter . Now, Use the conjugacy condition defined in [29] to derive 3 +1 = − +1 Rewrite the search direction as: From (21) and multiplying (25) by to get: Now the new proposed three new CG-formulas have been derived. Below we can write the outlines of the proposed TTCG-algorithm in general using the most comprehensive formula, namely, the third version, defined in (26). To compare its numerical results, we compute the percentage performance of our new TTCGalgorithm against the numerical results of (PR and HRM) CG-algorithms.
Step 2: Evaluate step size α k by Wolfe line search technique from (4) and (5).

An Accelerated Scheme of Wolfe Line Search Parameter.
In the first a part of this paper, we tend to mention the concept of standard CG strategies, showing us that analysis trends might not be gradated. To boost the performance of those strategies, search trends supported second order data square measure based mostly. Jorge Nocedal [3] gave some observations indicating that the lengths of steps within the CG strategies of one may be larger or smaller than one looking on however the matter is measured. Calculations show that the latter is a lot of victorious [7].
Here, we tend to talk to the acceleration theme in Andrei's [22]; primarily, this step length is changed during a multiplying way to improve the reduction of function values on the frequencies [20,24].
Step 3: Evaluate new point by z k =x k + α k d k , g z = ∇f(z) and y k = g k − g z Step 4: Compute a k ̅̅̅ = α k g k T d k and b k ̅̅̅ = −α k y k T d k Step 5: Acceleration scheme. If b k ̅̅̅ > 0 and the evaluate ε k = − a k ̅̅̅̅ b k ̅̅̅̅ and update the points as x k+1 = x k + ε k α k d k , else update the points as x k+1 = x k + α k d k Step 6: Evaluate θ k SBi where i=1,2,3 , such that (SB1) is defined in (16); (SB2) is defined in (18)   Step 9: Use Powell restarting criterion [16]; If |g k+1 T g k | > 0.2‖g k+1 ‖ 2 , then set d k+1 = −g k+1 Step 10: Update the iteration k=k+1 go to Step 2.

Full Study of the Sufficient Descent and Global Convergence Properties.
Let us consider the foremost vital assumptions accustomed demonstrate the potency of algorithms employed in previous studies of the related to gradient ways, see [20] for example:

Assumption A.
Let f(x) is delimited from below on the extent set = { ∈ , ( ) ≤ ( 0 )} , where 0 is that the place to begin.
(27) Now we derive the sufficient condition of three term from , such that i=1,2,3.

Since
> 0 and small and 0 2  k s this implies that we can get the sufficient condition of three term from 0 < 2 < 1. Then +1 +1 ≤ − 2 ‖ +1 ‖ 2 (31) Looking at revealed analysis papers during this field, has to mention some vital lemma and a crucial theorem for any CG methodology uses robust Wolfe line search. The subsequent general results hold.

Suppose that assumptions A and B hold. Consider any CG-type method in the form of
Then added the above result to get the following : Therefore, when 0 < ∅ k < 1, so we are able to prove that the new TTCG algorithm is global convergence, we are able to additionally say 0 = k g for a few k or (34) holds. It will be discovered, it's from the info of the new Theorem1 theory, the direction 1 + k d satisfied the sufficient descent condition severally of the line search.

Theorem [8].
Assume that Assumptions A and B hold. Contemplate the strategy (2) . After all this proof, the projected new TTCG formula with three versions features a convergent global characteristic by satisfying the wants of the Zoutendijk theorem and therefore the line search satisfy the robust Wolfe-Powell condition then from Gilbert and Nocedal in [8] these methodology has the global convergence property.

Numerical Experiments.
In this section, we glance at the tables and graphs of some computations to check the relative performance of the mathematical implementation of packaging, human resource management and projected new standards (TTCG Algorithm) (NEW) to a collection of thirty eight any old nonlinear issues. Its details will be found in [21,24,27]. We have chosen completely different test functions for each free and tiny scale optimization problem. Every one tested with completely different variables between one and 10,000. We have a tendency to test these issues mistreatment the Wolfe Powell technique.
These CG-Algorithm were enforced by mistreatment Visual FORTRAN version 2002, we've applied the Wolfe-Powell line search technique. All of the numerical experiments were run on identical laptop with associate Intel(R), Core TM, i7-3612QM (2.10GHZ) CPU, 6GB of RAM , associated mistreatment Windows seven as an operating system.
In order to assess the responsible-ness of the new projected scaled three-term CG-Algorithm, we tend to test this algorithmic rule against the well-known classical and changed ways of the PR and HRM CG-Algorithms mistreatment constant issues, and assumed that the most effective algorithmic rule ought to need fewer range of iterations, NOI; number of function and gradient evaluations NOFG; less time needed to induce the answer purpose processor and also the total work outlined by, TOTAL WORK = NOI + NOFG + TIME. All of those CG-Algorithms terminated once: ‖g k ‖ < 10 −6 . Numerical results were comparatively compared with the processor time, NOFG and number of iterations (NOI). The performance results square measure shown in Figures (1) and (2) severally, employing a performance profile introduced by Dolan and More [5].

20-DIXMAANC
In the beginning we used the three new versions of the new proposed scaled TTCG-algorithms in our implementation to obtain numerical results but later we noticed that the last version is the best one among these three version formulas and therefore within the numerical result Tables, we will list the CG-Algorithm of the third version only which is more important to highlight the numerical results and efficiency of the three CG-Algorithms. In Tables ( 2 & 3 ), the efficiency of the (NEW) TTCG-Algorithm was compared. The new algorithm gave the best numerical results according to our used 38-test problems. If we tend to calculate the number of the best take a look at functions once PR is compared with NEW in step with NOI, NOFG and C.P.U. in Table(4), we tend to see that the NEW rule is actually higher than PR, and if we tend to calculate the amount of best take a look at functions once HRM is compared with NEW in NOI, NOFG and C.P.U. in Table( 5) , show that the NEW rule is that the higher than HRM, that the NEW rule is that the most strong rule in step with the number of iteration , NOFG and TIME underneath the accelerated Wolfe-Powell line search: Figure(1) shows the performance file for all measures in CG-Algorithms with the required NOI. Figure(2) shows the performance file for all measures in CG-Algorithms with the required NOFG.
Accurate examination of all shapes indicates that the bottom curve represents the PR CG-Algorithm. Therefore, this CG-Algorithm has less performance. Also, HRM incorporates a slightly higher performance than PR. Finally, the exact check shows that the, NEW technique has the best performance by exploitation each tools; NOI and NOFG. .

Conclusions.
In this paper, we've investigated a replacement scaled TTCG-Algorithm within three completely different new versions for resolving a collection of unconstrained nonlinear minimization issues each on paper and by experimentation. The last version of this TTCG-Algorithm, namely SB3, is that the sturdy one. This is often compared numerically against the 2 accepted CG-Algorithms (PR and HRM). The numerical and theoretical studies during this paper show that the state of decent condition will be derived and converged globally if the Wolfe-Powell line search was used. Numerical results for 38 check issues, in general, and for this hand-picked kind of numerical examples, show that the NEW-Algorithm is the sturdy and effective one.