Investigation on Scaled CG-Type Algorithms for Unconstrained Optimization

In this paper, we describe two new algorithms which are modifications of the Hestens-stiefl CG-method. The first is the scaled CGmethod (obtained from function and gradient-values) which improves the search direction by multiplying to a scalar obtained from function value and its gradient at two successive points along the iterations. The second is the Preconditioned CG-method which uses an approximation at Hessein of the minimizing function. These algorithms are not sensitive to the line searches. Numerical experiments indicate that these new algorithms are effective and superior especially for increasing dimensionalities.


Introduction:
Unconstrained Optimization Problems expressed as: is twice differentiable real valued function, is one of the most active areas in optimization community. virtually appearing in every human activity.
For solving these problems many efficient methods have been suggested. Excellent Presentations of these methods can be found, for example (Fletcher, 1987;Gill et-al 1981;Boyd and Vavdenberhe, 2003;Nocedal, 1992;Edwin 2001). The most useful algorithms classify in: The Conjugate gradient and its variants, Newton method and its extensions; the DFP variable metric method, many different QN methods. All these methods are iterative and consider iterations of the form:  is a step length obtained by line search.Conjugate Gradient methods consider the search directions as: (4) where the scalar k  is chosen in such a manner that the method reduces to the linear Conjugate Gradient when the function is quadratic and line search is exact. The rest of the methods define the search directions by: ……………….…………………..(5) where k G is a non-singular symmetric matrix. Mainly the matrix k G is selected as I k G = (identity matrix which gives the steepest descent method), (the Newton method) or an approximation of the Hessian Different modifications are made to the CG-algorithm in different ways (see for example Hu and Story, 1990;Fletcher, 1993;Al-Baali 1985), most of these modifications are made to the search directions to improve then.
We end this general introduction by content of this paper which is organized as follows: In section(2) we review the Conjugate gradient, QNmethods and their Combinations, section(3) contains the development of the new algorithms, the last section includes the numerical results.

Review of the methods: 2-1 Conjugate Gradient Methods:
Non-linear Conjugate gradient (CG) is one of the most useful and the earlist techniques for solving Large-scale non-linear optimization problems.
Many variants of this original scheme have been proposed, and some are widely used in practice, CG-methods only use the first order derivatives information of the objective function and need not update the Hessian matrix at each iteration. First, these are used to solve the general unconstrained optimization problems by Fletcher and Reeves (1964).
Conjugate gradient methods depend on the fact that for quadratic, if we search along a set of n mutually conjugate directions Assuming equation (6) and considering : Finally, in many implementations of Conjugate gradient methods, The iteration (7) is restarted every n or (n+1) steps setting k  equal to zero i.e. taking steepest descent step. This ensures global convergence (Nocedal, 1992). However, different restarts are introduced (see Fletcher, 1987). One of the well-known restarts given by Powell (Powell 1977) is: This criterion will be used later in our suggested algorithms.

2-2 Quasi-Newton Methods:
Quasi-Newton methods are probably the most popular general purpose algorithms for unconstrained optimization problems. Many QNmethods are advantageous due to their fast convergence and absence of second order derivatives computation.
For the QN-methods assume that at the kth iteration at approximation point  (14) is called Rank one Correction formula, where and k y as in equation (9) k k (15) is the DFP formula

2-3 Preconditioned CG algorithm (PCG):
The Preconditioned CG methods (PCG) first appeared in paper by Axelsson (Axelsson, 1972). It was developed with object of accelerating the convergence of the CG-method by a transformation of variables while keeping the basic properties of the method. Such transformation was introduced by Allwright (Allwright, 1972), the symmetric positive definite matrix H can be factored in various ways for example as where L is lower triangular and non-singular (for more detail see Allwright, 1972).
Buckley (Buckley, 1978. a and b) introduced an algorithm in which conjugate gradient and quasi-Newton search directions occur together and which can be interpreted as a conjugate gradient algorithm with changing metric. Many authors have extended this type of algorithms (see for example Al-Bayati, 1996).
The search direction to the preconditioned (PCG) method is defined by: H is one of the forms given in equations (7) or (8) or (9). In this paper, our focus is to the PDF H .

Development of two suggested Algorithms 3-1 Scaled CG-method New1 (say)
One of the reasons for inefficiency of conjugate gradient algoriths is that non of the k  takes into consideration the effect of inexact line searches (Hu and Story, 1990). In order to do this and find an optimal  , Liu and Story (1991) introduced an algorithm that finds an optimal (    The outlines of the Self-Scaling CG-method (new1): Step (1): Set initial point 1 x and scalar  .

3-2 Self-Scaling PCG method (New2):
In this section, a new PCG method for solving unconstrained optimization problems is proposed .
This new PCG algorithm considered here has an additional property of being invariant under scaling of the function or of its variables where the objective function is twice continuously differentiable and search direction is descent i.e. . usual Euclidian norm, then the new2 algorithm can be organized as follows: The outlines of the Self-Scaling PCG method (new2): Step (1): Set Step (2): Step (3): Set Step(4): If Check if restart equation (12) is satisfied then set 1 k xx = go to step (2). otherwise go to step (5).

Conclusions:
Clearly, self-scaling techniques are very effective in unconstrained optimization algorithms. The two different approaches used in this paper proved to be very effective, especially for high dimension functions.
Clearly, our numerical results indicate that there are improvements of proposed self-scaling techniques over standard DFP and BFGS algorithms.