Modified the CG-Algorithm for Unconstrained Non- Linear Optimization by Using Oren’s Update

Abbas Y. Al-Bayati Abdulghafor M. Al-Rozbayani abdulghafor_rozbayani@uomosul.edu.iq College of Computer Sciences and Mathematics,University of Mosul, Iraq Received on: 20/08/2002 Accepted on: 12/10/2002 ABSTRACT In this paper we have modified a new extended generalized conjugate gradient steps with self-scaling variable metric updates for unconstrained optimization. The new proposed algorithm is based on the inexact line searches and it is examined by using different non-linear test functions with various dimensions.


1.Introduction:
We try to solve the unconstrained minimization problem min f (x), R n , (1) when f is twice continuously differentiable function.This problem is usually solved iteratively.Starting with an initial estimate x1 of the minimum point x * , each subsequent point xk+1, k 1  , will be derived by search along a descent direction dk , such that , 0 where k  is the step-length satisfying line search conditions such that g where 0<c1<c2<1.
We consider that vk=xk+1-xk (5) yk=gk+1-gk (6) Conjugate Gradient (CG) method is one of the few practical methods for solving large dimension problems because it does not require matrix storage and its iteration cost is very low.Now we begin from the initial direction d1 such that The search direction for the next iteration has the following form: The definition of k  in (9a) is due to Fletcher and Reeves [6] and k  in (9b) is due to Polak Ribiere [12].

Generalized CG-Methods:
CG-algorithm usually requires more function evaluations than the variable Metric (VM) method to solve small dimensionality problems.Therefore, many extensions and modifications have been proposed in this field .Liu and Storey [8] introduced a generalized PR algorithm.They studied the effect of the inexact line search on conjugacy in unconstrained optimization, and they showed that their algorithm has global convergence for twice continuously differentiable functions with a bounded level set.

Generalized CG-algorithm (Liu and Storey, 1991)
Step 1:let x1 be an initial point of the minimizer x* of f.
Step 2:set the iteration k=1 and the search direction dk=-gk Step 3:do a line search: set xk+1=xk+αkdk , set k=k+1, α is a line search parameter.
Step 5: if kngo to step 9; otherwise go to step 6.
Step 6:let rk then go to step 8; Otherwise go to step 9.
This algorithm was compared with currently available standard routines and their results demonstrate a general efficient GPR algorithm.
Usually CG-algorithms are implemented with restarts, in order to avoid the effects of an accumulation error.Fletcher [5] in his standard method suggested to restart his algorithm with the steepest descent direction every n or n+1 iteration, where n is a dimension of the problem, another restarting direction was suggested by Powell [13].He developed a new procedure for starting CG-methods.Powell checked that the new search direction dk+1 will be sufficiently downhill if their inequalities

3.Self -Scaling VM methods:
The practical and theoretical merits of the quasi Newton (QN) family of methods for unconstrained optimization have been systematically explored since the classic paper of Fletcher and Powell [7] analyzing Davidon's VM method.
In the family of VM updating, it is useful to multiply each Hk by some scale factor k  0 before using the update formula.With exact line searches, this can be shown to present the conjugacy property in the quadratic case, although we may no longer have Hk+1 -1 G .However, the focus here is to improve the single-step rather than the n-step convergence behavior of the algorithm.Methods that automatically prescript scale factor in a manner such that, if the function is quadratic then the eigenvalues of T k d HkGk+1 tend to be spread above and below are called self-scaling methods [3].
Thus in 1970's the self-scaling VM algorithms were introduced showing significant improvement in efficiency over standard VMmethods.In particular, in a series of papers by Oren [11], Al-Bayati [1], Nocedel [10] and Al-Bayati and Al-Salih [2] algorithms for minimizing an unconstrained nonlinear function f (x) were developed.Now we summarize the scaled BFGS algorithms due to Oren [11].

Oren's algorithm (1974)
Step 1: Start with an initial point x1, Set k=1 and choose H1=I, where I is the identity matrix.
and wk is vector defined by Step 5: Set k=k+1 and go to step 1.

Preconditioned CG-Methods:
Any problem in an unconstrained optimization can be solved by CG or VM methods, each of these methods has particular advantages and disadvantages.In general, a CG-method normally requires more iterations than a QN or VM method to obtain an equally good minimum point but a CG-method requires less storage for implementation per iteration.CGmethods have proved to be valuable where n is large because at each step a few n vectors have to be stored and hence the computational costs and storage requirements are affordable, even for large problems.
The new class of CG-methods has been developed, termed preconditioned conjugate gradient methods (PCG); the idea of the preconditioning has been extended directly to nonlinear problem [9].

New Modified Generalized CG-algorithm:
Step 1:Let x1 be an initial point of the minimizer x* of f and H1=I where I is the identity matrix.

5: If k=n or
rk then go to step 8;otherwise go to step 10. Step8: where wk= tksk-2 k u .

Derivation of the Modified Generalized CG Direction:
Let H be any symmetric positive-definite preconditioned matrix, then by Nazareth [9] with Choleski Factorization H can be factorized as: H=LL T , (12) where L is a real lower triangular matrix and non-singular matrix.Let f be the strictly quadratic function f(x)=x T Gx/2+b T x+c, (13) then the gradient is where z defines a new vector spaces and is defined as: which implies that (17) Eq.( 17) gives a relationship between the gradients in x-space and zspace, i.e. g L g where g is the gradient in z-space while g is the gradient in x-space, so that if h (z*)=0, then g (Lz*)=0.g + are the gradients of h (z) at the point zk,zk+1 ,respectively.By using Eq. ( 18), then Eq. (24) becomes = k y L T gk+1-L T gk=L T yk.
(25) Now consider applying the conjugate gradient method,  18), ( 23) and (25) we get multiply (26) by L and using (12) and LL -1 =I where I is the identity matrix get: Equation ( 27) is our proposed CG Method with Metric update H, where H is any positive-definite symmetric matrix, or the preconditioned CG Method (PCG).Thus the search direction defined in Eq. ( 27) is a new direction for the Generalized Polak-Ribiere CG-direction.

Numerical Results:
In this section we are going to employ the new proposed algorithm with a certain effective CG-method; namely Liu and Storey [8].
We will show that our algorithm is better than the Generalized CGof Liu and Storey's [8] algorithm.Using Oren [11] self-scaling VM update Oren [11] as an acceleration tool to the Generalized CG-algorithm to decrease the number of iterations (NOI) and number of function evaluations (NOF).The comparison tests involve ten well-known test functions with different dimensions.All the results were obtained using double precision on the (Pentium II Computer) using programs written in Fortran Language.The terminate criteria is (510 -5 ) with Powell's restarting criterion


and the algorithms use the cubic line search strategy, with function and gradient values and it is an adaptation from that published by Bunday [4].
The comparative performances of the algorithms are evaluated by considering both the total numbers of iterations (NOI) and total number function evaluations (NOF).Thus all our numerical results are presented in table (1).In this table we have compared our modified proposed algorithm with Liu and Storey [8] algorithm by using (Ten) cases and for dimension 2 1000 n   .
It is clear that the modified algorithm outperformance the standard Liu and Sorey's algorithm as results of this comparison.