The Extended CG Method for Non-Quadratic Models

Received on: 25 / 3 / 2007 Accepted on: 4 / 11 / 2007 ABSTRACT This paper investigates an interleaved algorithm which combines between the extended conjugate gradient with the hybrid method of Touati-Storey. This combined algorithm is based on the exact line search to solve a number of non-linear test functions with different dimensions. Experimental results indicate that the modified algorithm is more efficient than the original Sloboda algorithm.


Introduction
The problem to be solved by the Sloboda algorithm is to calculate the least value of a general differentiable function of several variables. Let n be the number of variables, x be the vector of variables and q(x) be the objective function and g(x) be the gradient of q(x). i.e.
Conjugate gradient algorithm does not require any explicit second derivatives and it is an iterative method. The sequences of points x1 , x2 , x3 , … are calculated by the successive iteration procedure and it should converge to the point in the space of the variable at which q(x) is least.
The conjugate gradient algorithm was first applied to the general unconstrained minimization problem by Fletcher & Reeves [7]. However, now there are several versions of the algorithm for this calculation. Let x1 be a given starting point in the space of the variable and let i denote the number of the current iteration starting with i=1. the iteration requires the gradient: gi = g(xi) , if i = 1, let di be the steepest descent direction, di = -gi (2) otherwise, for i > 1, we apply the formula: where βi and yi have the values: The vector norms being Euclidian and by searching for the least value of f(x) from xi along the direction di we can obtain the vector xi+1: Where i  is the value of  that minimizes the function of one variable.
This complete the value of the iteration, and another one is begun if f(xi+1) or gi+1 is not sufficiently small, [10].
Let A denote a symmetric and positive definite ii matrix. For i R x  , we define: Let i F : R R  denote a strictly monotonic increasing function and define: Such a function is called an extended quadratic function, Spedicato [12]. When a minimization algorithm is applied to f, the i th iterate is denoted by xi, the corresponding function value by fi and its gradient by Gi , the function value and gradient value of q are denoted by qi & gi , respectively and the derivative of

Non-Quadratic Sloboda Method
Sloboda [11] proposed a generalized conjugate gradient algorithm for minimizing a strictly convex function of the general form: This algorithm is as follows: ▪ Algorithm 1: Step 1: Set i = 1 ; d1 = -G1 ; Step 2: Step 3: Step 4: Test for convergence, if achieved stop. if not continue.
Step 6: Step 7: Compute the new search direction i i Step 8: Set 1 i + = i ; and go to step 2. Algorithm 1 terminates after i iterations in the case of a nonlinear scaled quadratic function using ELS.
A more general scaling has been considered by Spedicato [12], such a scaling transforms F into a new function where (9), is satisfied. He shows that the sequence of points generated is invariant with respect to nonlinear scaling if: However, this type of scaling also uses functions for which the analytic form is known apriority.
Al-Bayati [1] introduced another family of self-scaling VM-methods given by: where  is again a free parameter; If an estimate of the inverse Hessian in maintained (rather than an estimate of the Hessian itself which is sometime preferred) then there is a strong motivation for However, it is possible to generalize Al-Bayati's family of self-scaling VMupdates (12) to be invariant to a nonlinear scaling by the following algorithm, Al-Bayati [2]. ▪ Algorithm 2: Step 1: Set i = 1; H1 = I ; d1 = -H1 G1 ; Step 2: Compute Step 3: Set Step 4: Test for convergence; if not continue.
Step 6: Compute Step 7: Set 1 i + = i and go to Step 2.
In this section we shall describe also another algorithm which effectively interleaves CG & VM-steps. It is also related to one given originally by Buckley [6], but our implementation differs in that we use the scaled quadratic model instead of the quadratic itself.
CG-algorithm 1. and the generalized VM-algorithm 2. The objective here is to show that, using Al-Bayati's self-scaling VM-update (12), the sequence of the generating points is the same in the generalized CG-algorithm 1.
Before making a few more observations we shall outline briefly the proposed strategy for the interleaved generalized CG-VM method Al-Bayati [2]. Here i is the index of the matrix updated only at restart steps and k is the index of iteration and the algorithm is not converged, until a restart is indicated.
Step 3: If a restart is indicated, namely that the Powell [9] restarting criterion is satisfied, i.e.
Then reset t to the current k , update Hi by: Step 4: Replace i by i + 1 and repeat from (13).

Hybrid Conjugate Gradient Method
Despite the numerical superiority of PR-method over FR-method the later has better theoretical properties than the formal see Al-Baali [3]. Under certain conditions FR-method can be shown to have global convergence with exact line search Powell [10] and also with inexact line search satisfying the strong Wolf-Powell condition. This anomaly leads to speculation on the best way to choose βi .
Touati -Ahmed and Storey in (12) proposed the following hybrid method: Step , otherwise go to Step 3.
Step 3: Here m,  and  user supplied parameters. This hybrid was shown to be globally convergence under both exact and inexact line searches and to be quite competitive with PR-and FR-methods.
Step 5:If PR Step 6: Compute k k Step 7: If a restart is indicated, namely that the Powell, restarting criterion is satisfied, i.e.: k-1 k *T *T k k |g g | 0.2 |g g |  , then reset t to the current k, update Hi by: Step 8: Replace i by i+1 and repeat from (13).

Conclusions and Numerical results:
Several standard test functions were minimized (2 < n < 400) to compare the proposed algorithm with standard Sloboda algorithm which are coded in double precision Fortran 90. The proposed hybrid algorithm needs matrix calculation for 400400, this is the approximately the latest range for this computation of the matrix. The numerical results are obtained on personal Pentium IV Computer. The compete set of results are given in tables 5.1 and 5.2.
The linear search routine used was a cubic interpolation which use function and gradient values and it is adaptation of the routine published by Bunday [5].
We tabulate for all the algorithms; the number of functions evaluations (NOF) and the number of iterations (NOI). Overall totals are also given for NOF and NOI with each algorithm. Table 5.1 gives the comparison between the standard Sloboda algorithm and the proposed algorithm. Table 5.2 indicates that the suggested algorithm is more efficient than the standard Sloboda algorithm. Namely, there are an improvement of about (53 %) in both NOI and NOF according to our selected group of test functions.  All the algorithms terminated when -10 min

Appendix
These test function are from general literature [8].