On New Conjugate Pair Method

This paper involves the combination between the conjugate pair and hybrid conjugate gradient methods. The new combined algorithm is based on exact line search and it is examined by using different nonlinear test functions in various dimensions. Experimental results indicate that the updated algorithm is more effective than of the two original algorithms.


Introduction:
A very important concept in the theory of the unconstrained optimization of a function f(x), x∈R n is that of conjugate directions.Many algorithms for solving this problem consist basically of the following iteration.Starting with an initial approximation x 0 of the minimum point x * of f(x), 0,1,.... i , ... (1) the parameter λ i is chosen to minimize f(x i + λ i d i ) as a function of λ, and the vector d i can be interpreted as a direction in which we move from x i to x i+1 ,the optimal point in this direction.It follows immediately that the gradient vector at the point x i+1 is orthogonal to d i .

Definition 1:
A set of n nonzero vectors

.(3)
There exists at least one set of n vectors satisfying this definition, via, the eigenvectors of A, then conjugate vectors are also linearly independent.
A general method to produce a set of A-conjugate vectors is the Gram-Shemidt orthogonalization procedure, which we will now describe briefly.Let (v T ) be a set of n linearly independent vectors.Let u 1 =v 1 .Choose u 2 as a linear combination of v 2 and v 1 , that is u 2 =v 2 +λ 21 u 1 ; u 2 will be A-conjugate to u 1 if In general, having selected u 1 ,u 2 ,…,u k , we can choose u k+1 as Thus, the conjugate set {u i } can be developed from the set {v i } by application of the recursion formula

Conjugate Pair Method (CP):
Stewart (1977) introduced a generalization of the notation of conjugancy, leading to a variety of finitely terminating iterations for solving systems of linear equations.An adaptation of Stewart's ideas to minimization problems confirms not only the above-mentioned suspicion, but establishes a method with an even wider scope of generality.
We note that the definition of conjugancy can also be phrased as follows: If the vectors u 0 ,u 1 ,…,u n-1 are the columns of an nxn matrix V, then u 0 ,u 1 ,…,u n-1 are A-conjugate if U T AU is a diagonal (and of course nonsingular ).The generalization is achieved by introducing a second set of vectors v 0 ,…,v n-1.

Definition 2:
let A,U, and V be non singular nxn matrices.Then (U,V) is an Aconjugate pair if L=U T AV is lower triangular.
The generalized algorithm for solving the equations Gx+b=0 g k-1 =Gx k-1 +b Where k=1,2,…,n, and where 3. Formulation of A Generalized-Conjugate Pair (GCP) Method: Stewart (1977) developed an algorithm for constructing an Aconjugate pair (U,V) as follows.Given nonsingular matrices V, A and P, the vector u k is determined as a linear combination of p 0 ,p 1 ,…,p k , k=0,1,…,n-1, such that U and V are A-conjugate.The resulting algorithm is as follows: ] , : : The constant d k are chosen to give u k some predetermined scaling.
We will now formulate the analogous generalized conjugate-direction method for the minimization of a function f(x).Suppose that U and V form a conjugate pair set.
x 0 =arbitrary, 7b) If f is quadratic, the β i in eq.( 7) are the same as the µ i in eq.( 5).In fact if f takes the form:

Theorem (1):
If the iteration (7) is applied to the quadratic function where (U,V) form a generalized conjugate pair, the minimum is found in at most n iterations; moreover.x n lies in the subspase generated by x 0 and v 0 ,v 1 ,…,v n-1 .For the proof see (VanWyk 1977)

Special case of CG-pair method:
By varying the choice of vectors v i and p i in the conjugation algorithm (6), one may obtain from (7) various finitely terminating iterations of minimization.When ( 6) is applied to a general function, the Hessian G can be eliminated from the conjugation algorithm by substituting Putting the scaling constants equal to 1, the choice V=U reduces (6) to the Gram Schmidt procedure and (7) becomes an ordinary conjugate direction algorithm.We will conclude by showing that, in this case, variation of the vectors p i leads to some well-known algorithms.

Fletcher-Reeves Algorithms:
We follow the way of induction to show that, if the columns of P are chosen successively, The directions in (6) reduce to We have We note that …(8) (Bazaraa, 1993)

Fletcher-Powell Algorithm:
Here the direction are defined as: Where H i is initially (i=0) any positive-definite, symmetric matrix; therefore B i being the step length.By using the fact that the gradients g i are eigen vectors with unit eigen value of H i for i <j <n.The directions in the Fletcher-Powell and Fletcher-Reeves algorithms are respectively scalar multiples of each other.

Theorem (2):
If the minimum of f(x), that is x T Gx, in the direction u from the point x 0 is at x i ,i=0,1, then x 1 -x 0 is conjugate to u.

Hybrid Conjugate Gradient Algorithm:
Despite the numerical superiority of Polak-Riebiere (PR) algorithm over Fletcher-Reeves (FR) algorithm, the later has better theoretical properties than the former.Under certain conditions FR-method can be shown to have global convergence with exact line Search (Powell, 1986) and also with inexact line search satisfying the strong Wolfe-Powell condition (see Al-Baali, 1985).Normally this leads to speculation on the best way to choose β k .
Touati -Ahmed and Storey in 1995 proposed the following hybrid algorithm.
and λ>0, go to step 2 Otherwise set Here σ µ, and λ are supplied parameters.This hybridization was shown to be globally convergent under both exact and inexact line searches and to be quite competitive with PR-algorithm and FR-algorithm.(Touati and Storey, 1995).
Lui and Storey investigated the rate of this proposed algorithm in 1991, and they proved the following result concerning global convergence of conjugate gradient algorithms.( Lui and Storey, 1991).

Modification of the Pair Conjugate Algorithm:
Touati and Storey in 1995 suggested the following algorithm to compute the conjugancy coefficient β i : Step 1: ; return to main program.Otherwise, go to step 2.
Step 2: If ( ; return to main program.Otherwise go to step 3.
Step 3: where holds, then ; return to main program.Otherwise set ; return to main program.In order to improve the conjugate pair (CP) algorithm, we used the hybrid approach.Now we make a combined algorithm between CP algorithm and hybrid CG algorithm as follows:

The Outlines of the New Algorithm:
Step (1): i=1.

Otherwise set
.Otherwise continue.

Conclusion and Results:
Several standard test functions were minimized to compare the new algorithm with standard CP algorithm.The same line search was employed in each of algorithms, and the cubic interpolation is used.We tabulate for all the algorithms the number of functions evaluations (NOF), the number of iterations (NOI).Overall totals are also given for NOF and NOI with each algorithm.
Table (1) gives the comparison between the standard CP algorithm and the new algorithm, this table indicates that the new algorithm is better than the standard CP algorithm.x 0 =(-1;…) T .
be conjugate to each other with respect to a given positive definite symmetric matrix A if consequence of the positive definiteness of A,