An Efficient Line Search Algorithm for Large Scale Optimization

In this work we present a new algorithm of gradient descent type, in which the stepsize is computed by means of simple approximation of the Hessian Matrix to solve nonlinear unconstrained optimization function. The new proposed algorithm considers a new approximation of the Hessian based on the function values and its gradients in two successive points along the iterations one of them use Biggs modified formula to locate the new points. The corresponding algorithm belongs to the same class of superlinear convergent descent algorithms and it has been newly programmed to obtain the numerical results for a selected class of nonlinear test functions with various dimensions. Numerical experiments show that the new choice of the step-length required less computation work and greatly speeded up the convergence of the gradient algorithm especially, for large scaled unconstrained optimization problems.


Introduction
This Paper considers the unconstrained minimization problem ) ( min where the objective function f is a twice continuously differentiable function from n  into  . For problem (1) Barzilai and Borwein [1] and Fletcher,R. [7] suggested an algorithm which essentially is a gradient one, where the choice of step-size along the negative gradient is derived from a two point approximation to the Hessian of f at With this parameter the basic iterations of the (BB) method may be given the following iterative scheme: Mainly,the sequence   k x generated by the (BB) method uses two initial vectors 0 x and 1 x .Having in view its simplicity and numerical efficiency for well-conditioned problems, the (BB) method has received a great deal of attention. However, like all steepest descent and conjugate gradient methods, the (BB) method becomes slow when the problems happens to be more ill-conditioned [1].Neculai Andrei (NA) [10] suggested another gradient descent method for unconstrained optimization (its details give in section 2 of this paper).In contrast with (BB) method a simple interpretation of the secant equation for the step-length is computed. Here, in this paper, we are going to develop numerical techniques described by Dixon [6] and Biggs [2]. The central notion is the estimation of dominant degree of a one dimensional convex function. Previously their estimate have been quite successfully used as the basis of a line search process indeed it is not performing explicit minimizations along every search direction may require more than one function evaluation per iteration to obtain a satisfactory reduction f , but if we have some measure of the non-quadratic function f in the directions then we can attempt to improve upon the simple estimate of the second directional derivative and hence update the matrix k H using more accurate information .

Neculai Andrei (NA) Algorithm:
Neculai Andrei (NA), 2005 suggested a procedure for computing an approximation of the Hessian of the function f at k x which can be considered to get the step-size along the negative gradient considered the initial point 0 x where can immediately be computed. Using the backtracking procedure (initialized with 1 =  he computed the step-length 0  which the next estimate So the first step is computed using the backtracking along the negative gradient. So the point 0,1,2,..
.This is computed using the local information from point k x , therefore the parameter was used to compute the next estimation ,then the value of function f is reduced. If not the algorithm, will restart. This suggests ) ( min arg Using the back tracking procedure to complete the algorithm he considered this situation when 0 ) In this case, the step-size k  will be changed as To get a value for k  the parameter 0   is chosen small enough, and 1 + k  may be considered as: and a new value for ) ( 1 + k x  can be computed as :

Neculai Andrei Algorithm (NA):
Corresponding (NA) gradient descent algorithm may be listed as follows: as equation (14) where k  is given by equation (13) where  is Step5: Compute the initial step-size as equation (9) with which a backtracking procedure is performed in the next step Step6: Using a backtracking procedure, determine the step-length 1 + k  as equation (11).
Step7: Update the variables: and go to step2.

A new proposed Algorithm for solving problem (1)
In this section we are going to suggest another procedure for computing an approximation of the Hessian of the function f at k x which can be considered to get the step-size along the negative gradient for (NA) algorithm for equation (10) which shows that if 0 ) then the value of function f is reduced .This enable us to determine a step-size 1 + k  as defined in equation (11) using the new way to backtracking procedure. To complete the algorithm we must consider the situation when 0 ) .In this case we use Biggs VM-Update as backtracking procedure to make the step-size k  as k k   + in such a manner that :this

The new suggested Algorithm (New):
In order to increase the efficiency of algorithm (NA), cubic line search rule is used to find the best value of the step-size used Biggs VM parameter [2] is used as backtracking procedure in order to locate the new hybrid line search to f as shown below: Step1: Step2: Test for convergence, i.e if 5

If available storage is exceeded then employ a restart option either with
Step5: Update the variables: , go to step2 Now theoretically,to ensure that the new algorithm has a super-linear convergence let us consider the following theorems in the next section.

The convergence analysis of the new suggested Algorithm:
In the following section let us consider the convergence analysis of this proposed algorithm .Assume that f is strongly convex and the sublevel is closed .Strong convexity of f on S involves the existence the constants m and M such that For more details see [11], [12].

Theorem
For strongly convex function the new algorithm with backtracking has a superlinear, convergence and is a concave function, and for all )

Numerical results:
In this section we report some numerical results obtained by a newlyprogrammed FORTRAN. Implementation of the above gradient descent algorithms for 24 test functions with different dimensions (specified in the Appendix) [12].
The comparative performances of the algorithms are taken in the usual way by considering both the total number of function evaluations (NOF) and the total number of iterations (NOI).
In each case the convergence criterion is that the value of  Table  (2) gives the percentage of improvements of NOI and NOF. The important thing is that the new algorithm is very robust in many situations especially for large-scale unconstrained optimization problems; When the iterative process reaches the same precision.

Conclusions:
In this Paper, a new gradient descent algorithm is proposed in which the step-length is computed by backtracking using a simple approximation of the Hessian based on the function values in two successive points along the iteration using Biggs [2] parameter.
Numerical experiments show that new algorithm converge superlinearly and faster. It is more efficient than Neculai Andrei (NA) algorithm in many situations. The new algorithm is expected to solve illconditioned problems and it is clear that any procedure for step-length computation does not change the superlinear convergence property of the new algorithm. The convergence rate depends greatly on the condition number of the Hessian of the minimizing function. For well conditioned convex function both algorithms are doing well, while for ill-conditions problem the new algorithm is doing well .Also, the initial step in backtracking procedure of the new algorithm is lower than the corresponding initial step of (NA) algorithm.
Finally, (NA) has a linear convergence rate while the new algorithm has superlinear rate of convergence.