Performance Analysis of Trust Region Subproblem Solvers for Limited-Memory Distributed BFGS Optimization Method - Gao, Guohua; Florez, Horacio; Vink, Jeroen C.; Wells, Terence J.; Saaf, Fredrik; Blom, Carl P. A. (Frontiers in Applied Mathematics and Statistics, 2021)
The limited-memory BFGS (L-BFGS) optimization method performs very efficiently for large-scale problems. A trust-region search method generally performs more efficiently and robustly than a line search method, especially when the gradient of the objective function cannot be accurately evaluated. The computational cost of an L-BFGS trust-region subproblem (TRS) solver depend mainly on the number of unknown variables (n) and the number of variable shift vectors and gradient change vectors (m) used for Hessian updating, with m≪n for large-scale problems. In this paper, we analyze the performances of different methods to solve the L-BFGS TRS. The first method is the popular method proposed by More and Sorensen (MS) using Cholesky factorization of a dense n×n matrix, the second one is the method based on inverse quadratic (IQ) interpolation, and the third one is a new method that combines the matrix inversion lemma (MIL) with an approach to update associated matrices and vectors. The MIL approach is applied to reduce the dimension of the original problem with n variables to a new problem with m variables. Instead of directly applying expensive matrix-matrix and matrix-vector multiplications involved to solve the L-BFGS TRS, a more efficient approach is employed to update matrices and vectors iteratively. The L-BFGS TRS solver using the MIL method performs more efficiently than using the MS or IQ method. Testing on a representative suite of problems indicates that the new method can converge to optimal solutions comparable to those obtained using the popular MS TRS solver. Its computational cost represents only a modest overhead over the well-known L-BFGS line-search method but delivers improved stability in the presence of inaccurate gradients. When compared to the well-known MS TRS solver, the new TRS solver can reduce computational cost by a factor proportional to n^2/m for large-scale problems.