Convergence Analysis of an Adaptively Regularized Natural Gradient Method

Document Type

Article

Publication Date

1-1-2024

Abstract

In this paper, we study the convergence properties of the natural gradient methods. By reviewing the mathematical condition for the equivalence between the Fisher information matrix and the generalized Gauss-Newton matrix, as well as the comparisons on the computation and storage, we reveal the popularity of the natural gradient method. To ensure the global convergence, an adaptively regularized natural gradient method is proposed. By requiring sufficient probabilistic accurate estimations on both the function and the gradient evaluations, we establish the almost sure convergence. In the local convergence, we employ the local error bound condition and show the convergence rate can be quadratic by adding mild assumptions on the stochastic estimates of gradients and Fisher information matrices. Preliminary numerical experiments on the regularized logistic regression are performed to support our findings.

Publication Source (Journal or Book title)

IEEE Transactions on Signal Processing

First Page

2527

Last Page

2542

This document is currently not available here.

Share

COinS