We consider discrete ill-posed inverse problems of the form

\begin{equation} \label {eq: nonlinear} \mathbf {A}(\mathbf {y})\mathbf {x} \approx \mathbf {b} = \mathbf {b}_{\rm true} + \mathbf {\epsilon } \quad \mbox { with } \mathbf {A}(\mathbf {y}_{\rm true}) \mathbf {x}_{\rm true} = \mathbf {b}_{\rm true}, \end{equation}

where the vector \(\mathbf {b}_{\rm true} \in \mathbb {R}^m\) denotes an unknown error-free vector associated with available data and \(\mathbf {\epsilon } \in \mathbb {R}^m\) is an unknown vector that represents the noise/errors in \(\mathbf {b}\). The matrix \(\mathbf {A}(\mathbf {y}) \in \mathbb {R}^{m \times n}\) with \(m\geq n\) models a forward operator and is typically severely ill-conditioned. We assume that \(\mathbf {A}\) is unknown but can be parametrized by a vector \(\mathbf {y}\in \mathbb {R}^r\) with \(r \ll n\) in such a way that the map \(\mathbf {y}\mapsto \mathbf {A}(\mathbf {y})\) is differentiable. We aim to compute good approximations of \(\mathbf {x}_{\rm true}\) and \(\mathbf {y}_{\rm true}\), given a data vector \(\mathbf {b}\) and a matrix function that maps the unknown vector \(\mathbf {y}\) to an \(m \times n\) matrix \(\mathbf {A}\). To accomplish this task, we could solve

\begin{equation} \label {eq: nsl2} \min _{\mathbf {x}, \mathbf {y}} \frac {1}{2}\left \|\mathbf {A}(\mathbf {y})\mathbf {x} - \mathbf {b} \right \|_2^2 + \frac {\lambda ^2}{2} \left \|\mathbf {L} \mathbf {x}\right \|^2_2, \end{equation}

where \(\lambda >0\) is a regularization parameter and \(\mathbf {L} \in \mathbb {R}^{q\times n}\) is a regularization operator. We assume that \(\mathbf {L}\) satisfies that \(\mathcal {N}(\mathbf {A}(\mathbf {y})) \cap \mathcal {N} (\mathbf {L}) = \{0\}\) for all feasible values of \(\mathbf {y}\), so that the minimization problem (2) has a unique solution for \(\mathbf {y}\) fixed. We call problems of the form (2) regularized separable nonlinear inverse problems since the observations depend nonlinearly on the vector of unknown parameters \(\mathbf {y}\) and linearly on the solution vector \(\mathbf {x}\).

The Variable Projection (VarPro) method was originally developed in the 1970s by Golub and Pereyra [3] to solve (2) for \(\lambda =0\) and has been widely recognized for its efficiency in solving separable nonlinear least squares problems. VarPro eliminates the linear variables \(\mathbf {x}\) through projection and reduces the original problem to a smaller nonlinear least squares problem in the parameters \(\mathbf {y}\). This reduced nonlinear least squares problem can be solved using the Gauss-Newton Method.

In [1], Español and Pasha extended VarPro to solve inverse problems with general-form Tikhonov regularization for general matrices \(\mathbf {L}\). They named this method GenVarPro. For special cases where computing the generalized singular value decomposition (GSVD) of the pair \(\{\mathbf {A}(\mathbf {y}),\mathbf {L}\}\) for a fixed value of \(\mathbf {y}\) is feasible or a joint spectral decomposition exists, they provided efficient ways to compute the Jacobian matrix and the solution of the linear subproblems. For large-scale problems, where matrix decompositions are not an option, they proposed computing a reduced Jacobian and applying projection-based iterative methods and generalized Krylov subspace methods to solve the linear subproblems. Following on this theme, Español and Jeronimo introduced in [2], the Inexact-GenVarPro that considers a new approximate Jacobian where iterative methods such as LSQR and LSMR are used to solve the linear subproblems. Furthermore, specific stopping criteria were proposed to ensure Inexact-GenVarPro’s local convergence.

In this talk, we will show how to extend GenVarPro and Inexact-GenVarPro to solve

\begin{equation} \label {eq: nlls_regy} \min _{\mathbf {x}, \mathbf {y}} \frac {1}{2}\left \|\mathbf {A}(\mathbf {y})\mathbf {x} - \mathbf {b} \right \|_2^2 + \frac {\lambda ^2}{2} \left \|\mathbf {L} \mathbf {x}\right \|^2_2 + \mu \mathcal {R}(\mathbf {y}), \end{equation}

where \(\mu >0\) is another regularization parameter and \(\mathcal {R}(\mathbf {y})\) plays the role of regularization on the parameter vector \(\mathbf {y}\). Similar variational formulations have appeared in recent papers in the context of training neural networks [4] and computerized tomographic reconstruction [5]. We will motivate the need to incorporate this regularization term on \(\boldsymbol {y}\) in the context of a semi-blind image deblurring problem by showing some examples where, without it, the solution of the reduced problem does not exist (i.e., no minimizer exists) or is trivial (e.g., \(\boldsymbol {y}=\boldsymbol {0}\) and \(\boldsymbol {A}(\boldsymbol {y})\) becomes the identity matrix). We will show in particular, how to extend GenVarPro and Inexact-GenVarPro to the case when \(\mathcal {R(\mathbf {y})}=\|\mathbf {y}-\mathbf {y}_0\|^2_2\) and \(\mathcal {R(\mathbf {y})}=-\sum _{j}\log (y_j)\) in the context of a large-scale semi-blind image deblurring problem. Furthermore, we will present theoretical results with sufficient conditions on the matrices involved to ensure local convergence. Numerical experiments will also be presented to illustrate their efficiency and confirm the theoretical results.

References

• [1]  M. I. Español and M. Pasha. Variable projection methods for separable nonlinear inverse problems with general-form Tikhonov regularization. Inverse Problems, 39(8):084002, 2023.

• [2]  M. I. Español and G. Jeronimo. Convergence analysis of a variable projection method for regularized separable nonlinear inverse problems. arXiv preprint arXiv:2402.08568, 2024.

• [3]  G. H. Golub and V. Pereyra. The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM Journal on Numerical Analysis, 10(2):413–432, 1973.

• [4]  E. Newman, L. Ruthotto, J. Hart, and B. van Bloemen Waanders. Train like a (Var) Pro: Efficient training of neural networks with variable projection. SIAM Journal on Mathematics of Data Science, 3(4):1041–1066, 2021.

• [5]  F. Uribe, J. M. Bardsley, Y. Dong, P. C. Hansen, and N. A. B. Riis. A hybrid Gibbs sampler for edge-preserving tomographic reconstruction with uncertain view angles. SIAM/ASA Journal on Uncertainty Quantification, 10(3):1293–1320, 2022.