We introduce the convex bundle method to solve convex, non-smooth optimization problems on Riemannian manifolds of bounded sectional curvature. Each step of our method is based on a model that involves the convex hull of previously collected subgradients, parallelly transported into the current serious iterate. This approach generalizes the dual form of classical bundle subproblems in Euclidean space. We prove that, under mild conditions, the convex bundle method converges to a minimizer. Several numerical examples implemented using Manopt$.$jl illustrate the performance of the proposed method and compare it to the subgradient method, the cyclic proximal point algorithm, as well as the proximal bundle method.
As the title suggests, this is the third paper in a series addressing bilevel optimization problems that are governed by the Kantorovich problem of optimal transport. These tasks can be reformulated as mathematical problems with complementarity constraints in the space of regular Borel measures. Due to the nonsmoothness that is introduced by the complementarity constraints, such problems are often regularized, for instance, using entropic regularization. In this series of papers, however, we apply a quadratic regularization to the Kantorovich problem. By doing so, we enhance its numerical properties while preserving the sparsity structure of the optimal transportation plan as much as possible. While the first two papers in this series focus on the well-posedness of the regularized bilevel problems and the approximation of solutions to the bilevel optimization problem in the infinite-dimensional case, in this paper, we reproduce these results for the finite-dimensional case and present findings that go well beyond the ones of the previous papers and pave the way for the numerical treatment of the bilevel problems. 27 pages, 15 figures
A solution algorithm for a special class of optimal control problems subject to an ordinary differential equation is proposed. The controls possess a continuous-or-off structure and are priced by a convex function. Additionally, a total variation regularization is applied to penalize switches. Our solution method combines a trust-region method and a proximal gradient method. The subproblems are solved via Bellman's optimality principle. Convergence with respect to a criticality measure is proven. As a numerical example, we solve a simple optimal control problem involving an SIR model.
It is well known that functions (resp. operators) satisfying a property~$p$ on a subset $Q\subset \mathbb{R}^d$ cannot necessarily be extended to a function (resp. operator) satisfying~$p$ on the whole of~$\mathbb{R}^d$. Given $Q \subseteq \mathbb{R}^d$, this work considers the problem of obtaining necessary and ideally sufficient conditions to be satisfied by a function (resp. operator) on $Q$, ensuring the existence of an extension of this function (resp. operator) satisfying $p$ on $\mathbb{R}^d$. More precisely, given some property $p$, we present a refinement procedure to obtain stronger necessary conditions to be imposed on $Q$. This procedure can be applied iteratively until the stronger conditions are also sufficient. We illustrate the procedure on a few examples, including the strengthening of existing descriptions for the classes of smooth functions satisfying a Łojasiewicz condition, convex blockwise smooth functions, Lipschitz monotone operators, strongly monotone cocoercive operators, and uniformly convex functions. In most cases, these strengthened descriptions can be represented, or relaxed, to semi-definite constraints, which can be used to formulate tractable optimization problems on functions (resp. operators) within those classes.
A proximal safeguarded augmented Lagrangian method for minimizing the difference of convex (DC) functions over a nonempty, closed and convex set with additional linear equality as well as convex inequality constraints is presented. Thereby, all functions involved may be nonsmooth. Iterates (of the primal variable) are obtained by solving convex optimization problems as the concave part of the objective function gets approximated by an affine linearization. Under the assumption of a modified Slater constraint qualification, both convergence of the primal and dual variables to a generalized Karush-Kuhn-Tucker (KKT) point is proven, at least on a subsequence. Numerical experiments and comparison with existing solution methods are presented using some classes of constrained and nonsmooth DC problems. 38 pages,4 figures; final version: proof of Lemma 3.3 has been restructured
We investigate a globalized inexact semismooth Newton method applied to strongly convex optimization problems in Hilbert spaces. Here, the semismooth Newton method is appplied to the dual problem, which has a continuously differentiable objective. We prove global strong convergence of iterates as well as transition to local superlinear convergence. The latter needs a second-order Taylor expansion involving semismooth derivative concepts. The convergence of the globalized method is demonstrated in numerical examples, for which the local unglobalized method diverges.
We consider minimization problems with structured objective function and smooth constraints, and present a flexible framework that combines the beneficial regularization effects of (exact) penalty and interior-point methods. In the fully nonconvex setting, a pure barrier approach requires careful steps when approaching the infeasible set, thus hindering convergence. We show how a tight integration with a penalty scheme mitigates this issue and enables the construction of subproblems whose domain is independent of the explicit constraints. This decoupling allows us to leverage efficient solvers designed for unconstrained or suitably structured optimization tasks. The key behind all this is a marginalization step: closely related to a conjugacy operation, this step effectively merges (exact) penalty and barrier into a smooth, full domain functional object. When the penalty exactness takes effect, the generated subproblems do not suffer the ill-conditioning typical of barrier methods, nor do they exhibit the nonsmoothness of exact penalty terms. We provide a theoretical characterization of the algorithm and its asymptotic properties, deriving convergence results for fully nonconvex problems. Stronger conclusions are available for the convex setting, where optimality can be guaranteed. Illustrative examples and numerical simulations demonstrate the wide range of problems our theory and algorithm are able to cover.
Bilevel optimisation is used in inverse imaging problems for hyperparameter learning/identification and experimental design, for instance, to find optimal regularisation parameters and forward operators. However, computationally, the process is costly. To reduce this cost, recently so-called single-loop approaches have been introduced. On each step of an outer optimisation method, they take just a single gradient step towards the solution of the inner problem. In this paper, we flexibilise the inner algorithm to include standard methods in inverse imaging. Moreover, as we have recently shown, significant performance improvements can be obtained in PDE-constrained optimisation by interweaving the steps of conventional iterative linear system solvers with the optimisation method. We now demonstrate how the adjoint equation in bilevel problems can also benefit from such interweaving. We evaluate the performance of our approach on identifying the deconvolution kernel for image deblurring, and the subsampling operator for magnetic resonance imaging (MRI).
We address second-order optimality conditions for optimal control problems involving sparsity functionals which induce spatio-temporal sparsity patterns. We employ the notion of (weak) second subderivatives. With this approach, we are able to reproduce the results from Casas, Herzog, and Wachsmuth (ESAIM COCV, 23, 2017, p. 263-295). Our analysis yields a slight improvement of one of these results and also opens the door for the sensitivity analysis of this class of problems.
In this paper, we consider optimization problems with $L^0$-cost of the controls. Here, we take the support of the control as independent optimization variable. Topological derivatives of the corresponding value function with respect to variations of the support are derived. These topological derivatives are used in a novel gradient descent algorithm with Armijo line-search. Under suitable assumptions, the algorithm produces a minimizing sequence.
A broad class of optimization problems can be cast in composite form, that is, considering the minimization of the composition of a lower semicontinuous function with a differentiable mapping. This paper investigates the versatile template of composite optimization without any convexity assumptions. First- and second-order optimality conditions are discussed. We highlight the difficulties that stem from the lack of convexity when dealing with necessary conditions in a Lagrangian framework and when considering error bounds. Building upon these characterizations, a local convergence analysis is delineated for a recently developed augmented Lagrangian method, deriving rates of convergence in the fully nonconvex setting. Comment: 36 pages
Lorena Bociu, Paul Manns, Marvin Severitt, Sarah Strikwerda
We revisit a class of integer optimal control problems for which a trust-region method has been proposed and analyzed in arXiv:2106.13453v3 [math.OC]. While the algorithm proposed in arXiv:2106.13453v3 [math.OC] successfully solves the class of optimization problems under consideration, its convergence analysis requires restrictive regularity assumptions. There are many examples of integer optimal control problems involving partial differential equations where these regularity assumptions are not satisfied. In this article we provide a way to bypass the restrictive regularity assumptions by introducing an additional partial regularization of the control inputs by means of mollification and proving a $\Gamma$-convergence-type result when the support parameter of the mollification is driven to zero. We highlight the applicability of this theory in the case of fluid flows through deformable porous media equations that arise in biomechanics. We show that the regularity assumptions are violated in the case of poro-visco-elastic systems, and thus one needs to use the regularization of the control input introduced in this article. Associated numerical results show that while the homotopy can help to find better objective values and points of lower instationarity, the practical performance of the algorithm without the input regularization may be on par with the homotopy.
In this paper, we extend the previous convergence results for the generalized alternating projection method applied to subspaces in [arXiv:1703.10547] to hold also for smooth manifolds. We show that the algorithm locally behaves similarly in the subspace and manifold settings and that the same rates are obtained. We also present convergence rate results for when the algorithm is applied to non-empty, closed, and convex sets. The results are based on a finite identification property that implies that the algorithm after an initial identification phase solves a smooth manifold feasibility problem. Therefore, the rates in this paper hold asymptotically for problems in which this identification property is satisfied. We present a few examples where this is the case and also a counter example for when this is not.
Point source localisation is generally modelled as a Lasso-type problem on measures. However, optimisation methods in non-Hilbert spaces, such as the space of Radon measures, are much less developed than in Hilbert spaces. Most numerical algorithms for point source localisation are based on the Frank-Wolfe conditional gradient method, for which ad hoc convergence theory is developed. We develop extensions of proximal-type methods to spaces of measures. This includes forward-backward splitting, its inertial version, and primal-dual proximal splitting. Their convergence proofs follow standard patterns. We demonstrate their numerical efficacy.
Motivated by fatigue damage models, this paper addresses optimal control problems governed by a non-smooth system featuring two non-differentiable mappings. This consists of a coupling between a doubly non-smooth history-dependent evolution and an elliptic PDE. After proving the directional differentiability of the associated solution mapping, an optimality system which is stronger than the one obtained by classical smoothening procedures is derived. If one of the non-differentiable mappings becomes smooth, the optimality conditions are of strong stationary type, i.e., equivalent to the primal necessary optimality condition.
Paul Manns, Mirko Hahn, Christian Kirches, Sven Leyffer, Sebastian Sager
Binary trust-region steepest descent (BTR) and combinatorial integral approximation (CIA) are two recently investigated approaches for the solution of optimization problems with distributed binary-/discrete-valued variables (control functions). We show improved convergence results for BTR by imposing a compactness assumption that is similar to the convergence theory of CIA. As a corollary we conclude that BTR also constitutes a descent algorithm on the continuous relaxation and its iterates converge weakly-$^*$ to stationary points of the latter. We provide computational results that validate our findings. In addition, we observe a regularizing effect of BTR, which we explore by means of a hybridization of CIA and BTR.
We address composite optimization problems, which consist in minimizing the sum of a smooth and a merely lower semicontinuous function, without any convexity assumptions. Numerical solutions of these problems can be obtained by proximal gradient methods, which often rely on a line search procedure as globalization mechanism. We consider an adaptive nonmonotone proximal gradient scheme based on an averaged merit function and establish asymptotic convergence guarantees under weak assumptions, delivering results on par with the monotone strategy. Global worst-case rates for the iterates and a stationarity measure are also derived. Finally, a numerical example indicates the potential of nonmonotonicity and spectral approximations. Comment: 18 pages, 1 algorithm, 1 figure
We study no-gap second-order optimality conditions for a non-uniformly convex and non-smooth integral functional. The integral functional is extended to the space of measures. The obtained second-order derivatives contain integrals on lower-dimensional manifolds. The proofs utilize the convex pre-conjugate, which is an integral functional on the space of continuous functions. Applications to non-smooth optimal control problems are given.
This article is largely concerned with the time-discretization of descriptor-variable systems coupled to with complementarity constraints. They are named descriptor-variable linear complementarity systems (DVLCS). More speci cally passive DVLCS with minimal state space representation are studied. The Euler implicit discretization of DVLCS is analysed: the one-step non-smooth problem (OSNSP), that is a generalized equation, is shown to be well-posed under some conditions. Then the convergence of the discretized solutions is studied. Several examples illustrate the applicability and the limitations of the developments.
We compute the minimal angle spread with respect to the uniform distribution in the probability simplex. The resulting optimization problem is analytically solved. The formula provided shows that the minimal angle spread approaches zero as the dimension tends to infinity. We also discuss an application in cognitive science.
