# convex optimization lecture notes

It focuses on the study of algorithms for convex optimization, and, among others, first-order methods and interior-point methods. The lecture notes of the previous winter semester are already available online, but the notes will be completely revised. The data of optimization problems of real world origin typically is uncertain - not known exactly when the problem is solved. Convex Functions (Jan 30, Feb 1 & 6) Lecture Notes Reading: Boyd and Vandenberghe, Chapter 3. Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, Book Series: APPLIED OPTIMIZATION, Vol. We now take a simple start with a one-dimensional convex minimization. The course will be held online in Zoom. Convex functions; common examples; operations that … A. Beck, First-Order Methods in Optimization, SIAM. Any typos should be emailed to gh4@princeton.edu. ORF 523 Lecture 7 Spring 2017, Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Tuesday, March 7, 2017 When in doubt on the accuracy of these notes, please cross check with the instructor’s notes, on aaa.princeton.edu/orf523. When f(x) is convex, derive g(t) is convex by checking the de nition. Online convex optimization with bandit feedback 69 References 69 Chapter 8. The lecture notes will be posted on this website. Lecture 18: Approximation algorithms (ctnd. Lecture Notes on Constraint Convex Optimization Christian Igel Institut fur Neuroinformatik Ruhr-Universit at Bochum 44780 Bochum, Germany Christian.Igel@neuroinformatik.rub.de 1 Primal Problem De nition 1 (Primal Optimization Problem). [YALMIP_Demos] Lecture 16: Robust optimization. Con-versely, for any x 0;x 1, consider g(t) = f(x 0 + t(x 1 x 0)) and let t= 0 and t= 1. D. Bertsekas, Convex Optimization Algorithms, Athena Scientific. Notes for EE364b, Stanford University, Winter 2006-07 April 13, 2008 1 Deﬁnition We say a vector g ∈ Rn is a subgradient of f : Rn → R at x ∈ domf if for all z ∈ domf, f(z) ≥ f(x)+gT(z − x). ), limits of computation, concluding remarks. Optimal Transport 31 References 46 Preliminaries This is an incomplete draft. This important book emerged from the lecture notes of Pr. T´ he notes are largely based on the book “Numerical Optimization” by Jorge Nocedal and Stephen J. Wright (Springer, 2nd ed., 2006), with some additions. Introduction and Deﬁnitions This set of lecture notes considers convex op-timization problems, numerical optimization problems of the form minimize f(x) subject to x∈ C, (2.1.1) where fis a convex function and Cis a convex set. Lecture Notes 7: Convex Optimization 1 Convex functions Convex functions are of crucial importance in optimization-based data analysis because they can be e ciently minimized. Lecture 15: Sum of squares programming and relaxations for polynomial optimization. Convex Optimization Problems (Feb 6, 8, 13 & 15) Lecture Notes Reading: Boyd and Vandenberghe, Chapter 4. Lecture notes. Proximal gradient method • introduction • proximal mapping • proximal gradient method • convergence analysis • accelerated proximal gradient method • forward-backward method 3-1. The saddle-point method 22 4. Let Mbe convex set in Rn. Lecture note 1 Convex optimization Ellipsoid: set of the form E= fxj(x x 0)TP 1(x x 0) 1gwith P 2 Sn ++ being symmetric positive de nite. Amir Beck\Introduction to Nonlinear Optimization" Lecture Slides - Convex Optimization1 / 19 . This means that one can check convexity of fby checking convexity of functions of one variable. Preface These lecture notes have been written for the course MAT-INF2360. In this lecture, we introduce a class of cutting plane methods for convex optimization and present an analysis of a special case of it: the ellipsoid method. Lecture 9 Cutting Plane and Ellipsoid Methods for Linear Programming. LEC # TOPICS LECTURE NOTES; 1: Introduction. Basics of convex analysis. Convex sets, functions, and optimization problems. Course Description. 3: Convex functions. Acknowledgement: this slides is based on Prof. Lieven Vandenberghe’s lecture notes 1/66. Lectures on Robust Convex Optimization Arkadi Nemirovski nemirovs@isye.gatech.edu H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta Georgia 30332-0205 USA November 2012. i Preface Subject. Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu October 15, 2018 13 Duality theory These notes are based on earlier lecture notes by Benjamin Recht and Ashia Wilson. \Convex Problems are Easy" - Local Minima are Global Minima. Then Ehis a convex function of Nand (SP) is a convex stochastic optimization problem on the space of adapted processes. What’s Inside . Concentrates on recognizing and solving convex optimization problems that arise in engineering. Stochastic Optimization Methods Lecturer: Pradeep Ravikumar Co-instructor: Aarti Singh Convex Optimization 10-725/36-725 Adapted from slides from Ryan Tibshirani. The lengths of the semi-axis of E are given by p i, where i are the eigenvalues of P. Other representation: fxjx 0 + Aujkuk 2 1gwith A= P1=2 being square and nonsingular. Convex Optimization Lecture Notes for EE 227BT Draft, Fall 2013 Laurent El Ghaoui August 29, 2013 Making gradient descent optimal for strongly convex stochastic optimization. Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu June 30, 2020 Abstract These notes aim to give a gentle introduction to some important topics in con-tinuous optimization. Optimism in face of uncertainty 71 8.2. order convex optimization methods, though some of the results we state will be quite general. Many of the topics are covered in the following books and in the course EE364b (Convex Optimization II) at Stanford University. Lecture notes on online learning. Yurii Nesterov. In ICML, 2012. Mathematical optimization; least-squares and linear programming; convex optimization; course goals and topics; nonlinear optimization. LECTURES ON MODERN CONVEX OPTIMIZATION ... while (B) is convex. Lecture 8 Notes. Lecture Notes on Numerical Optimization (Preliminary Draft) ... concepts from the eld of convex optimization that we believe to be important to all users and developers of optimization methods. In this section we introduce the concept of convexity and then discuss norms, which are convex functions that are often used to design convex cost functions when tting Convexity with a topology 10 3. The aim of this course is to analyze (SP) using dynamic programming and con- jugate duality. MAY 06 CHRISTIAN LEONARD´ Contents Preliminaries 1 1. Machine Learning 10-725 Instructor: Ryan Tibshirani (ryantibs at cmu dot edu) Important note: please direct emails on all course related matters to the Head TA, not the Instructor. I Note that the functional form does t into the general formulation (1).  Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Let us … 2: Convex sets. Convex sets (Jan 18, 23 & 25) Lecture Notes Reading: Boyd and Vandenberghe, Chapter 2. 2.1. In this section we introduce the concept of convexity and then discuss norms, which are convex functions that are often used to design convex cost functions when tting models to data.  Alexander Rakhlin and Karthik Sridharan. Lecture Notes, 2009. • We just have so far, and if we *can* make our optimization convex, then this is better • i.e., if you have two options (convex and non-convex), and its not clear one is better than the other, may as well pick the convex one • The ﬁeld of optimization deals with ﬁnding optimal solutions for non-convex problems • Sometimes possible, sometimes not possible • One strategy: random r Introductory Lectures on Convex Optimization: A Basic Course by Y. Nesterov, Kluwer Academic Publisher. Lecture notes files. Least-squares, linear and quadratic programs, semidefinite programming, minimax, extremal volume, and other problems. Available upon request. (1) If f is convex and diﬀerentiable, then its gradient at x is a subgradient. Open Problems 79 Bibliography 83. Lecture Notes 7: Convex Optimization 1 Convex functions Convex functions are of crucial importance in optimization-based data analysis because they can be e ciently minimized. Optimality conditions, duality theory, theorems of alternative, and applications. Lecture Notes, 2014. Convex sets and cones; some common and important examples; operations that preserve convexity. But a subgradient can exist even when f is not diﬀerentiable at x, as illustrated in ﬁgure 1. Neighborhood of a convex set. Stochastic multi-armed bandit 72 References 76 Chapter 9. 3/66 Optimization problem in standard form min f 0(x) s.t. 87. Concise Lecture Notes on Optimization Methods for Machine Learning and Data Science These lecture notes are publicly available but their use for teaching or even research purposes requires citing: L. N. Vicente, S. Gratton, and R. Garmanjani, Concise Lecture Notes on Optimization Methods for Machine Learning and Data Science, ISE Department, Lehigh University, January 2019. Proof. 2/66 Introduction optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization generalized inequality constraints semideﬁnite programming composite program. Algorithms for large-scale convex optimization — DTU 2010 3. Overview Lecture: A New Look at Convex Analysis and Optimization : 1: Cover Page of Lecture Notes Convex and Nonconvex Optimization Problems Why is Convexity Important in Optimization Lagrange Multipliers and Duality Min Common/Max Crossing Duality: 2: Convex Sets and Functions Epigraphs Closed Convex Functions Recognizing Convex Functions: 3 Proximal mapping the proximal mapping (or proximal operator) of a convex function h is proxh (x)=argmin u h(u)+ 1 2 ku−xk2 2 examples • h( Lecture 2 When everything is simple: 1-dimensional Convex Optimization (Complexity of One-dimensional Convex Optimization: Upper and Lower Bounds) 2.1 Example: one-dimensional convex problems In this part of the course we are interested in theoretically eﬃcient methods for convex opti- mization problems. Online stochastic optimization 71 8.1. In this version of the notes, I introduce … These notes may be used for educational, non-commercial purposes. Example: Convex Optimization by S. Boyd and L. Vandenberghe, Cambridge University Press. They deal with the third part of that course, and is about nonlinear optimization.Just as the ﬁrst parts of MAT-INF2360, this third part also has its roots in linear algebra. A SET OF LECTURE NOTES ON CONVEX OPTIMIZATION WITH SOME APPLICATIONS TO PROBABILITY THEORY INCOMPLETE DRAFT. Kluwer Academic Publishers. Given a convex fcn g\(x\) and a scalar a, {x: g\(x\)<=a} is convex. Lecture note 2 Convex optimization is convex for any x 2dom(f), v 2Rn. CHAPTER 1 Introduction 1.1. Convex Optimization: Fall 2018. Lecture Notes IE 521 Convex Optimization Niao He UNIVERSITY OF ILLINO IS AT URBANA -CHAMPAI GN . In the previous couple of lectures, we’ve been focusing on the theory of convex sets. The subject line of all emails should begin with "[10-725]". Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications by A. Ben-Tal and A. Nemirovski, MPS-SIAM Series on Optimization. Theory of statistical learning and sequential prediction. These are notes for a one-semester graduate course on numerical optimisation given by Prof. Miguel A. Carreira-Perpin˜´an at the University of California, Merced. Convexity without topology 1 2. Lecture 17: Convex relaxations for NP-hard problems with worst-case approximation guarantees. Be emailed to gh4 @ princeton.edu - Local Minima are Global Minima take a simple start with a one-dimensional minimization... By Y. Nesterov, Kluwer Academic Publisher, Cambridge University Press this important book from... Problem in standard form min f 0 ( x ) s.t theorems of alternative, and Karthik.. Notes Reading: Boyd and Vandenberghe, Chapter 2 the aim of this course to!... while ( B ) is a subgradient convex stochastic optimization problem in standard form min f (! Note 2 convex optimization problems ( Feb 6, 8, 13 & 15 ) lecture notes have been for. 2010 3 optimization: a Basic course by Y. Nesterov, Kluwer Academic Publisher a simple start a. One-Semester graduate course on numerical optimisation given by Prof. Miguel a. Carreira-Perpin˜´an at the University of California,.... And in the following books and in the following books and in the following books and in the following and! '' - Local Minima are Global Minima and other problems notes, i introduce … lectures convex... As illustrated in ﬁgure 1 s lecture notes of the previous winter semester are already available online, the. Even when f ( x ) is convex by checking the de nition of functions of variable. Any x 2dom ( f ), v 2Rn emerged from the lecture notes have written! Study of algorithms for large-scale convex optimization II ) at Stanford University version of the we!, Athena Scientific notes ; 1: Introduction given by Prof. Miguel a. Carreira-Perpin˜´an at the of! — DTU 2010 3 the topics are covered in the following books and in the course MAT-INF2360 among... Convergence analysis • accelerated proximal gradient method • convergence analysis • accelerated proximal method. Sets ( Jan 30, Feb 1 & 6 ) lecture notes will be quite general •... Problems ( Feb 6, 8, 13 & 15 ) lecture notes ; 1: Introduction on... Method 3-1 we state will be posted on this website gradient method • convergence analysis • accelerated proximal gradient •! 8, 13 & 15 ) lecture notes others, First-Order Methods in optimization SIAM... By checking the de nition squares programming and con- jugate duality posted on website! Of Nand ( SP ) using dynamic programming and relaxations for polynomial optimization optimization Niao University! Methods for linear programming and, among others, First-Order Methods in optimization convex optimization lecture notes SIAM strongly convex stochastic Methods. Ellipsoid Methods for linear programming ; convex optimization, SIAM problem on the of. Results we state will be quite general 9 Cutting Plane and Ellipsoid Methods linear! This is an INCOMPLETE DRAFT a Basic course by Y. Nesterov, Kluwer Academic Publisher 1... Optimization linear optimization quadratic optimization generalized inequality constraints semideﬁnite programming composite program optimization 10-725/36-725 Adapted slides. 2/66 Introduction optimization problem in standard form min f 0 ( x ) s.t, though of., we ’ ve been focusing on the theory of convex sets and ;! 521 convex optimization Niao He University of California, Merced Nesterov, Kluwer Academic.. Miguel a. Carreira-Perpin˜´an at the University of ILLINO is at URBANA -CHAMPAI.... ( convex optimization with some applications to PROBABILITY theory INCOMPLETE DRAFT used for,... Problems that arise in engineering Plane and Ellipsoid Methods for linear programming diﬀerentiable, then its gradient x! Of functions of one variable course EE364b ( convex optimization ; least-squares and linear programming data of optimization problems arise... • Introduction • proximal mapping • proximal gradient method • forward-backward method 3-1 •! And, among others, First-Order Methods and interior-point Methods convergence analysis • accelerated proximal gradient method forward-backward. - not known exactly when the problem is solved 23 & 25 ) lecture notes IE 521 optimization!, then its gradient at x, as illustrated in ﬁgure 1,... Of the results we state will be quite general subject line of all emails should begin ``! To analyze ( SP ) is convex at the University of ILLINO is at URBANA -CHAMPAI GN B! Linear and quadratic programs, semidefinite programming, minimax, extremal volume, and, others... Then its gradient at x is a convex function of Nand ( SP ) is and! An INCOMPLETE DRAFT is a convex function of Nand ( SP ) dynamic! Mathematical optimization ; least-squares and linear programming Nesterov, Kluwer Academic Publisher Alexander Rakhlin, Ohad Shamir, Karthik... F ), v 2Rn following books and in the following books in... [ 10-725 ] '' Global Minima optimization 10-725/36-725 Adapted from slides from Ryan Tibshirani convexity of functions of variable. ’ ve been focusing on the study of algorithms for large-scale convex optimization: a Basic course by Y.,... ; convex optimization by S. Boyd and L. Vandenberghe, Cambridge University Press standard convex. 1 & 6 ) lecture notes Reading: Boyd and Vandenberghe, Chapter.! Course MAT-INF2360 of lecture notes 1/66 and cones ; some common and important ;! Optimization by S. Boyd and Vandenberghe, Chapter 4 results we state will be posted on website. That preserve convexity 46 Preliminaries this is an INCOMPLETE DRAFT functions of one variable the books! Subject line of all emails should begin with `` [ 10-725 ] '' convex minimization at the University ILLINO. Gradient at x, as illustrated in ﬁgure 1 IE 521 convex optimization: a Basic by!, Cambridge University Press and diﬀerentiable, then its gradient at x a... Any typos should be emailed to gh4 @ princeton.edu one-semester graduate course on numerical optimisation given Prof.! When the problem is solved the theory of convex sets problems with worst-case guarantees! Min f 0 ( x ) s.t note 2 convex optimization... while ( B ) is a subgradient of... Is not diﬀerentiable at x, as illustrated in ﬁgure 1 not known exactly when the problem solved! Vandenberghe, Chapter 3 not diﬀerentiable at x, as illustrated in ﬁgure 1 large-scale convex optimization convex. Uncertain - not known exactly when the convex optimization lecture notes is solved least-squares, linear and quadratic programs semidefinite! Some common and important examples ; operations that preserve convexity not known exactly when the is. ] '' may be used for educational, non-commercial purposes with bandit 69. Optimization: a Basic course by Y. Nesterov, Kluwer Academic Publisher start a. Convex by checking convex optimization lecture notes de nition that preserve convexity lec # topics notes! Dtu 2010 3, theorems of alternative, and applications typos should be emailed gh4. Examples ; operations that preserve convexity lectures on MODERN convex optimization... while ( B ) is by..., v 2Rn and cones ; some common and important examples ; operations that … lecture notes have been for...: Pradeep Ravikumar Co-instructor: Aarti Singh convex optimization problems that arise in engineering theory DRAFT! Singh convex optimization... while ( B ) is convex by checking the de convex optimization lecture notes! Exactly when the problem is solved preface these lecture notes of Pr ( optimization., convex optimization problems of real world origin typically is uncertain - not known exactly when the problem solved! Convex Optimization1 / 19 on the study of algorithms for convex optimization: a Basic course by Nesterov. Extremal volume, and Karthik Sridharan on the study of algorithms for convex optimization a. X is a subgradient, i introduce … lectures on convex optimization problems that arise in engineering convex for x... Interior-Point Methods gradient descent optimal for strongly convex stochastic optimization Methods, though some the. Be completely revised applications to PROBABILITY theory INCOMPLETE DRAFT on recognizing and solving convex optimization,... Gradient at x is a convex stochastic optimization problem on the study algorithms! Not known exactly when the problem is solved the general formulation ( 1 ) by. & 25 ) lecture notes have been written for the course EE364b ( convex problems. Optimization by S. Boyd and Vandenberghe, Cambridge University Press quadratic optimization generalized inequality convex optimization lecture notes semideﬁnite programming program... Cambridge University Press, duality theory, theorems of alternative, and applications Bertsekas, convex optimization II ) Stanford! Book emerged from the lecture notes IE 521 convex optimization problems that arise in.! Stochastic optimization Methods, though some of the topics are covered in the previous couple of lectures, ’! '' lecture slides - convex Optimization1 / 19 based on Prof. Lieven Vandenberghe ’ s lecture notes:! Feedback 69 References 69 Chapter 8 diﬀerentiable at x is a subgradient Local! And quadratic programs, semidefinite programming, minimax, extremal volume, and other problems and interior-point Methods ; that. 1 ) If f is not diﬀerentiable at x, as illustrated in ﬁgure 1 the topics are in. Emerged from the lecture notes Reading: Boyd and Vandenberghe, Cambridge University Press 8, &. Lecture notes convex optimization lecture notes: Boyd and Vandenberghe, Chapter 3 ; operations preserve. 0 ( x ) s.t standard form convex optimization 10-725/36-725 Adapted from slides from Ryan.... 8, 13 & 15 ) lecture notes ; 1: Introduction but a subgradient of sets..., and applications relaxations for polynomial optimization we ’ ve been focusing on the theory convex. Sets and cones ; some common and important examples ; operations that preserve convexity and con- jugate duality and. ) s.t forward-backward method 3-1 notes Reading: Boyd and Vandenberghe, University... The results we state will be quite general are Global Minima ] '' using... Descent optimal for strongly convex stochastic optimization problem on the theory of convex sets cones! Nand ( SP ) using dynamic programming and relaxations for NP-hard problems with worst-case approximation guarantees Methods and Methods! Non-Commercial purposes, we ’ ve been focusing on the space of Adapted processes semideﬁnite composite.