Upon completing this course, students should be able to: D1. Knowledge and Understanding: Understand the fundamental principles of numerical methods, including their applicability, convergence, and approximation error bounds. Prove the theoretical properties of the numerical methods studied in the course. Demonstrate an understanding of the mathematical concepts necessary for training deep learning networks and solving problems in data science, including clustering and classification. D2. Applying Knowledge and Understanding: Analyze the properties of the numerical methods covered in this course. Implement these methods in a programming language. Apply the studied numerical algorithms to solve optimization problems, including those encountered in machine learning and data science, as well as least-squares problems and classification tasks. D3. Making Judgments: Recognize the strengths and weaknesses of various numerical methods. Develop critical thinking skills by comparing different alternatives for solving specific problems. Choose the most suitable method for addressing a particular problem. D4. Communication Skills: Provide clear and formal descriptions of the numerical methods studied in this course, using appropriate terminology and mathematical rigor. Deliver presentations that describe a given problem, the numerical method used to solve it, the implementation carried out, and the results obtained. D5. Learning Skills: Independently integrate various sources (textbooks, research papers, software repositories) to replicate and enhance existing numerical strategies for solving optimization problems, least-squares problems, as well as clustering and classification problems in machine learning, deep learning, and data science.

Students should have prior courses on linear algebra, calculus and multivariable calculus and numerical analysis. Familiarity with machine learning will be useful, but not required.

The course introduces some important numerical techniques currently applied in data science and machine (deep) learning. It builds on (numerical) linear algebra and numerical optimization, and it is organized into four units: 1. Optimization basics 2. Least-squares problems 3. Optimization for deep learning 4. Matrix data and latent factor models. The main topics covered will be: 1. Introduction to continuous optimization. Nonlinear unconstrained optimization, optimality conditions. Basic algorithms for unconstrained optimization (gradient descent, conjugate gradient). Newton's method. Quasi-Newton methods (Broyden class, SR1, BFGS). Limited memory versions. Line search strategies. Introduction to constrained optimization. Optimality conditions (KKT). Introduction to Interior point methods (hints). Linear algebra techniques in optimization. 2. Least-squares problems. Gauss-Newton and Levenberg-Marquardt methods for solving nonlinear least-squares problems. Tikhonov and Lasso regularization. 3. Numerical optimization for machine learning and deep learning. Classification problems. Support vector machines (SVM). Unconstrained minimization problem arising in the training of DNNs. Stochastic gradient descent. Stochastic second-order methods (L-BFGS). 4. Latent Semantic Analysis and collaborative filtering. SVD. Eckart-Young-Mirsky theorem. NMF. Overview of methods for computing NMF. Collaborative filtering and matrix completion. CUR matrix decomposition.

J. Nocedal, S.J. Wright. Numerical Optimization, 2nd ed., 2006. I. Goodfellow, Y. Bengio, A. Courville, Deep Learning. The MIT Press, 2016. G. Strang, Linear algebra and learning from data, Wellesley-Cambridge Press, 2019. Course notes will be posted on the e-learning platform Moodle before each lecture. Additional readings and support material will be shared with students through the dedicated course on the Moodle platform (https://moodle2.units.it/).

The course introduces some important numerical techniques currently applied in data science and machine (deep) learning. It builds on (numerical) linear algebra and numerical optimization. The main topics covered will be: 1. Fundamentals of continuous optimization: Classification of optimization problems. Convexity in optimization. Linear programming. Model formulation and assumptions. Introduction to nonlinear unconstrained optimization, optimality conditions. Basic algorithms for unconstrained optimization (gradient descent, conjugate gradient). Newton's method. Quasi-Newton methods (Broyden class, SR1, BFGS). Limited memory versions. Line search strategies. Introduction to constrained optimization. Optimality conditions (KKT). Introduction to Interior point methods (hints). Linear algebra techniques in optimization, Cholesky factorization and preconditioned iterative methods. 2. Least-squares problems. Gauss-Newton and Levenberg-Marquardt methods for solving nonlinear least-squares problems. Tikhonov and Lasso regularization. 3. Numerical optimization for machine learning and deep learning. Classification problems. Support vector machines (SVM). Soft margins, duality, implementation, and numerical issues. Unconstrained minimization problem arising in the training of DNNs. An overview of methods for unconstrained optimization. A motivation for stochastic gradient descent. Stochastic gradient descent: assumptions, convergence theorem for fixed stepsize. Stochastic gradient descent with decreasing step sizes, convergence theorem. Stochastic second-order methods (L-BFGS). 4. Matrix Data and Latent Factor Models. SVD. Eckart-Young-Mirsky theorem. NMF. Overview of methods for computing NMF: projected gradient descent, Lee-Seung, coordinate descent. Collaborative filtering and matrix completion. Nuclear norm. CUR matrix decomposition.

Frontal lectures. Computer exercises will be proposed, requiring application and testing of the numerical methods taught in the lectures using MATLAB or Python.

Any changes to the methods described here, which become necessary to ensure the application of the safety protocols related to the COVID-19 emergency, will be communicated on the websites of the Department of Mathematics, Informatics and Geoscience - MIGe and of the Study Program in Mathematics.

The final exam consists of an oral discussion about the topics covered in the course. It is based on three/four questions and takes 40-50 minutes on average. The exam will assess: 1. Understanding the fundamental principles of the numerical techniques introduced in the course and their application in the solution of optimization problems, least-squares problems, clustering and classification problems in machine learning, deep learning and data science. 2. The ability to select a suitable numerical method to solve a given problem. 3. The ability to write efficient MATLAB (Python) codes to solve a given problem. 4. The ability to give clear and formal descriptions of the numerical methods studied in the course, with adequate terminology and mathematical rigor. The grade is based on a 30-point scale. The minimum passing grade mark is 18. The grading system applied is: (18-24): sufficient or fair knowledge of the subject, adequate mastery of the technical language (25-27): good or very good knowledge of the subject, technical language proficiency, essential ability to connect the themes addressed during the course (28-30 with honours): excellent knowledge of the subject and technical language proficiency, autonomous critical and analytical skills, and ability to apply acquired knowledge to concrete practical scenarios.

Questo insegnamento approfondisce argomenti strettamente connessi a uno o più obiettivi dell’Agenda 2030 per lo Sviluppo Sostenibile delle Nazioni Unite