MGT-418 Convex Optimization:

Optimality Conditions
Daniel Kuhn
(daniel.kuhn@epfl.ch)

128
129

Complementary Slackness
Assume that strong duality holds and that x? is optimal in P, <latexit sha1_base64="jr5dqs93o2Tk7xbmd1F6Yyjuf00=">AAADvHicbVLLjtMwFHUmPIbymoElm0CFNEhVlYxAwIoiQLBgUSTKjNSUynFuMlZtJ7Kd0irqH/E17BB8DNdpNKLpOIp9fM6xfe17k1JwY8Pwt3fgX7t+4+bhrd7tO3fv3T86fvDNFJVmMGGFKPR5Qg0IrmBiuRVwXmqgMhFwlizeOf1sCdrwQn216xJmkuaKZ5xRi9T82PsQK/jBCimpSus429RxUojUrCUOm96OKBmqktoLRvekpJWSpKNUApVKpaBdiB2xWG5cd6U2Bl2iWuIQP24+BzumRFSApuYd6mbSMWhIL3WHO3KuAdSlYTtDSwoZBrQl5bo16RwvGQ6jQRAOX7gu2vRW32Njqe7Nj/rhMGxasA+iFvRJ28b48PM4LVglQVkmqDHTKCztrKbacibcW1QGSsoWNIcpQkUlmFndhLQJniKTBlmh8Vc2aNj/V9SJ3NmhptK4lOJKlyTT1Rx5lTatbPZqVnNVVhYU2x6cVSKwReCqKUi5BmbFGgFlmmPsAbugmjKLNbdzSrN3CWw3rlWlOCtS6LDCrqymSBqwknLlblm/1bypu4bDzRx58p7n3JrBZyxnNfiIiVo8a42YkqibgH0wOR2+HoZfnvdHozY3h+QReUJOSERekhH5RMZkQpj30/vl/fH++m988Be+3FoPvHbNQ7LT/OU/JGVFEQ==</latexit>

while (λ? , μ ? ) is optimal in D. Then, <latexit sha1_base64="RDvEjAF3wW6mt36AqV7tuc+vL8w=">AAADznicbVJdb9MwFHUWPkb52uCRl8CE1ElVlSAQ8FYEEjzw0EmUTmpK5Tg3nVXbiWxnWxVFvPKr+B+88wq/ges0mmg6R7GPzzm2r31vUghubBj+8vb8Gzdv3d6/07t77/6DhweHj76avNQMJiwXuT5NqAHBFUwstwJOCw1UJgKmyeq906fnoA3P1Re7LmAu6VLxjDNqkVocetNYwQXLpaQqreKsruIkF6lZSxzq3pYoGaqS2jNGd6SklZKko5QClVKloF2IHTE/r113rTYGXaBa4BA/bT4HO6ZElICm5h2qZtIxaEivdIc78lIDqCvDZoaWFDIMaEPKdWvSS7xkOIwGQTh85bqo7vVjgW+d0m+xsVQPYllu0HFvcXAUDsOmBbsgasERadsYU7GI05yVEpRlghozi8LCziuqLWfCvU5poKBsRZcwQ6ioBDOvmiDr4DkyaZDlGn9lg4b9f0WVyK0dKiqNSzKudGkzXc2R12mz0mZv5hVXRWlBsc3BWSkCmweuvoKUa2BWrBFQpjnGHrAzqimzWIVbpzR7F8C247osFWd5Ch1W2EurKZIGrKRcuVtW7zRvKrHhcDNH9j/wJbdm8BkLXA0+YupWx60RUxJ1E7ALJi+Gb4fhycuj0ajNzT55Qp6RPonIazIin8iYTAjzfnq/vT/eX//Ev/Br//vGuue1ax6Treb/+AeAy0w5</latexit>

f0 (x ) = g(λ , μ ) (strong duality)

m p
= inf f0 (x) + λi fi (x) + μ i hi (x) (definition of g)
x
i=1 i=1
m p
f0 (x ) + λi fi (x ) + μ i hi (x ) find o castraints

i=1 i=1
f0 (x ) (λ 0, f(x ) 0, h(x ) = 0)

and thus all inequalities hold as equalities. We conclude:

• x? minimizes L(x, λ? , μ ? ) 1101 1 foix

Enfikh EEIF
<latexit sha1_base64="jr5dqs93o2Tk7xbmd1F6Yyjuf00=">AAADvHicbVLLjtMwFHUmPIbymoElm0CFNEhVlYxAwIoiQLBgUSTKjNSUynFuMlZtJ7Kd0irqH/E17BB8DNdpNKLpOIp9fM6xfe17k1JwY8Pwt3fgX7t+4+bhrd7tO3fv3T86fvDNFJVmMGGFKPR5Qg0IrmBiuRVwXmqgMhFwlizeOf1sCdrwQn216xJmkuaKZ5xRi9T82PsQK/jBCimpSus429RxUojUrCUOm96OKBmqktoLRvekpJWSpKNUApVKpaBdiB2xWG5cd6U2Bl2iWuIQP24+BzumRFSApuYd6mbSMWhIL3WHO3KuAdSlYTtDSwoZBrQl5bo16RwvGQ6jQRAOX7gu2vRW32Njqe7Nj/rhMGxasA+iFvRJ28b48PM4LVglQVkmqDHTKCztrKbacibcW1QGSsoWNIcpQkUlmFndhLQJniKTBlmh8Vc2aNj/V9SJ3NmhptK4lOJKlyTT1Rx5lTatbPZqVnNVVhYU2x6cVSKwReCqKUi5BmbFGgFlmmPsAbugmjKLNbdzSrN3CWw3rlWlOCtS6LDCrqymSBqwknLlblm/1bypu4bDzRx58p7n3JrBZyxnNfiIiVo8a42YkqibgH0wOR2+HoZfnvdHozY3h+QReUJOSERekhH5RMZkQpj30/vl/fH++m988Be+3FoPvHbNQ7LT/OU/JGVFEQ==</latexit>

<latexit sha1_base64="94lz2dsap2zQpHpPzgii6SHwFrc=">AAAD0XicbVJdb9MwFHUWPkb52uCRl0CF1ElVlUwg4K0IJHjYQ5FWNtGUynFuOqu2E9nOWIkqIV75VfwMfgGv8A+4TqOJpnMU+/icY/va9yaF4MaG4S9vx792/cbN3Vud23fu3ru/t//go8lLzWDMcpHr04QaEFzB2HIr4LTQQGUi4CRZvHH6yTlow3N1bJcFTCWdK55xRi1Ss33vU6zgC8ulpCqt4mxVxUkuUrOUOKw6G6JkqEpqzxjdkpJGSpKWUgpUSpWCdiG2xPx85bortRHoAtUCh/hx/TnYMiWiBDTV71DVk5ZBQ3qpO9yS5xpAXRrWM7SkkGFAa1IuG5Oe4yXDQdQPwsFz10WrzlHvoh8LfO6Ufo6Npbofy3KNDjqzvW44COsWbIOoAV3StBFmYxanOSslKMsENWYShYWdVlRbzoR7oNJAQdmCzmGCUFEJZlrVca6Cp8ikQZZr/JUNavb/FVUiN3aoqDQuz7jSZc60NUdepU1Km72cVlwVpQXF1gdnpQhsHrgSC1KugVmxRECZ5hh7wM6opsxiIW6cUu9dANuM66JUnOUptFhhL6ymSBqwknLlblm91rwuxprDzRzZe8vn3Jr+Eda46r/D7C0OGiOmJGonYBuMDwevBuGHZ93hsMnNLnlEnpAeicgLMiTvyYiMCfN+er+9P95f/9j/6n/zv6+tO16z5iHZaP6Pf3kETUc=</latexit>

Pm ? ?
• i=1 λi fi (x ) = 0 , which implies complementary slackness, i.e.,
<latexit sha1_base64="3sOX7yBgI339DiKBSETTbbfRcO4=">AAAD63icbVJdb9MwFE0bPkbHxwaPvBgqpE6aqmQCAQ+TikCCBx6KRNmkposc56azajuR7YxWUf8FT4hXfhTiz3CdRhNN5yj29TnH9rXPTQrBjQ2CP52uf+v2nbt793r79x88fHRw+PibyUvNYMJykevzhBoQXMHEcivgvNBAZSLgLFm8d/zZFWjDc/XVrgqYSTpXPOOMWoTiw46NFHxnuZRUpVWUrasoyUVqVhKHdW+LlAxZSe0loztU0lBJ0mJKgUypUtAuxRaZX61ddyM3Bl0gW+AQPas/F7ZEiSgBRfU7VPWkJdCQXvMubtFzDaCuBZsZSlLIMKENKFeNSM/xksEwPCbB8JXrQlRaWFpjVwJIZEoZV/w0XF9IEgm0IKUXkbFUx5xkMR8sN7Oj06AXH/SDYVA3shuETdD3mjZGn+IozVkpQVkmqDHTMCjsrKLacibc05UGCsoWdA5TDBWVYGZVfYM1eYFISrJc468sqdH/V1SJ3NqhotK4CsCVzlPT5hx4EzctbfZmVnFVlBYU2xyclYLYnLjiIynXwKxYYUCZ5pg7YZdUU2axRLdOqfcugG3ntSwVZ3kKLVTYpdUUQQNWUq7cLat3mtdlWmO4mQMHH/icW3P8GatfHX9EXxdHjRAtCdsG7AaTk+HbYfDlZX80arzZ8556z72BF3qvvZH3yRt7E491/nY73V5331f+D/+n/2sj7XaaNU+8reb//gfPB1Ur</latexit>

<latexit sha1_base64="mMR08EGewHTOlnDqtjK5H2CwvX0=">AAAEGnicbVNNj9MwEE0TPpbytQtHLoEKqStVVbICwSIWFYEEBw5FouxKTakcZ9q1ajvBdpZWUX8KJ34KJ8SVC/+GcRIt23QdxRm/92Y89kzijDNtguBvy/WuXL12fedG++at23fu7u7d+6zTXFEY0ZSn6iQmGjiTMDLMcDjJFBARcziOF28sf3wGSrNUfjKrDCaCzCWbMUoMQtO91o9IwjeaCkFkUkSzdRHFKU/0SuBn3d4gBUVWEHNKyRYV11QcN5icI5PLBJRNsUGmZ2s7XcoNQWXIZviJHpaPNRuimOeAovIeinLREChIznlrN+i5ApDngmqFkgRmmFAFilUtUnM8ZNAPe37Qf2qnEJUcrzohXyJtiJqyV0HEBJYNtD+bsu6ywvePguhrThI/MrA0BW68rtYXNS//uzaCHgXT3U7QD8rhbxthbXScegyxrNMoSWkuQBrKidbjMMjMpCDKMMrtTecaMkIXZA5jNCURoCdFeeC1/xgRTC5V+Erjl+hFjyIWGxEKIrRtGPS0LaCbnAUv48a5mT2fFExmuQFJq41nOfdN6tte9ROmgBq+QoNQxTB3n54SRajBjt7YpYydAd3Ma5lLRtMEGig3S6MIghqMIEzaUxavFSu7usQwmAW7b9mcGd37gD+L7L3DNljs10IsSdgswLYxOugf9oOPTzqDQV2bHeeB88jpOqHzzBk4752hM3Ko23YD99B94X33fnq/vN+V1G3VPvedjeH9+QfxuGe/</latexit>
λ?i > 0 =) fi (x? ) = 0 and fi (x? ) < 0 =) λ?i = 0
130

Karush-Kuhn-Tucker (KKT) Conditions

Theorem: Assume that f0 , . . . , fm , h1 , . . . , hp are differentiable, x? <latexit sha1_base64="jr5dqs93o2Tk7xbmd1F6Yyjuf00=">AAADvHicbVLLjtMwFHUmPIbymoElm0CFNEhVlYxAwIoiQLBgUSTKjNSUynFuMlZtJ7Kd0irqH/E17BB8DNdpNKLpOIp9fM6xfe17k1JwY8Pwt3fgX7t+4+bhrd7tO3fv3T86fvDNFJVmMGGFKPR5Qg0IrmBiuRVwXmqgMhFwlizeOf1sCdrwQn216xJmkuaKZ5xRi9T82PsQK/jBCimpSus429RxUojUrCUOm96OKBmqktoLRvekpJWSpKNUApVKpaBdiB2xWG5cd6U2Bl2iWuIQP24+BzumRFSApuYd6mbSMWhIL3WHO3KuAdSlYTtDSwoZBrQl5bo16RwvGQ6jQRAOX7gu2vRW32Njqe7Nj/rhMGxasA+iFvRJ28b48PM4LVglQVkmqDHTKCztrKbacibcW1QGSsoWNIcpQkUlmFndhLQJniKTBlmh8Vc2aNj/V9SJ3NmhptK4lOJKlyTT1Rx5lTatbPZqVnNVVhYU2x6cVSKwReCqKUi5BmbFGgFlmmPsAbugmjKLNbdzSrN3CWw3rlWlOCtS6LDCrqymSBqwknLlblm/1bypu4bDzRx58p7n3JrBZyxnNfiIiVo8a42YkqibgH0wOR2+HoZfnvdHozY3h+QReUJOSERekhH5RMZkQpj30/vl/fH++m988Be+3FoPvHbNQ7LT/OU/JGVFEQ==</latexit>

?
is optimal in P, , μ ? ) is optimal in D and strong duality holds (but
<latexit sha1_base64="Zx7UcXS0cFgYFt5r59AzCfibaA4=">AAAD0nicbVLLjtMwFHUmPIbymoElm0CFNEhVlSAQsCsCCRYsikSZkdoqcpyb1qofke0MU0VdILZ8FX/BH7CFL+A6DSOaGUexr885tq99blYKbl0c/wz2witXr13fv9G7eev2nbsHh/c+W10ZBhOmhTYnGbUguIKJ407ASWmAykzAcbZ64/njUzCWa/XJrUuYS7pQvOCMOoTSw2A6U/CFaSmpyutZsalnmRa5XUscNr0dUjJkJXVLRi9QWUtlWYepBDKVysH4FDukPt347lJuDKZEtsRh9rD5fNgRZaICFDXvUDeTjsBAfs77uEMvDIA6F2xnKMmhwIS2oFy3IrPAS8bDZBDFw+e+Sza9Io0HM5FrZwdFKgfLNPk3XaZlLz3ox8O4adHFIGmDPmnbGO1IZ7lmlQTlmKDWTpO4dPOaGseZ8C9UWSgpW9EFTDFUVIKd102im+gxInlUaIO/clGD/r+izuTODjWV1huNK711tst58DJuWrni5bzmqqwcKLY9uKhE5HTkayzKuQHmxBoDygzH3CO2pIYyh5W4c0qzdwlsN6+zSnGmc+igwp05QxG04CTlyt+yfm14U40Nhpt58OgtX3D04AMWuRq8Q/tWT1ohWpJ0DbgYTJ4OXw3jj8/6o1HrzT55QB6RI5KQF2RE3pMxmRAW/Ah+Bb+DP+EkrMOv4betdC9o19wnOy38/hfshk03</latexit>

(λ
P may be nonconvex). Then, (x? , λ? , μ ? ) satisfies:
<latexit sha1_base64="RDvEjAF3wW6mt36AqV7tuc+vL8w=">AAADznicbVJdb9MwFHUWPkb52uCRl8CE1ElVlSAQ8FYEEjzw0EmUTmpK5Tg3nVXbiWxnWxVFvPKr+B+88wq/ges0mmg6R7GPzzm2r31vUghubBj+8vb8Gzdv3d6/07t77/6DhweHj76avNQMJiwXuT5NqAHBFUwstwJOCw1UJgKmyeq906fnoA3P1Re7LmAu6VLxjDNqkVocetNYwQXLpaQqreKsruIkF6lZSxzq3pYoGaqS2jNGd6SklZKko5QClVKloF2IHTE/r113rTYGXaBa4BA/bT4HO6ZElICm5h2qZtIxaEivdIc78lIDqCvDZoaWFDIMaEPKdWvSS7xkOIwGQTh85bqo7vVjgW+d0m+xsVQPYllu0HFvcXAUDsOmBbsgasERadsYU7GI05yVEpRlghozi8LCziuqLWfCvU5poKBsRZcwQ6ioBDOvmiDr4DkyaZDlGn9lg4b9f0WVyK0dKiqNSzKudGkzXc2R12mz0mZv5hVXRWlBsc3BWSkCmweuvoKUa2BWrBFQpjnGHrAzqimzWIVbpzR7F8C247osFWd5Ch1W2EurKZIGrKRcuVtW7zRvKrHhcDNH9j/wJbdm8BkLXA0+YupWx60RUxJ1E7ALJi+Gb4fhycuj0ajNzT55Qp6RPonIazIin8iYTAjzfnq/vT/eX//Ev/Br//vGuue1ax6Treb/+AeAy0w5</latexit>

<latexit sha1_base64="pXg7F3vq/5XxG6iZm+GSjAkK08w=">AAAD1nicbVLLjtMwFHUmPIbymoElm0CF1JGqKkEgYFcEEixYFIkyg5pSOc5Nx6rtRLYztIoqVogtX8U/8A9sYc11GkY0HUexj885tq99b1IIbmwY/vT2/EuXr1zdv9a5fuPmrdsHh3c+mLzUDMYsF7k+SagBwRWMLbcCTgoNVCYCjpPFS6cfn4E2PFfv7aqAqaRzxTPOqEVqdujNYgWfWS4lVWkVZ+sqTnKRmpXEYd3ZEiVDVVJ7yuiOlDRSkrSUUqBSqhS0C7El5mdr112ojUAXqBY4xPfrz8GWKREloKl+h6qetAwa0nPd4ZY81wDq3LCZoSWFDAPakHLVmPQcLxkOon4QDp64Llp3estPsbFU92OBb57SfzNZbtBRZ3bQDQdh3YJdEDWgS5o2qlOS5qyUoCwT1JhJFBZ2WlFtORPulUoDBWULOocJQkUlmGlVB7sOHiKTBlmu8Vc2qNn/V1SJ3NqhotK4ZONKlz7T1hx5kTYpbfZsWnFVlBYU2xyclSKweeDqLEi5BmbFCgFlmmPsATulmjKL1bh1Sr13AWw7rmWpOMtTaLHCLq2mSBqwknLlblm90LyuyJrDzRzZe8Xn3Jr+Wyx01X+NKVwcNUZMSdROwC4YPxo8H4TvHneHwyY3++QeeUB6JCJPyZC8ISMyJsz74f3yfnt//I/+F/+r/21j3fOaNXfJVvO//wVoU0+h</latexit>

f(x ) 0, h(x ) = 0 : primal feasibility

λ 0 : dual feasibility

λi fi (x ) = 0 i = 1, . . . , m : complementary slackness
m p
f0 (x ) + λi fi (x ) + μi hi (x ) = 0 : stationarity
i=1 i=1

Proof: Primal and dual feasibility follow immediately from

optimality. Complementary slackness was derived on the previous
slide. Stationary holds because x? minimizes L(x, λ? , μ ? ). <latexit sha1_base64="jr5dqs93o2Tk7xbmd1F6Yyjuf00=">AAADvHicbVLLjtMwFHUmPIbymoElm0CFNEhVlYxAwIoiQLBgUSTKjNSUynFuMlZtJ7Kd0irqH/E17BB8DNdpNKLpOIp9fM6xfe17k1JwY8Pwt3fgX7t+4+bhrd7tO3fv3T86fvDNFJVmMGGFKPR5Qg0IrmBiuRVwXmqgMhFwlizeOf1sCdrwQn216xJmkuaKZ5xRi9T82PsQK/jBCimpSus429RxUojUrCUOm96OKBmqktoLRvekpJWSpKNUApVKpaBdiB2xWG5cd6U2Bl2iWuIQP24+BzumRFSApuYd6mbSMWhIL3WHO3KuAdSlYTtDSwoZBrQl5bo16RwvGQ6jQRAOX7gu2vRW32Njqe7Nj/rhMGxasA+iFvRJ28b48PM4LVglQVkmqDHTKCztrKbacibcW1QGSsoWNIcpQkUlmFndhLQJniKTBlmh8Vc2aNj/V9SJ3NmhptK4lOJKlyTT1Rx5lTatbPZqVnNVVhYU2x6cVSKwReCqKUi5BmbFGgFlmmPsAbugmjKLNbdzSrN3CWw3rlWlOCtS6LDCrqymSBqwknLlblm/1bypu4bDzRx58p7n3JrBZyxnNfiIiVo8a42YkqibgH0wOR2+HoZfnvdHozY3h+QReUJOSERekhH5RMZkQpj30/vl/fH++m988Be+3FoPvHbNQ7LT/OU/JGVFEQ==</latexit>

<latexit sha1_base64="94lz2dsap2zQpHpPzgii6SHwFrc=">AAAD0XicbVJdb9MwFHUWPkb52uCRl0CF1ElVlUwg4K0IJHjYQ5FWNtGUynFuOqu2E9nOWIkqIV75VfwMfgGv8A+4TqOJpnMU+/icY/va9yaF4MaG4S9vx792/cbN3Vud23fu3ru/t//go8lLzWDMcpHr04QaEFzB2HIr4LTQQGUi4CRZvHH6yTlow3N1bJcFTCWdK55xRi1Ss33vU6zgC8ulpCqt4mxVxUkuUrOUOKw6G6JkqEpqzxjdkpJGSpKWUgpUSpWCdiG2xPx85bortRHoAtUCh/hx/TnYMiWiBDTV71DVk5ZBQ3qpO9yS5xpAXRrWM7SkkGFAa1IuG5Oe4yXDQdQPwsFz10WrzlHvoh8LfO6Ufo6Npbofy3KNDjqzvW44COsWbIOoAV3StBFmYxanOSslKMsENWYShYWdVlRbzoR7oNJAQdmCzmGCUFEJZlrVca6Cp8ikQZZr/JUNavb/FVUiN3aoqDQuz7jSZc60NUdepU1Km72cVlwVpQXF1gdnpQhsHrgSC1KugVmxRECZ5hh7wM6opsxiIW6cUu9dANuM66JUnOUptFhhL6ymSBqwknLlblm91rwuxprDzRzZe8vn3Jr+Eda46r/D7C0OGiOmJGonYBuMDwevBuGHZ93hsMnNLnlEnpAeicgLMiTvyYiMCfN+er+9P95f/9j/6n/zv6+tO16z5iHZaP6Pf3kETUc=</latexit>
131

KKT Conditions for Convex Problems

?
Theorem (Sufficiency): If P is convex and (x? , λ , μ ? ) satisfies
the KKT conditions, then, x? solves P and (λ? , μ ? ) solves D.
<latexit sha1_base64="pXg7F3vq/5XxG6iZm+GSjAkK08w=">AAAD1nicbVLLjtMwFHUmPIbymoElm0CF1JGqKkEgYFcEEixYFIkyg5pSOc5Nx6rtRLYztIoqVogtX8U/8A9sYc11GkY0HUexj885tq99b1IIbmwY/vT2/EuXr1zdv9a5fuPmrdsHh3c+mLzUDMYsF7k+SagBwRWMLbcCTgoNVCYCjpPFS6cfn4E2PFfv7aqAqaRzxTPOqEVqdujNYgWfWS4lVWkVZ+sqTnKRmpXEYd3ZEiVDVVJ7yuiOlDRSkrSUUqBSqhS0C7El5mdr112ojUAXqBY4xPfrz8GWKREloKl+h6qetAwa0nPd4ZY81wDq3LCZoSWFDAPakHLVmPQcLxkOon4QDp64Llp3estPsbFU92OBb57SfzNZbtBRZ3bQDQdh3YJdEDWgS5o2qlOS5qyUoCwT1JhJFBZ2WlFtORPulUoDBWULOocJQkUlmGlVB7sOHiKTBlmu8Vc2qNn/V1SJ3NqhotK4ZONKlz7T1hx5kTYpbfZsWnFVlBYU2xyclSKweeDqLEi5BmbFCgFlmmPsATulmjKL1bh1Sr13AWw7rmWpOMtTaLHCLq2mSBqwknLlblm90LyuyJrDzRzZe8Xn3Jr+Wyx01X+NKVwcNUZMSdROwC4YPxo8H4TvHneHwyY3++QeeUB6JCJPyZC8ISMyJsz74f3yfnt//I/+F/+r/21j3fOaNXfJVvO//wVoU0+h</latexit>

<latexit sha1_base64="jr5dqs93o2Tk7xbmd1F6Yyjuf00=">AAADvHicbVLLjtMwFHUmPIbymoElm0CFNEhVlYxAwIoiQLBgUSTKjNSUynFuMlZtJ7Kd0irqH/E17BB8DNdpNKLpOIp9fM6xfe17k1JwY8Pwt3fgX7t+4+bhrd7tO3fv3T86fvDNFJVmMGGFKPR5Qg0IrmBiuRVwXmqgMhFwlizeOf1sCdrwQn216xJmkuaKZ5xRi9T82PsQK/jBCimpSus429RxUojUrCUOm96OKBmqktoLRvekpJWSpKNUApVKpaBdiB2xWG5cd6U2Bl2iWuIQP24+BzumRFSApuYd6mbSMWhIL3WHO3KuAdSlYTtDSwoZBrQl5bo16RwvGQ6jQRAOX7gu2vRW32Njqe7Nj/rhMGxasA+iFvRJ28b48PM4LVglQVkmqDHTKCztrKbacibcW1QGSsoWNIcpQkUlmFndhLQJniKTBlmh8Vc2aNj/V9SJ3NmhptK4lOJKlyTT1Rx5lTatbPZqVnNVVhYU2x6cVSKwReCqKUi5BmbFGgFlmmPsAbugmjKLNbdzSrN3CWw3rlWlOCtS6LDCrqymSBqwknLlblm/1bypu4bDzRx58p7n3JrBZyxnNfiIiVo8a42YkqibgH0wOR2+HoZfnvdHozY3h+QReUJOSERekhH5RMZkQpj30/vl/fH++m988Be+3FoPvHbNQ7LT/OU/JGVFEQ==</latexit>

<latexit sha1_base64="RDvEjAF3wW6mt36AqV7tuc+vL8w=">AAADznicbVJdb9MwFHUWPkb52uCRl8CE1ElVlSAQ8FYEEjzw0EmUTmpK5Tg3nVXbiWxnWxVFvPKr+B+88wq/ges0mmg6R7GPzzm2r31vUghubBj+8vb8Gzdv3d6/07t77/6DhweHj76avNQMJiwXuT5NqAHBFUwstwJOCw1UJgKmyeq906fnoA3P1Re7LmAu6VLxjDNqkVocetNYwQXLpaQqreKsruIkF6lZSxzq3pYoGaqS2jNGd6SklZKko5QClVKloF2IHTE/r113rTYGXaBa4BA/bT4HO6ZElICm5h2qZtIxaEivdIc78lIDqCvDZoaWFDIMaEPKdWvSS7xkOIwGQTh85bqo7vVjgW+d0m+xsVQPYllu0HFvcXAUDsOmBbsgasERadsYU7GI05yVEpRlghozi8LCziuqLWfCvU5poKBsRZcwQ6ioBDOvmiDr4DkyaZDlGn9lg4b9f0WVyK0dKiqNSzKudGkzXc2R12mz0mZv5hVXRWlBsc3BWSkCmweuvoKUa2BWrBFQpjnGHrAzqimzWIVbpzR7F8C247osFWd5Ch1W2EurKZIGrKRcuVtW7zRvKrHhcDNH9j/wJbdm8BkLXA0+YupWx60RUxJ1E7ALJi+Gb4fhycuj0ajNzT55Qp6RPonIazIin8iYTAjzfnq/vT/eX//Ev/Br//vGuue1ax6Treb/+AeAy0w5</latexit>

Proof: The KKT conditions imply

f0 (x? ) = L(x? , λ? , μ ? ) (complementary slackness)
= inf L(x, λ? , μ ? ) (stationarity and convexity of L in x)
x
<latexit sha1_base64="lIgTSqn/TzajGt4qHGKxpEPlOuQ=">AAAEu3icjVNNb9QwEE3DAmX5auHIxdAFbaXVKotAwAFURCU49FAkllZqlpXjTFJT24lspySK9hfxazgCf4ZxNq26aQ84SjLz3vN4xmNHueDGBsHvNf9a7/qNm+u3+rfv3L13f2PzwVeTFZrBlGUi04cRNSC4gqnlVsBhroHKSMBBdPLB8QenoA3P1Bdb5TCTNFU84YxahOab/m6o4AfLpKQqrsNkUYdRJmJTSfwt+iukZMhKao8ZvURFLRVFHaYQyBQqBu1S7JDZ6cJ9ruT2QefI5vgLHzePMzuiSBSAomYf6sbpCDTE57yzO3SqAdS5YOmhJIYEE1qCsmpFOsUig/FkRILxS/eZoDKClKuaak0rVDCBG5PMg2H5LTSW6m3y7O0zsnfmjkKBnYnpmSeLcxkJLZS2HmJmuQAJCuGKGEHZiQJjthdh2HexwpibXNDK2EoACblK5qVb4D9Co+t6TjW3FcHqCcvUKZTOyxIy2BsQrsigHLi1MG+cmA5Xo5ILYcf9EHAHl5XPN7aCcdAMctmYtMaW1479+ebaPIwzVrhCmaDGHE2C3M4wnOVMuKNQGMixeJrCEZqKSjCzuunIgjxFJCZJpvFVljToxRl1JFci1FQad6Jxpjujpss58CruqLDJ61nNVV5YUGy5cFIIYjPiLhOJuQZmRYUGZbivnBF2TDVlFq/cyipN7BzYal5loTjLYuigwpZWUwQNWEmxx1hl/V7z5to1GAZz4HCXp9ya0R52Vo0+4jk92W6F2JJJtwGXjenz8Ztx8PnF1s5O25t175H3xBt6E++Vt+N98va9qcf8n/4v/4//t/euF/e+98RS6q+1cx56K6NX/ANMJ5+H</latexit>
= g(λ? , μ ? ).
Thus, x? and (λ? , μ ? ) are primal and dual feasible with the same <latexit sha1_base64="jr5dqs93o2Tk7xbmd1F6Yyjuf00=">AAADvHicbVLLjtMwFHUmPIbymoElm0CFNEhVlYxAwIoiQLBgUSTKjNSUynFuMlZtJ7Kd0irqH/E17BB8DNdpNKLpOIp9fM6xfe17k1JwY8Pwt3fgX7t+4+bhrd7tO3fv3T86fvDNFJVmMGGFKPR5Qg0IrmBiuRVwXmqgMhFwlizeOf1sCdrwQn216xJmkuaKZ5xRi9T82PsQK/jBCimpSus429RxUojUrCUOm96OKBmqktoLRvekpJWSpKNUApVKpaBdiB2xWG5cd6U2Bl2iWuIQP24+BzumRFSApuYd6mbSMWhIL3WHO3KuAdSlYTtDSwoZBrQl5bo16RwvGQ6jQRAOX7gu2vRW32Njqe7Nj/rhMGxasA+iFvRJ28b48PM4LVglQVkmqDHTKCztrKbacibcW1QGSsoWNIcpQkUlmFndhLQJniKTBlmh8Vc2aNj/V9SJ3NmhptK4lOJKlyTT1Rx5lTatbPZqVnNVVhYU2x6cVSKwReCqKUi5BmbFGgFlmmPsAbugmjKLNbdzSrN3CWw3rlWlOCtS6LDCrqymSBqwknLlblm/1bypu4bDzRx58p7n3JrBZyxnNfiIiVo8a42YkqibgH0wOR2+HoZfnvdHozY3h+QReUJOSERekhH5RMZkQpj30/vl/fH++m988Be+3FoPvHbNQ7LT/OU/JGVFEQ==</latexit>

<latexit sha1_base64="RDvEjAF3wW6mt36AqV7tuc+vL8w=">AAADznicbVJdb9MwFHUWPkb52uCRl8CE1ElVlSAQ8FYEEjzw0EmUTmpK5Tg3nVXbiWxnWxVFvPKr+B+88wq/ges0mmg6R7GPzzm2r31vUghubBj+8vb8Gzdv3d6/07t77/6DhweHj76avNQMJiwXuT5NqAHBFUwstwJOCw1UJgKmyeq906fnoA3P1Re7LmAu6VLxjDNqkVocetNYwQXLpaQqreKsruIkF6lZSxzq3pYoGaqS2jNGd6SklZKko5QClVKloF2IHTE/r113rTYGXaBa4BA/bT4HO6ZElICm5h2qZtIxaEivdIc78lIDqCvDZoaWFDIMaEPKdWvSS7xkOIwGQTh85bqo7vVjgW+d0m+xsVQPYllu0HFvcXAUDsOmBbsgasERadsYU7GI05yVEpRlghozi8LCziuqLWfCvU5poKBsRZcwQ6ioBDOvmiDr4DkyaZDlGn9lg4b9f0WVyK0dKiqNSzKudGkzXc2R12mz0mZv5hVXRWlBsc3BWSkCmweuvoKUa2BWrBFQpjnGHrAzqimzWIVbpzR7F8C247osFWd5Ch1W2EurKZIGrKRcuVtW7zRvKrHhcDNH9j/wJbdm8BkLXA0+YupWx60RUxJ1E7ALJi+Gb4fhycuj0ajNzT55Qp6RPonIazIin8iYTAjzfnq/vT/eX//Ev/Br//vGuue1ax6Treb/+AeAy0w5</latexit>

objective value. Thus, they are both optimal.

Theorem (Necessity): Assume that a convex P with differentiable

objective and constraint functions satisfies Slater’s condition. If x? <latexit sha1_base64="jr5dqs93o2Tk7xbmd1F6Yyjuf00=">AAADvHicbVLLjtMwFHUmPIbymoElm0CFNEhVlYxAwIoiQLBgUSTKjNSUynFuMlZtJ7Kd0irqH/E17BB8DNdpNKLpOIp9fM6xfe17k1JwY8Pwt3fgX7t+4+bhrd7tO3fv3T86fvDNFJVmMGGFKPR5Qg0IrmBiuRVwXmqgMhFwlizeOf1sCdrwQn216xJmkuaKZ5xRi9T82PsQK/jBCimpSus429RxUojUrCUOm96OKBmqktoLRvekpJWSpKNUApVKpaBdiB2xWG5cd6U2Bl2iWuIQP24+BzumRFSApuYd6mbSMWhIL3WHO3KuAdSlYTtDSwoZBrQl5bo16RwvGQ6jQRAOX7gu2vRW32Njqe7Nj/rhMGxasA+iFvRJ28b48PM4LVglQVkmqDHTKCztrKbacibcW1QGSsoWNIcpQkUlmFndhLQJniKTBlmh8Vc2aNj/V9SJ3NmhptK4lOJKlyTT1Rx5lTatbPZqVnNVVhYU2x6cVSKwReCqKUi5BmbFGgFlmmPsAbugmjKLNbdzSrN3CWw3rlWlOCtS6LDCrqymSBqwknLlblm/1bypu4bDzRx58p7n3JrBZyxnNfiIiVo8a42YkqibgH0wOR2+HoZfnvdHozY3h+QReUJOSERekhH5RMZkQpj30/vl/fH++m988Be+3FoPvHbNQ7LT/OU/JGVFEQ==</latexit>

solves P, there is (λ? , μ ? ) such that (x? , λ? , μ ? ) satisfies the KKT <latexit sha1_base64="RDvEjAF3wW6mt36AqV7tuc+vL8w=">AAADznicbVJdb9MwFHUWPkb52uCRl8CE1ElVlSAQ8FYEEjzw0EmUTmpK5Tg3nVXbiWxnWxVFvPKr+B+88wq/ges0mmg6R7GPzzm2r31vUghubBj+8vb8Gzdv3d6/07t77/6DhweHj76avNQMJiwXuT5NqAHBFUwstwJOCw1UJgKmyeq906fnoA3P1Re7LmAu6VLxjDNqkVocetNYwQXLpaQqreKsruIkF6lZSxzq3pYoGaqS2jNGd6SklZKko5QClVKloF2IHTE/r113rTYGXaBa4BA/bT4HO6ZElICm5h2qZtIxaEivdIc78lIDqCvDZoaWFDIMaEPKdWvSS7xkOIwGQTh85bqo7vVjgW+d0m+xsVQPYllu0HFvcXAUDsOmBbsgasERadsYU7GI05yVEpRlghozi8LCziuqLWfCvU5poKBsRZcwQ6ioBDOvmiDr4DkyaZDlGn9lg4b9f0WVyK0dKiqNSzKudGkzXc2R12mz0mZv5hVXRWlBsc3BWSkCmweuvoKUa2BWrBFQpjnGHrAzqimzWIVbpzR7F8C247osFWd5Ch1W2EurKZIGrKRcuVtW7zRvKrHhcDNH9j/wJbdm8BkLXA0+YupWx60RUxJ1E7ALJi+Gb4fhycuj0ajNzT55Qp6RPonIazIin8iYTAjzfnq/vT/eX//Ev/Br//vGuue1ax6Treb/+AeAy0w5</latexit> <latexit sha1_base64="pXg7F3vq/5XxG6iZm+GSjAkK08w=">AAAD1nicbVLLjtMwFHUmPIbymoElm0CF1JGqKkEgYFcEEixYFIkyg5pSOc5Nx6rtRLYztIoqVogtX8U/8A9sYc11GkY0HUexj885tq99b1IIbmwY/vT2/EuXr1zdv9a5fuPmrdsHh3c+mLzUDMYsF7k+SagBwRWMLbcCTgoNVCYCjpPFS6cfn4E2PFfv7aqAqaRzxTPOqEVqdujNYgWfWS4lVWkVZ+sqTnKRmpXEYd3ZEiVDVVJ7yuiOlDRSkrSUUqBSqhS0C7El5mdr112ojUAXqBY4xPfrz8GWKREloKl+h6qetAwa0nPd4ZY81wDq3LCZoSWFDAPakHLVmPQcLxkOon4QDp64Llp3estPsbFU92OBb57SfzNZbtBRZ3bQDQdh3YJdEDWgS5o2qlOS5qyUoCwT1JhJFBZ2WlFtORPulUoDBWULOocJQkUlmGlVB7sOHiKTBlmu8Vc2qNn/V1SJ3NqhotK4ZONKlz7T1hx5kTYpbfZsWnFVlBYU2xyclSKweeDqLEi5BmbFCgFlmmPsATulmjKL1bh1Sr13AWw7rmWpOMtTaLHCLq2mSBqwknLlblm90LyuyJrDzRzZe8Xn3Jr+Wyx01X+NKVwcNUZMSdROwC4YPxo8H4TvHneHwyY3++QeeUB6JCJPyZC8ISMyJsz74f3yfnt//I/+F/+r/21j3fOaNXfJVvO//wVoU0+h</latexit>

conditions. Staters's edt strictly

fi E
Proof: Slater implies strong duality and solvability of D (to be
shown later!). The claim then follows from the KKT theorem.
132

Who invented it?

William Karush Harold W. Kuhn Albert W. Tucker

The KKT conditions were first named after Kuhn and Tucker, two
famous professors from Princeton, who published them in 1951.

Later it was discovered that the conditions had been stated in the
MSc thesis of the student William Karush in 1939.
133

Relevance of the KKT Conditions

The KKT conditions play an important role in optimization.
• Many algorithms for convex optimization are conceived as
methods for solving the KKT conditions.
• Sometimes it is possible to solve the KKT conditions (and thus,
the optimization problem) analytically.

1 >
Example: minimize 2 x Px + q> x + r
(P 2 Sn+ )
subject to
<latexit sha1_base64="p82EPg3BUAsMulSZubLrCTC2k7k=">AAAEN3icbVNNb9QwEE13+SjLR1s4cjFUQKuuVkkFAg5IrUCCA0iLYNtKzXblOJOtqe2ktlOyRPljnPgb3DghrvwDxtmoYtM6ij157409nplEmeDG+v7PpU73ytVr15dv9G7eun1nZXXt7p5Jc81gxFKR6oOIGhBcwchyK+Ag00BlJGA/Onnt+P0z0Ian6rOdZTCWdKp4whm1CE3Wln6ECr6yVEqq4jJMqjKMUhGbmcSl6i2QkiErqT1m9AIVNVQUtZhcIJOrGLQLsUWmZ5WbLuWGoDNkM1zCB/XjzJYoEjmgqM5DWX+0BBric97ZLXqqAdS5YP6FkhgSDGgOylkj0lO8pD8I+sQfPHNTgMoIplyVVGs6q0rh0mKhsKXkikv+DSrymISJpqwMqnK7IsVRaNOMDIut07lVbOkwbJxMHn0BZolNndsuKV5FvRAwzPn24elpTmOyMQy5woSTT5OtI7XZm6yu+wO/HuSiETTGuteMIdZ8EsYpyyUoywQ15jDwMzvGUyxnwpUhN5BRdkKncIimohLMuKyzUZFHiMQkSTW+ypIa/d+jjOTCDiWVxnUTerr+MG3OgZdxh7lNXoxLrrLcgmLzg5NcYHaIa2QSc43JEjM0KNMcYyfsmGKqLbb7win13hmwxbiKXHGWxtBChS2spggasJJy5W5Z7mpet3yN4WYO3HjDp9ya/nv8k1T/LfbIyWYjxJIE7QJcNEbbg5cD/+PT9Z2dpjbL3n3vobfhBd5zb8d75w29kcc6TzofOnud/e737q/u7+6fubSz1Pjc8xZG9+8/8PxyoA==</latexit>
Ax = b

KKT conditions: Ax? = b, <latexit sha1_base64="c3HdRUaV74vPVDFKI80hzNWjIdk=">AAAD43icbVLLjtMwFHUbHkN5zcCSTaBCGjRVlSAQsEBqBRIsWBSJMiM1ncpxbjpW/cj4MbSK+gesEFu+ihWfwg47jUY0HUexj885tq99b1owqk0U/Wm1g2vXb9zcu9W5fefuvfv7Bw++amkVgTGRTKqTFGtgVMDYUMPgpFCAecrgOF288/rxBShNpfhiVgVMOZ4LmlOCjaNmBy2WCPhGJOdYZGWSr8sklSzTK+6GdWdL5MSpHJszgnektJbStKFY5hQrMlA+xIYoL9a+u1IbgSqcWrgheVx9HjZMKbPgTNU7lNWkYVCQXeoeN+S5AhCXhs3MWTLIXUAbkq9qk5q7S0b9uBdG/Ze+i9ed4fI00Qart2kvObc4C0c1cXR+NDxNjCzChNvaE3Vm+92oH1Ut3AVxDbqobiOXn1mSSWI5CEMY1noSR4WZllgZSph/MquhwGSB5zBxUGAOelpWka/Dp47Jwlwq9wsTVuz/K8qUb+1QYq595t1Kn0vd1Dx5lTaxJn89LakorAFBNgfnloVGhr7owowqIIatHMBEURd7SM6wwsS40tw6pdq7ALId19IKSmQGDZaZpVHYkRoMx1T4W5ZDRavyrDi3mScP39M5Nbr3yVW96H1w+Vw8q40uJXEzAbtg/Lz/ph99ftEdDOrc7KFH6Ak6RDF6hQboIxqhMSKt362/7XY7CPLge/Aj+Lmxtlv1modoqwW//gHlVVJc</latexit>

Px? + q + A> μ ? = 0
✓ >
◆✓ ◆ ✓ ◆
P A x?
q
() =
<latexit sha1_base64="uh7WHjkWYpjkQAnvbLx3T9Tj6uk=">AAAESHicbVPPb9MwFM7SAaP82AZHLoYKNKRRpQjEOCBtAgkOHIpE2aS6nRznJbNmO5ntjFZR/0JOHPkzOCFuPKfRRNM5iv38fd+zn/2e40IK66Lo10bY2bxx89bW7e6du/fub+/sPvhm89JwGPFc5uYkZhak0DBywkk4KQwwFUs4js/fe/74EowVuf7q5gVMFMu0SAVnDqHT3Y2fVMN3nivFdFLRdFHROJeJnSscFt0VUnFkFXNnnK1RcUPFcYspJTKlTsD4EFtkfrnw3bXcEEyBbIEDfVx/3myJYlkCiup7qOpJS2AgueK93aIzA6CvBMsZShJIMaAlqOaNyGR4yKg/2CdR/7XvBqgUaUroRckSQmPIhK4KvAUjZovukDwjR1Pq8oJScoSTqEsB97wStB1mU2odM6imqlzaLY93bZcXF6iOW6rTnV7Uj+pG1o1BY/SCpg2xCE5pkvNSgXZcMmvHg6hwk4oZJ7j0eSktFIyfswzGaGqmwE6q+noW5CkiCUlzg792pEb/96hitbJCxZT15YWevmBsm/Pgddy4dOnBpBK6KB1ovtw4LSVxOfGVTRJhgDs5R4NxIzB2ws+YYdxh/a/sUq9dAF+Na1ZqwfMEWqh0M2cYghacYkL7U1ZHRtRvoMZwMQ/ufRCZcHb/Mz4tvf8Ri+b8eSPElAzaCVg3Ri/7b/vRl1e9w8MmN1vBo+BJsBcMgjfBYfApGAajgIcH4TTMwrPOj87vzp/O36U03Gh8HgYrbTP8ByWeeNI=</latexit>
A 0 μ? b
134

Separable Problems
Problem with a separable objective and a single constraint:
Pn
minimize f0 (x) = i=1 fi (xi )
subject to a> x = b <latexit sha1_base64="0JgnCXha4hnPRTf9p/9m5VcFi0E=">AAAEInicbVNdb9MwFE1bPkb52sYjL4EK1EpVlSAQ42FSB0jwwEORKJu0dJXt3HRmthPZzmiJ8nN44qfwhHhC4sdwnUYTzeYo9s05515fX9/QTHBjg+BPq925dv3Gza1b3dt37t67v72z+9mkuWYwZalI9RElBgRXMLXcCjjKNBBJBRzSszeOPzwHbXiqPtlVBjNJFoonnBGL0Hyn9SNS8JWlUhIVF1FSFhFNRWxWEpeyu0FKhqwk9pSRSxStKUobTC6QyVUM2qXYINPz0k1XchPQGbIZLtGj6nFmQ0RFDiiq6lBUHw2BhviCd3aDXmgAdSFYf6EkhgQTWoNyVYv0Ag8ZjMKhH4xeuClEJYUFVwXRmqzKQriyWFjaQnLFJf8Gpf/UT+ZBfznYj0wu5wXfD8sThRjvL+d8EEW1g8npF2DWt6lzISeRTTN/uU+7EWCe6/jd+XYvGAXV8C8bYW30vHpM8HrnUZyyXIKyTBBjjsMgszOMZzkTruK5gYywM7KAYzQVkWBmRXXw0n+CSOwnqcZXWb9C//coqNyIUBBpXOOgp2sF0+QceBV3nNtkb1ZwleUWFFtvnOQCi+G7nvVjrrE2YoUGYZpj7j47JZowi529sUsVOwO2mdcyV5ylMTRQYZdWEwQNWEm4cqcsDjSvurvCMJgD+2/5glsz/IA/jRq+w3Y4G9RCvJKweQGXjemz0atR8PF5bzyu72bLe+g99vpe6L30xt57b+JNPdbebe+1D9qvO987Pzu/Or/X0nar9nngbYzO339ATmpq</latexit>

The Lagrangian is also separable:

n
X n
X
L(x, μ) = fi (xi ) + μ(b a> x) = bμ + (fi (xi ) μai xi )
<latexit sha1_base64="iZ6r4a935HX583NvUMtWHznfsak=">AAAEGnicbVPLbtNAFHUSHm14tbBkM1AhJWoabASiXUQKAgkWXRSJ0EpxGsbj63TUmbE1My6JrHwKG36FFWLLhr/hjhMi4nQszxyfc+687nWUCW6s7/+p1Rs3bt66vbXdvHP33v0HO7sPP5s01wwGLBWpPouoAcEVDCy3As4yDVRGAk6jy7dOP70CbXiqPtlZBiNJJ4onnFGL1Hi39j1U8JWlUlIVF2EyL8IoFbGZSRzmzTVRMlQltReMbkjRUoqiipILVHIVg3ZbrIjp1dx112onoDNUMxzCJ+XjYMUUiRzQVN5DUX5UDBrile5wRZ5oALUyLL7QEkOCG1qQcrY06Qke0u8GHeJ3X7kumDePW9NOKPN2j4Qml+OC94L5uSLJmLemY97eR60VHdDz0KYZmbZ7ERJkf80bCkhsaxVy4Bx0zBGHmk8ubLs53tnzu37ZyCYIlmDPW7YTzOqXME5ZLkFZJqgxw8DP7Kig2nIm3EXnBjLKLukEhggVlWBGRXneOXmGTEySVOOrLCnZ/yOKSK7NUFBpXL1gpKsAU9UceZ02zG1yOCq4ynILii0WTnJBbEpcqZKYa2BWzBBQpjnunbALqimzWNBrq5RzZ8DW9zXNFWdpDBVW2KnVFEkDVlKu3CmLN5qXRV1yOJkjW+/4hFvTOcZ/RXXeYxVctv8ZMSdBNQObYPCie9T1P77c6/eXydnyHntPvZYXeK+9vvfBO/EGHqtv15/XD+tHjW+NH42fjV8La722jHnkrbXG779OJ2Qo</latexit>
i=1 i=1

The dual objective is also separable:

Pn
g(μ) = bμ + inf Pnx i=1 (fi (xi ) μai xi )
= bμ + Pi=1 inf xi (fi (xi ) μai xi )
n ⇤
= bμ f
i=1 i (μai ) <latexit sha1_base64="1k1U2xxK90Nhji6bXeFxa9Y3ux4=">AAAEgHicjVNNb9QwEE3TBUr56sKRS6Ci2oV2SRCID6lSEUhw4FAkllZqtqnjTFJrbSeynbKraH8GV/hd/BvG2VBI2gOOYo/fezMee+y44Ewb3/+14q72rly9tnZ9/cbNW7fvbPTvftV5qSiMac5zdRgTDZxJGBtmOBwWCoiIORzE03eWPzgDpVkuv5h5ARNBMslSRolBKOq7a6GEbzQXgsikCtNFFcY5T/Rc4LBYb5GCIiuIOaXkAhU3VBx3mJIjU8oElE2xQ+ZnC9tdyu2DKpAtcAgf1J81O6KYl4Ci+hyqetIRKEjOeWt36EwByHPBcoaSBFJMaAmKeSNSGW7SHwXbnj96YbsAlTFkTFZEKTJHBcVzyQahKIdbu1sxjt6TkMk0moW6FFHFdoPFsfRCDqkZeGnEBrOIDXesjkQM7VCx7NQMw/Cve8vRxqpQt/i/EF4dY6cVA12OHw8a/TAEPIVl9tHGpj/y6+ZdNILG2HSath/1V07CJKelAGkoJ1ofBX5hJhjOMMptOUsNBaFTksERmpII0JOqPtWF9wiRxEtzhb80Xo3+61HFohWhIkLbW4me9p7pLmfBy7ij0qSvJhWTRWlA0uXCack9k3v2QXgJU0ANn6NBqGKYu0dPiSLU4LNprVLHLoC285qVktE8gQ7KzcwogqAGIwgWDndZvVWsfjo1hsEsOHjPMmb09id8kXL7A9616fCPEGsSdCtw0Rg/G70e+Z+fb+7tNcVZc+47D52BEzgvnT3no7PvjB3q5u5394f7s7faG/ae9oKl1F1pfO45rdZ78xsGQYf1</latexit>

D is a scalar optimization problem and thus simple. Given μ ?, we

solve the problems inf xi (fi (xi ) μ ? ai xi ) separately to find x?i .
<latexit sha1_base64="cnxG5ssuURDXfRJpubRm8bQf2X0=">AAADvXicbVJdb9MwFHUWPsb42uCRl0CFNKSqSibQxlsRCHjgoUiUTWpK5Tg3nVV/RLazrYr6k/g1PCHBf+E6jSaazlHs43OO7Wvfm5WCWxfHv4Od8NbtO3d37+3df/Dw0eP9gyffra4MgzHTQpuzjFoQXMHYcSfgrDRAZSbgNFu89/rpBRjLtfrmliVMJZ0rXnBGHVKzg+BjquCSaSmpyuu0WNVppkVulxKH1d6GKBmqkrpzRrekrJWyrKNUApVK5WB8iB1RX6x8d6M2AlOiWuKQPm8+DzumTFSApuYd6mbSMRjIr3WPO/LcAKhrw3qGlhwKDGhNymVrMnO8ZDxI+lE8eOO7BJ2y+pFaR81svxcP4qZF2yBpQY+0bYQvP0tzzSoJyjFBrZ0kcemmNTWOM+Efo7JQUragc5ggVFSCndZNTKvoJTJ5VGiDv3JRw/6/os7kxg41ldbnFFf6LNmu5smbtEnlipNpzVVZOVBsfXBRicjpyJdTlHMDzIklAsoMx9gjdk4NZQ6LbuOUZu8S2GZcV5XiTOfQYYW7coYiacFJypW/Zf3O8KbwGg438+ThBz7nzva/YD2r/ifM1OJVa8SUJN0EbIPx0eDtIP76ujcctrnZJc/IC3JIEnJMhuQzGZExYcHP4FfwJ/gbDsMiFKFaW3eCds1TstHCy39QsUXX</latexit>

<latexit sha1_base64="KNL04GFYUT43Ux22Omm4h4E7y+E=">AAADvXicbVJdb9MwFHUWPkb52uCRl0CFNKSqSiYQ8FYEAh54KBJlk5pSOc5NZ9Ufke1sraL+JH4NT0jwX7hOo4lmcxT7+Jxj+9r3ZqXg1sXx72AvvHHz1u39O7279+4/eHhw+Oi71ZVhMGFaaHOaUQuCK5g47gSclgaozAScZMv3Xj85B2O5Vt/cuoSZpAvFC86oQ2p+GHxMFVwwLSVVeZ0WmzrNtMjtWuKw6e2IkqEqqTtj9IqUtVKWdZRKoFKpHIwPsSPq843vrtXGYEpUSxzSp83nYceUiQrQ1LxD3Uw6BgP5pe5xR14YAHVp2M7QkkOBAW1JuW5NZoGXjIfJIIqHr3yXbHqrH6l11Mz5/KAfD+OmRVdB0oI+adsYX36e5ppVEpRjglo7TeLSzWpqHGfCP0ZloaRsSRcwRaioBDurm5g20XNk8qjQBn/loob9f0WdyZ0daiqtzymu9FmyXc2T12nTyhVvZjVXZeVAse3BRSUipyNfTlHODTAn1ggoMxxjj9gZNZQ5LLqdU5q9S2C7ca0qxZnOocMKt3KGImnBScqVv2X9zvCm8BoON/Pk0Qe+4M4OvmA9q8EnzNTyRWvElCTdBFwFk+Ph22H89WV/NGpzs0+ekGfkiCTkNRmRz2RMJoQFP4NfwZ/gbzgKi1CEamvdC9o1j8lOCy/+AVVBRdk=</latexit>

<latexit sha1_base64="f7NA3/Xt+pvvGDzLRhqu3jQvshI=">AAAD8XicbVNNj9MwEE0bWJblY7tw5BKokFqpVCkCAbcikODAoUiUXakpxXEmrVXbiezJ0ijqH+GEuPKLuPFTmKRhRbPrKPb4vWdn7DcJUyks+v7vVtu9dv3gxuHNo1u379w97pzc+2yTzHCY8kQm5ixkFqTQMEWBEs5SA0yFEk7D9ZuSPz0HY0WiP2GewlyxpRax4AwJWpy08kDDN54oxXRUBPG2CMJERjZXNGyP9kjFiVUMV5xdosKaCsMGk0liMh2BKVNskMn5tuyu5CZgUmJTGoKH1VOGDVEoMyBRdQ9FNWkIDEQXfBk36KUB0BeC3YwkEcSU0A5UeS0ySzqkPxwNPH/4vOxGpETYoMVcghcIHS+KzUJsAwkx9rx4IXo07T8JVPYlsMiMxxaCkMCI5Qr7i07XH/pV8y4HozroOnWbkFlfgyjhmQKNXDJrZyM/xXnBDAouy/vLLKSMr9kSZhRqpsDOi+oYW+8xIZEXJ4ZejV6F/r+iCNXeDgVTtiwDWlkaa5tcCV7FzTKMX84LodMMQfPdh+NMeph4ZQV6kTDAUeYUMG4E5e7xFTOMI9Xp3leqvVPg+3ltMi14EkEDlbhBwwi0gIqRGXTK4rURVa1WGG1Wgr23YinQDj7QL6AH78jcdf+fkDwZNR24HEyfDl8N/Y/PuuNxbc6h88B55PSckfPCGTvvnYkzdXjrT/ugfdzuuOh+d3+4P3fSdqtec9/Za+6vv/f8V/E=</latexit>
135

Main Take-Away Points

• KKT conditions: primal feasibility; dual feasibility;

complementary slackness; stationarity
• KKT theorems: every KKT point of a convex problem
constitutes a primal-dual solution pair; every (primal)
solution of a convex problem corresponds to a KKT point