CS329 Machine Learning Quiz 2

Posted on 2026-01-11 In CS

Quiz 2 for ML, ML and MAP over Guassian Mixture and Logistic Regression

Before Quiz

$$
\begin{align*}
&-(a_0-1)\ln\pi - (b_0-1)\ln(1-\pi) \quad &\text{prior of }\pi\
&+\frac 1 2(\mu_1-m_{10})^\text T \Sigma_{10}^{-1}(\mu_1-m_{10}) \quad &\text{prior of }\mu_1\
&+\frac 1 2(\mu_2-m_{20})^\text T \Sigma_{20}^{-1}(\mu_2-m_{20})\quad &\text{prior of }\mu_2\
&+\sum\limits_{n=1}^N[t_n\ln\left(\pi\mathcal N (x\vert\mu_1,\Sigma_1)\right)+(1-t_n)\ln((1-\pi)\mathcal N (x\vert\mu_2,\Sigma_2))]\quad&\text{Likelihood}
\end{align*}
$$

Question 1 Generative Gaussian Mixture

Question 1.1

$p(x)=\pi\mathcal N(x\vert\mu_1,\Sigma_1)+(1-\pi)\mathcal N(x\vert\mu_2,\Sigma_2)$

Given $\mathbf x=[x_1,\dots,x_N]$, $\mathbf t=[t_1,\dots,t_N]$

What’s the ML estimation of $\mu_1,\Sigma_1,\mu_2,\Sigma_2,\pi$?

Solution 1.1

Derive log-likelihood function w.r.t. $\pi$,
$$
\frac{\partial}{\partial \pi}L=\frac{\partial}{\partial \pi}\sum\limits_{n=1}^N {t_n\ln\pi+(1-t_n)\ln(1-\pi)}=\frac{N_1}{\pi}+\frac{N_2}{\pi-1}=0
$$
Derive log-likelihood function w.r.t. $\mu_i$,
$$
\begin{align*}
\frac{\partial}{\partial \mu_1}L &= \frac{\partial}{\partial \mu_1} \sum\limits_{n=1}^N t_n\ln\mathcal N(x_n\vert,\mu_1,\Sigma_1)\
&=\frac{\partial}{\partial \mu_1} -\frac 1 2 \sum\limits_{n=1}^N t_n(x_n-\mu_1)^\text T \Sigma_1^{-1}(x_n-\mu_1)\
&=0\
\frac{\partial}{\partial \mu_2}L &= \frac{\partial}{\partial \mu_2} \sum\limits_{n=1}^N (1-t_n)\ln\mathcal N(x_n\vert,\mu_2,\Sigma_2)\
&=\frac{\partial}{\partial \mu_2} -\frac 1 2 \sum\limits_{n=1}^N (1-t_n)(x_n-\mu_2)^\text T \Sigma_1^{-1}(x_n-\mu_2)\
&=0
\end{align*}
$$
Derive log-likelihood function w.r.t. $\Sigma_i$,
$$
\begin{align*}
\frac{\partial}{\partial \Sigma_1} L &= \frac{\partial}{\partial \Sigma_1} \left(-\frac 1 2 \sum\limits_{n=1}^N t_n\ln |\Sigma_1|-\frac1 2 \sum\limits_{n=1}^N t_n(x_n-\mu_1)^\text T \Sigma_1^{-1}(x_n-\mu_1)\right)=0\
\frac{\partial}{\partial \Sigma_2} L &= \frac{\partial}{\partial \Sigma_2} \left(-\frac 1 2 \sum\limits_{n=1}^N (1-t_n)\ln |\Sigma_2|-\frac1 2 \sum\limits_{n=1}^N (1-t_n)(x_n-\mu_2)^\text T \Sigma_2^{-1}(x_n-\mu_2)\right)=0
\end{align*}
$$
Hence we have
$$
\begin{align*}
\pi_\text{ML} &= \frac 1 N \sum\limits_{n=1}^N t_n =\frac{N_1}{N}=\frac{N_1}{N_1+N_2}\
\mu_{1\text{ML}} &= \frac 1 {N_1}\sum\limits_{n=1}^Nt_n x_n\
\mu_{2\text{ML}} &= \frac 1 {N_2}\sum\limits_{n=1}^N(1-t_n) x_n\
\Sigma_{1\text{ML}} &=\frac 1 {N_1}\sum\limits_{x_n\in\mathcal C_1} (x_n-\mu_{1\text{ML}})(x_n-\mu_{1\text{ML}})^\text T\
\Sigma_{2\text{ML}} &=\frac 1 {N_2}\sum\limits_{x_n\in\mathcal C_2} (x_n-\mu_{2\text{ML}})(x_n-\mu_{2\text{ML}})^\text T
\end{align*}
$$

where

$N_1$ is the number of data points in class 1,
$N_2$ is the number of data points in class 2.

Question 1.2

$\pi\sim beta(a_0,b_0), p(\mu_i) = \mathcal N(m_{i0},\Sigma_{i0}), i=1,2$

Given $\mathbf x=[x_1,\dots,x_N]$, $\mathbf t=[t_1,\dots,t_N]$

What’s the MAP estimation of $\mu_1,\Sigma_1,\mu_2,\Sigma_2,\pi$?

Solution 1.2

Posterior of $\pi$:
$$
\begin{align*}
p(\pi|\mathbf{t}) &\propto p(\mathbf{t}|\pi) \cdot p(\pi)\
&= \pi^{N_1} \cdot (1 - \pi)^{N_2} \cdot \pi^{a_0-1} \cdot (1-\pi)^{b_0-1}\
&= \pi^{N_1 + a_0 - 1} \cdot (1 - \pi)^{N_2 + b_0 - 1}
\end{align*}
$$
By using the property of product of Gaussian distributions, we obtain the MAP estimation of $\mu_i$ and $\Sigma_i$
$$
\begin{align*}
\pi_\text{MAP} &= \frac{N_1+a_{0}-1}{N+a_0+b_0-2}=\frac{N_1+a_0-1}{N_1+N_2+a_0+b_0-2}\
\Sigma_{1\text{MAP}} &=({\Sigma_{1\text{ML}}}^{-1}+{\Sigma_{10}}^{-1})^{-1}\
\Sigma_{2\text{MAP}} &=({\Sigma_{2\text{ML}}}^{-1}+{\Sigma_{20}}^{-1})^{-1}\
\mu_{1\text{MAP}} &= \Sigma_{1\text{MAP}}({\Sigma_{1\text{ML}}}^{-1}\mu_{1\text{ML}}+{\Sigma_{10}}^{-1}m_{10})\
\mu_{2\text{MAP}} &= \Sigma_{2\text{MAP}}({\Sigma_{2\text{ML}}}^{-1}\mu_{2\text{ML}}+{\Sigma_{20}}^{-1}m_{20})\
\end{align*}
$$

Question 1.3

What’s $p(\mathcal C_1\vert x)$ for ML and MAP models respectively?

Solution 1.3

By Bayes’ Theorem,
$$
\begin{align*}
p_\text{ML}(\mathcal C_1\vert x)&=\frac{p_\text{ML}(x,\mathcal C_1)}{p_\text{ML}(x)}=\frac{\pi_\text{ML}\mathcal N(x\vert\mu_\text{1ML},\Sigma_\text{1ML})}{\pi_\text{ML}\mathcal N(x\vert\mu_\text{1ML},\Sigma_\text{1ML})+(1-\pi_\text{ML})\mathcal N(x\vert\mu_\text{2ML},\Sigma_\text{2ML})}\
p_\text{MAP}(\mathcal C_1\vert x)&=\frac{p_\text{MAP}(x,\mathcal C_1)}{p_\text{MAP}(x)}=\frac{\pi_\text{MAP}\mathcal N(x\vert\mu_\text{1MAP},\Sigma_\text{1MAP})}{\pi_\text{MAP}\mathcal N(x\vert\mu_\text{1MAP},\Sigma_\text{1MAP})+(1-\pi_\text{MAP})\mathcal N(x\vert\mu_\text{2MAP},\Sigma_\text{2MAP})}
\end{align*}
$$

Question 2 Discriminative Logistic Regression

Question 2.1

$y=\sigma(w^\text T\phi(x))$

Given $\mathbf x=[x_1,\dots,x_N]$, $\mathbf t=[t_1,\dots,t_N]$

What’s the ML estimation of $q(w)$?

Solution 2.1

By Gauss-Newton iteration: $w^\text{new}=w^\text{old}-H^{-1}\nabla E(w)$, we obtain $w_\text{ML}$,

where
$$
\begin{align*}
E(w) &= -\sum\limits_{n=1}^N [t_n\ln y_n + (1-t_n)\ln(1-y_n)]\
\nabla E(w) &= \sum\limits_{n=1}^N (y_n-t_n)\phi(x_n)\
\text{step size}\quad H &= \nabla^2 E(w) = \sum\limits_{n=1}^N y_n(1-y_n)\phi(x_n)\phi(x_n)^\text T
\end{align*}
$$
Hence
$$
q(w) = \mathcal N(w\vert w_\text{ML}, H^{-1})
$$

Question 2.2

$y=\sigma(w^\text T\phi(x))$, $p(w)\sim\mathcal N(m_0,\Sigma_0)$

Given $\mathbf x=[x_1,\dots,x_N]$, $\mathbf t=[t_1,\dots,t_N]$

What’s the MAP estimation of $q(w)$?

Solution 2.2

By Gauss-Newton iteration: $w^\text{new}=w^\text{old}-H^{-1}\nabla E(w)$, we obtain $w_\text{ML}$,

where
$$
\begin{align*}
E(w) &= \frac{1}{2} (w-m_0)^\text T\Sigma_0^{-1}(w-m_0)-\sum\limits_{n=1}^N [t_n\ln y_n + (1-t_n)\ln(1-y_n)]\
\nabla E(w) &=\Sigma_0^{-1}(w-m_0)+ \sum\limits_{n=1}^N (y_n-t_n)\phi(x_n)\
\text{step size}\quad H &= \nabla^2 E(w) = \Sigma_0^{-1}+\sum\limits_{n=1}^N y_n(1-y_n)\phi(x_n)\phi(x_n)^\text T
\end{align*}
$$
Hence
$$
q(w) = \mathcal N(w\vert w_\text{MAP}, H^{-1})
$$

Question 2.3

What’s $p(t\vert y(w,x))$ for ML and MAP estimation, respectively?

Solution 2.3

Probability of $t=1$ w.r.t. ML and MAP estimations
$$
\begin{align*}
p_\text{ML}(t=1\vert y(w_\text{ML},x)) &= \sigma(w_\text{ML}^\text T\phi(x)) = \frac{1}{1+\exp(-w_\text{ML}^\text T\phi(x))}\
p_\text{MAP}(t=1\vert y(w_\text{MAP},x)) &= \sigma(w_\text{MAP}^\text T\phi(x))= \frac{1}{1+\exp(-w_\text{MAP}^\text T\phi(x))}\
\end{align*}
$$