CS329 Machine Learning Quiz 3

Before Quiz

1. Learning

   $p(\theta |\mathcal{D})$

   • $p(\mathcal{D}|\theta )p(\theta )$ closed-form solution
   • $L(\theta )$
     • $b={\mathrm{\nabla }}_{\theta }L(\theta )$
     • $H={\mathrm{\nabla }}_{\theta }^{2}L(\theta )$

   ${\theta }^{+}\leftarrow \theta -H^{-1}b$

   $p(\theta |\mathcal{D})=\mathcal{N}({\theta }_{\text{MAP}},H_{\text{MAP}}^{-1})$

2. Prediction

   $$p(t_{N+1}|x_{N+1},\mathcal{D})=\int p(t_{N+1},\theta |x_{N+1},\mathcal{D})\text{d}\theta =\int p(t_{N+1}|x_{N+1},\theta )p(\theta |\mathcal{D})\text{d}\theta$$

   • $t_{N+1}=y(x_{N+1},\theta )+v$, $v\sim \mathcal{N}(0,{\beta }^{-1})$

     $p(t_{N+1}|x_{N+1},\mathcal{D})=\mathcal{N}(y(x_{N+1},{\theta }_{\text{MAP}})+{\overline{q}}_{\text{MAP}}^{\text{T}}{H}_{\text{MAP}}^{-1}{\overline{q}}_{\text{MAP}})$

   • $y(x_{N+1},\theta )=\delta ?({\mathrm{\Phi }}^{\text{T}}(x_{N+1})\theta )$

     $p(t_{N+1}|x_{N+1},\mathcal{D})=p(t_{N+1}|x_{N+1},{\theta }_{\text{MAP}})=(y(x_{N+1},{\theta }_{\text{MAP}}){)}^{t_{N+1}}(1-y(x_{N+1},{\theta }_{\text{MAP}}){)}^{1-t_{N+1}}$

3. Evaluation

   $$\begin{array}{rl}p(\mathcal{D})& =\int p(\mathcal{D}|\theta )p(\theta )\text{d}\theta \\ -\ln p(\mathcal{D})& =\int -\ln p(\mathcal{D}|\theta )-\ln p(\theta )\text{d}\theta \\ & =-\ln p(\mathcal{D}|{\theta }_{\text{MAP}})-\ln p({\theta }_{\text{MAP}})+\int \frac{1}{2}(\theta -{\theta }_{\text{MAP}}{)}^{\text{T}}{H}_{\text{MAP}}^{-1}(\theta -{\theta }_{\text{MAP}})\text{d}\theta \\ & =-\ln p(\mathcal{D}|{\theta }_{\text{MAP}})-\ln p({\theta }_{\text{MAP}})-\frac{M}{2}\ln (2\pi )+\frac{M}{2}\ln |H_{\text{MAP}}|\end{array}$$

---

Question 1 Neural Networks without Prior

Question 1.1

What are the gradients of $\frac{\partial y_k}{\partial w_{kj}}$, $\frac{\partial y_k}{\partial w_{ji}}$ for regression and classification, respectively?

Solution 1.1

1. Regression

   $$\begin{array}{rl}{y}_k& =a_k\\ \frac{\partial y_k}{\partial w_{kj}}& =\frac{\partial a_k}{\partial w_{kj}}=z_j\\ z_j& =h(a_j),\frac{\partial a_j}{\partial w_{ji}}=z_i\\ \frac{\partial y_k}{\partial w_{ji}}& =\frac{\partial a_k}{\partial z_j}\frac{\partial z_j}{\partial a_j}\frac{\partial a_j}{\partial w_{ij}}=w_{kj}{h}^{\prime }(a_j)z_i\end{array}$$

2. Classification

   $$\begin{array}{rl}{y}_k& =\sigma (a_k)\\ \frac{\partial y_k}{\partial w_{kj}}& =\frac{\partial y_k}{\partial a_k}\frac{\partial a_k}{\partial w_{kj}}={\sigma }^{\prime }(a_k)z_j=y_k(1-y_k)z_j\\ z_j& =h(a_j),\frac{\partial a_j}{\partial w_{ji}}=z_i\\ \frac{\partial y_k}{\partial w_{ji}}& =\frac{\partial y_k}{\partial a_k}\frac{\partial a_k}{\partial z_j}\frac{\partial z_j}{\partial a_j}\frac{\partial a_j}{\partial w_{ij}}=y_k(1-y_k)w_{kj}{h}^{\prime }(a_j)z_i\end{array}$$

Question 1.2

What are the gradients of $\frac{\partial E_n}{\partial w_{kj}}$, $\frac{\partial E_n}{\partial w_{ji}}$ for regression and classification, respectively?

Solution 1.2

$$\begin{array}{rl}{\delta }_k& \equiv y_k-t_k\\ {\delta }_j& \equiv \sum _{k=1}^K\frac{\partial E_n}{\partial a_k}\frac{\partial a_k}{\partial a_j}=h^{\prime }(a_j)\sum _{k=1}^K{w}_{kj}{\delta }_k\end{array}$$

1. Regression

   $$\begin{array}{rl}{E}_n& =\frac{1}{2}\sum _{k=1}^K(y_k-t_k{)}^2\\ \frac{\partial E_n}{\partial w_{kj}}& =\frac{\partial E_n}{\partial y_k}\frac{\partial y_k}{\partial w_{kj}}=(y_k-t_k)z_j={\delta }_k{z}_j\\ \frac{\partial E_n}{\partial w_{ji}}& =\frac{\partial E_n}{\partial a_j}\frac{\partial a_j}{\partial w_{kj}}=h^{\prime }(a_j)\sum _{k=1}^N{w}_{kj}{\delta }_k{z}_i={\delta }_j{z}_i\end{array}$$

2. Classification

   $$\begin{array}{rl}{E}_n& =-\sum _{k=1}^K{t}_k\ln y_k+(1-t_k)\ln (1-y_k)\\ \frac{\partial E_n}{\partial w_{kj}}& =\frac{\partial E_n}{\partial y_k}\frac{\partial y_k}{\partial a_k}\frac{\partial a_k}{\partial w_{kj}}=(y_k-t_k)z_j={\delta }_k{z}_j\\ \frac{\partial E_n}{\partial w_{ji}}& =\frac{\partial E_n}{\partial a_j}\frac{\partial a_j}{\partial w_{kj}}=h^{\prime }(a_j)\sum _{k=1}^N{w}_{kj}{\delta }_k{z}_i={\delta }_j{z}_i\end{array}$$

Question 1.3

What’s the gradients of $\frac{\partial y_k}{\partial z_i}$ for regression and classification, respectively?

Solution 1.3

1. Regression

   $$\begin{array}{r}\frac{\partial y_k}{\partial z_i}=\frac{\partial y_k}{\partial a_k}\frac{\partial a_k}{\partial z_j}\frac{\partial z_j}{\partial a_j}\frac{\partial a_j}{\partial z_i}=w_{kj}{h}^{\prime }(a_j)w_{ji}\end{array}$$

2. Classification

   $$\begin{array}{r}\frac{\partial y_k}{\partial z_i}=\frac{\partial y_k}{\partial a_k}\frac{\partial a_k}{\partial z_j}\frac{\partial z_j}{\partial a_j}\frac{\partial a_j}{\partial z_i}=y_k(1-y_k)w_{kj}{h}^{\prime }(a_j)w_{ji}\end{array}$$

Question 2 Neural Networks with Prior

If the prior of $w\sim \mathcal{N}(m_0,{\mathrm{\Sigma }}_0^{-1})$ for both regression and classification, then

Question 2.1

What are the MAP solutions of $w,p(w|\mathcal{D})$ for both cases?

Solution 2.1

By iterating $w^{\text{new}}=w^{\text{old}}-A^{-1}\mathrm{\nabla }E(w)$, we obtain $w_{\text{MAP}}$.

1. Regression

   $$\begin{array}{rl}E(w)& =-\ln p(w|\mathbf{t})=\frac{\alpha }{2}{w}^{\text{T}}w+\frac{\beta }{2}\sum _{n=1}^N[y(x_n,w)-t_n{]}^2+C\\ \mathrm{\nabla }E(w)& =\alpha w+\beta \sum _{n=1}^N(y_n-t_n){\mathbf{g}}_n\\ \mathbf{g}& ={\mathrm{\nabla }}_{w}y(\mathbf{x},w){|}_{w=w_{\text{MAP}}}\\ A& ={\mathrm{\nabla }}^{2}E(w)=\alpha \mathbf{I}+\beta \mathbf{H}\end{array}$$

2. Classification

   $$\begin{array}{rl}E(w)& =-\ln p(w|\mathbf{t})=\frac{\alpha }{2}{w}^{\text{T}}w-\sum _{n=1}^N[t_n\ln y_n+(1-t_n)\ln (1-y_n)]\\ \mathrm{\nabla }E(w)& =\alpha w+\sum _{n=1}^N(y_n-t_n){\mathbf{g}}_n\\ \mathbf{g}& ={\mathrm{\nabla }}_{w}y(\mathbf{x},w){|}_{w=w_{\text{MAP}}}\\ A& ={\mathrm{\nabla }}^{2}E(w)=\alpha \mathbf{I}+\mathbf{H}\end{array}$$

where $\mathbf{H}$ is the Hessian matrix of the sum of error function.

Hence we have $p(w_{\text{MAP}}|\mathcal{D})=\mathcal{N}(w|w_{\text{MAP}},A^{-1})$.

Question 2.2

What are the predictive distributions of a new data input $x_{N+1}$ and label $t_{N+1}$ for both cases?

Solution 2.2

1. Regression

   $$p(t_{N+1}|x_{N+1},\mathcal{D})=\mathcal{N}(y(x,w_{\text{MAP}}),{\beta }^{-1}+g^{\text{T}}Ag)$$$$p(t_{N+1}|x_{N+1},\mathcal{D})=\mathcal{N}(y(x_{N+1},{\theta }_{\text{MAP}})+{\overline{q}}_{\text{MAP}}^{\text{T}}{H}_{\text{MAP}}^{-1}{\overline{q}}_{\text{MAP}})$$

1. Classification

   $$p(t_{N+1}|x_{N+1},\mathcal{D})=\sigma \left(\kappa \left({\sigma }_a^2\right)a_{MAP}\right)$$

$$p(t_{N+1}|x_{N+1},\mathcal{D})=p(t_{N+1}|x_{N+1},{\theta }_{\text{MAP}})=(y(x_{N+1},{\theta }_{\text{MAP}}){)}^{t_{N+1}}(1-y(x_{N+1},{\theta }_{\text{MAP}}){)}^{1-t_{N+1}}$$