Gaussian mixture model

This is the equation

(1) $\begin{equation*}\begin{matrix}p(x | \theta) = \sum_{n=1}^K\pi_{k}\mathcal{N}(\mu, \Sigma) \\\\0 \leq \pi_k \leq 1, \quad \sum_{n=1}^K\pi_k = 1\end{matrix}\end{equation*}$

Source: http://www.oranlooney.com/post/ml-from-scratch-part-5-gmm/

This is the parameters we are going to learn

(2) $\begin{equation*}\theta = \{\mu_k, \Sigma_k, \pi_k: k=1, …, K\}\end{equation*}$

Algorithm

Step 1: Initialization

$\mu$ = mean generated from K-means or randomly selected data point.

$\pi = \frac{1}{K}$ , the equal probability

$\Sigma$ = covariance of this dataset

Step 2: E-Step (get responsibility)

(3) $\begin{align*}r_{nk} = \frac{\pi_k \mathcal{N}(x_n | \mu_k, \Sigma_k)}{\sum_{j=1}^K\pi_j\mathcal{N}(x_n | \mu_j, \Sigma_j)} \\\sum_{j=1}^Kr_{nk} = 1, r_{nk} \geq 0\end{align*}$

Step 3: M-Step (update $\pi$ , $\mu$ , $\Sigma$ )

(4) $\begin{align*}\mu_k & = \frac{1}{N_k}\sum_{n=1}^Nr_{nk}x_n \\\Sigma_k & = \frac{1}{N_k}\sum_{n=1}^Nr_{nk}(x_n - \mu_n)(x_n - \mu_n)^T \\\pi_k & = \frac{N_k}{N}\end{align*}$

Where $\mu_k$ is the weighted mean, $\Sigma_k$ is the weighted variance, $\pi_k$ is the normalized probability.

Share this post:

Related posts:

Leave a Reply