Maximal Likehood Estimate (MLE)
Here we change to another topic, the probability field. We know a lot of prob. distributions, like gassuan, possion, etc.. Now the setting is: you got plenty data samples, and you tell me what is the dist. of these samples thus I can use the dist. to predict other unknow samples.
So, now you have the data set:
$$ D = [ x_1, x_2, x_3, \dots, x_N ]$$
And bunch parameters that we dont know:
$$\theta=[ \theta_1,\theta_2,…\theta_k ]$$
Since now we have the dataset samples, we assume they all from the same dist sharing the same parameters.
$$ P(D,\theta)=\prod_{i=1}^{N} P(x_i;\theta) $$
In probability statistics, the joint probability of observed samples is called likelihood, which is usually represented by the symbol and sometimes also called likelihood function.
$$ L(D,\theta)=\prod_{i=1}^{N} P(x_i;\theta) $$
We want to max the likehood function then we can find these parameters:
$$ \hat{\theta}=argmaxL(D,\theta)=argmax\prod_{i=1}^{N} P(x_i;\theta) $$
And we want to avoid multiplying multiple p results in the product approaching 0. So, do the log:
$$l(\theta,D)=log(L(D,\theta))$$
$$\hat{\theta}=argmax l(D,\theta)=argmax\sum_{i=1}^{N} log(P(x_i;\theta))$$
MLE in categorical distribution
Now, there K classes, categorical distribution:
$$ P(x=c_k;\theta)=\theta_k $$
Then:
$$P(x;\theta)=\prod_{k=1}^{K} \theta_k^{1(x,x_k)}$$
Also we have to:
$$\sum_i \theta_i=1$$
Thus, recall the MLE we got:
$$L(\theta;D)=\prod_{i=1}^{N} P(x;\theta)=\prod_{i=1}^{N}\prod_{k=1}^{K} \theta_k^{1(x_i,x_k)}$$
$$L(\theta;D)=\prod_{k=1}^{K} \theta_k^{n_k}$$
$n_k$ is the number of class k shown in the D. Thus, we can get log-likehood function:
$$l(\theta;D)=\sum_{k=1}^{K} n_klog(\theta_k)=\sum_{k=1}^{K} n_klog(P(x=c_k;\theta))$$
Then we can use Lagrange multiplier to find the solution, cuz this is a constraint optimatizion problem: $$\hat{\theta}=argmax \sum_{k=1}^{K}n_klog(P(x=c_k;\theta))$$
$$La=\sum_{k=1}^{K}n_klog(P(x=c_k;\theta))+\lambda (\sum_{k=1}^{K}\theta_k-1)$$
$$La=\sum_{k=1}^{K}n_klog(\theta_k)+\lambda (\sum_{k=1}^{K}\theta_k-1)$$ Set the partial derivative equal to 0:
$$\frac{\partial La}{\partial \theta _k} =\frac{n_k}{\theta_k} + \lambda =0$$
$$\frac{\partial La}{\partial \lambda}=\theta_1+\theta_2+…+\theta_K-1=0$$
Thus we got:
$$\Rightarrow \lambda=-N$$ $$\Rightarrow \hat{ \theta_k}=\frac{n_k}{N}$$
Back to likehood function: $$l(\theta;D)\propto \frac{l(\theta;D)}{N} =\sum_{k=1}^{K} \hat{\theta_k}log(\theta_k)$$ If we multiply it with -1, we can get the cross-entropy: About the entropy, you can see htttp://asdadada
$$ H(p,q)=-\sum_{j=1} p_j log(q_j) $$ Here we go:
$$ -l(\theta;D)\propto -\frac{l(\theta;D)}{N} =-\sum_{k=1}^{K} \hat{\theta_k}log(\theta_k)=H(\theta,\hat{\theta}) $$
So, if we wanna maximize the log-likehood function it means we are minimizing the Corss-Entopy between the two distributions. Also it mean decreasing the KL distance between the two distributions. That make sense!
MLE in gaussian distribution
Now, assume the difference between $X$ and $\hat {X}$ fit the gaussian distribution $\mathcal{N}(0,\sigma)$ . Thus $X \sim \mathcal{N}(\hat{X},\sigma)$ $$ P(x;\theta)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{}{2\sigma}} $$ Thus, recall the MLE we got: $$ L(\theta;D)=\prod_{i=1}^{N} P(x_i;\theta)=\prod_{i=1}^{N} \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(X-\hat{X})^2}{2\sigma^2}} $$
Thus, we can get log-likehood function: $$ l(\theta;D)=\sum_{i=1}^{N}log\frac{1}{\sqrt{2\pi\sigma^2}}-\frac{(X-\hat{X})^2}{2\sigma^2} $$
$$\hat{\theta}=argmax \sum_{k=1}^{K}n_klog(P(x=c_k;\theta)) $$
$$La=\sum_{k=1}^{K}n_klog(P(x=c_k;\theta))+\lambda (\sum_{k=1}^{K}\theta_k-1) $$
$$La=\sum_{k=1}^{K}n_klog(\theta_k)+\lambda (\sum_{k=1}^{K}\theta_k-1) $$
