VAD on Junfu's Blog

Max Likehood Estimate(MLE)

Mon, 16 Jun 2025 02:46:20 +0800

Maximal Likehood Estimate (MLE)

Here we change to another topic, the probability field. We know a lot of prob. distributions, like gassuan, possion, etc.. Now the setting is: you got plenty data samples, and you tell me what is the dist. of these samples thus I can use the dist. to predict other unknow samples.

So, now you have the data set:

$$ D = [ x_1, x_2, x_3, \dots, x_N ]$$

And bunch parameters that we dont know:

$$\theta=[ \theta_1,\theta_2,…\theta_k ]$$

Since now we have the dataset samples, we assume they all from the same dist sharing the same parameters.

$$ P(D,\theta)=\prod_{i=1}^{N} P(x_i;\theta) $$

In probability statistics, the joint probability of observed samples is called likelihood, which is usually represented by the symbol and sometimes also called likelihood function.

$$ L(D,\theta)=\prod_{i=1}^{N} P(x_i;\theta) $$

We want to max the likehood function then we can find these parameters:

$$ \hat{\theta}=argmaxL(D,\theta)=argmax\prod_{i=1}^{N} P(x_i;\theta) $$

And we want to avoid multiplying multiple p results in the product approaching 0. So, do the log:

$$l(\theta,D)=log(L(D,\theta))$$

$$\hat{\theta}=argmax l(D,\theta)=argmax\sum_{i=1}^{N} log(P(x_i;\theta))$$

MLE in categorical distribution

Now, there K classes, categorical distribution:

$$ P(x=c_k;\theta)=\theta_k $$

Then:

$$P(x;\theta)=\prod_{k=1}^{K} \theta_k^{1(x,x_k)}$$

Also we have to:

$$\sum_i \theta_i=1$$

Thus, recall the MLE we got:

$$L(\theta;D)=\prod_{i=1}^{N} P(x;\theta)=\prod_{i=1}^{N}\prod_{k=1}^{K} \theta_k^{1(x_i,x_k)}$$

$$L(\theta;D)=\prod_{k=1}^{K} \theta_k^{n_k}$$

$n_k$ is the number of class k shown in the D. Thus, we can get log-likehood function:

$$l(\theta;D)=\sum_{k=1}^{K} n_klog(\theta_k)=\sum_{k=1}^{K} n_klog(P(x=c_k;\theta))$$

Then we can use Lagrange multiplier to find the solution, cuz this is a constraint optimatizion problem: $$\hat{\theta}=argmax \sum_{k=1}^{K}n_klog(P(x=c_k;\theta))$$

$$La=\sum_{k=1}^{K}n_klog(P(x=c_k;\theta))+\lambda (\sum_{k=1}^{K}\theta_k-1)$$

$$La=\sum_{k=1}^{K}n_klog(\theta_k)+\lambda (\sum_{k=1}^{K}\theta_k-1)$$ Set the partial derivative equal to 0:

$$\frac{\partial La}{\partial \theta _k} =\frac{n_k}{\theta_k} + \lambda =0$$

$$\frac{\partial La}{\partial \lambda}=\theta_1+\theta_2+…+\theta_K-1=0$$

Thus we got:

$$\Rightarrow \lambda=-N$$ $$\Rightarrow \hat{ \theta_k}=\frac{n_k}{N}$$

Back to likehood function: $$l(\theta;D)\propto \frac{l(\theta;D)}{N} =\sum_{k=1}^{K} \hat{\theta_k}log(\theta_k)$$ If we multiply it with -1, we can get the cross-entropy: About the entropy, you can see htttp://asdadada

$$ H(p,q)=-\sum_{j=1} p_j log(q_j) $$ Here we go:

$$ -l(\theta;D)\propto -\frac{l(\theta;D)}{N} =-\sum_{k=1}^{K} \hat{\theta_k}log(\theta_k)=H(\theta,\hat{\theta}) $$

So, if we wanna maximize the log-likehood function it means we are minimizing the Corss-Entopy between the two distributions. Also it mean decreasing the KL distance between the two distributions. That make sense!

MLE in gaussian distribution

Now, assume the difference between $X$ and $\hat {X}$ fit the gaussian distribution $\mathcal{N}(0,\sigma)$ . Thus $X \sim \mathcal{N}(\hat{X},\sigma)$ $$ P(x;\theta)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{}{2\sigma}} $$ Thus, recall the MLE we got: $$ L(\theta;D)=\prod_{i=1}^{N} P(x_i;\theta)=\prod_{i=1}^{N} \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(X-\hat{X})^2}{2\sigma^2}} $$

Thus, we can get log-likehood function: $$ l(\theta;D)=\sum_{i=1}^{N}log\frac{1}{\sqrt{2\pi\sigma^2}}-\frac{(X-\hat{X})^2}{2\sigma^2} $$

$$\hat{\theta}=argmax \sum_{k=1}^{K}n_klog(P(x=c_k;\theta)) $$

$$La=\sum_{k=1}^{K}n_klog(P(x=c_k;\theta))+\lambda (\sum_{k=1}^{K}\theta_k-1) $$

$$La=\sum_{k=1}^{K}n_klog(\theta_k)+\lambda (\sum_{k=1}^{K}\theta_k-1) $$

Periodic Solids

Mon, 16 Jun 2025 02:46:20 +0800

Structures of crystals

A crystal is an ordered state of matter in which the positions of the nuclei (and consequently all properties) are repeated periodically in space.

Crystal structure = Bravais lattice + basis

Paper read: Video Anomaly Detection and Explanation via Large Language Models

Mon, 01 Jan 0001 00:00:00 +0000

Research on Anomaly Detection Methods Based on Diffusion Models

link: https://www.arxiv.org/pdf/2401.05702

Quick Shot

Gap: 传统异常检测算法基于异常评分的方法，其阈值设定的复杂度高，检测结果可解释性低。

**idea：**用VLLM嵌入到VAD任务。设计了一个新的模块：长期上下文模块用于弥补VLLM在长程作用不足。设计了三阶段的训练方法。

contribution: 点数涨了3~4%，还可以为异常输出文本解释原因

challenges:

VLLM上下文建模能力低
VAD数据集基本都是弱监督，只给你一个0-1的异常标签，没有文本指令数据ft

solutions：

设计了一个 LongTerm Context (LTC)模块：首先，我们将视频分割成多个片段，并使用 Video-LLaMA 的视频编码器 (VE) 提取每个片段的特征。VADor 以这些特征作为输入，输出每个片段的异常分数。根据topk排序异常分数，把相应片段的特征堆叠为一个列表。给定下一个片段的原始特征，用CA和ADD整合当前正常和异常的特征列表。
设计了一个3阶段训练：
1. 第一阶段：先训练一个VADor，检测异常分数。这样可以构建一个新的VAD数据集。
2. 第二阶段：用VADor和LTC进行联合训练
3. 第三阶段：基于上述数据集，我们手动编写简单的文本模板（见第 3.1 节）来生成指令调整数据，然后使用这些数据仅训练 Video-LLaMA 的投影层。

Method

LTC：

具体来说，我们收集片段级的 K 个异常分数最低（最高）的 VE 特征，并将它们堆叠成一个正常（异常）列表。我们将正常列表表示为 N将异常列表表示为 A 。这两个列表在线更新，每个新片段都会根据其异常分数立即进行评估，以更新列表（或不更新）。此外，我们在 LTC 模块中引入了交叉注意力机制，将两个列表的信息集成到 VE 特征中。LTC 模块的输出特征不仅作为 AP 的输入，还会与 VE 特征堆叠在一起，作为 LLaMA 的视觉提示（输入嵌入）。基于 LTC 增强的特征，我们能够推导出更鲁棒的 VADor，并为 LLaMA 提供全面的视频上下文。

**AP：**2层MLP预测分数

**Adapter：**用于投影proj LTC和VE raw feature融合后的feature到LLaMA的输入embedding空间

三阶段训练

第一阶段： 训VE出来的特征直接进AP学异常得分

第二阶段： 训VE出来的和异常相关的feature CA结合的AP 打分器

第一阶段： 训练Adaptor proj到正常LLaMA embedding空间