https://kairosxy.cn/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E5%9F%BA%E7%A1%80/ Recent content in 机器学习基础 on zz.blog Hugo zh-CN Mon, 20 Jul 2026 00:00:00 +0000 https://kairosxy.cn/posts/entropy/ Mon, 20 Jul 2026 00:00:00 +0000 https://kairosxy.cn/posts/entropy/ <blockquote> <p>用一组具体数字，把这三个概念一次讲透</p></blockquote> <h2 id="1-固定场景">1. 固定场景</h2> <p>有两个分布：</p> <ul> <li><strong>Teacher 分布 (P)</strong>：老师认为的真实概率</li> <li><strong>Student 分布 (Q)</strong>：学生学出来的概率</li> </ul> <p>假设有 3 个类别：A / B / C</p> <table> <thead> <tr> <th style="text-align: left">类别</th> <th style="text-align: center">Teacher P</th> <th style="text-align: center">Student Q</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">A</td> <td style="text-align: center">0.7</td> <td style="text-align: center">0.6</td> </tr> <tr> <td style="text-align: left">B</td> <td style="text-align: center">0.2</td> <td style="text-align: center">0.3</td> </tr> <tr> <td style="text-align: left">C</td> <td style="text-align: center">0.1</td> <td style="text-align: center">0.1</td> </tr> </tbody> </table> <p>数组表示：</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">P = [0.7, 0.2, 0.1] </span></span><span class="line"><span class="cl">Q = [0.6, 0.3, 0.1] </span></span></code></pre></div><hr> <h2 id="2-熵-entropy">2. 熵 (Entropy)</h2> <h3 id="定义">定义</h3> <p>$$H(P) = -\sum_i P_i \log P_i$$</p> <h3 id="计算">计算</h3> <p>已知：</p> <ul> <li>$\log 0.7 \approx -0.357$</li> <li>$\log 0.2 \approx -1.609$</li> <li>$\log 0.1 \approx -2.303$</li> </ul> <p>代入：</p> <p>$$ \begin{aligned} H(P) &amp;= -(0.7 \times -0.357 + 0.2 \times -1.609 + 0.1 \times -2.303) \ &amp;= 0.250 + 0.322 + 0.230 \ &amp;\approx 0.802 \end{aligned} $$</p>