3,094
个编辑
更改
→目标函数
\mathcal{J}_{GRPO}=\mathbb{E}_{q\sim P(Q),\{o_i\}_{i=1}^G\sim \pi_{\theta_{old}}(O|q)}\frac{1}{G}\sum_{i=1}^G\left(\min\left(\frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}A_i,\mathrm{clip}\left(\frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{old}}(o_i|q)},1-\epsilon,1+\epsilon\right)A_i\right)-\beta\cdot\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})\right),
\mathcal{J}_{GRPO}=\mathbb{E}_{q\sim P(Q),\{o_i\}_{i=1}^G\sim \pi_{\theta_{old}}(O|q)}\frac{1}{G}\sum_{i=1}^G\left(\min\left(\frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}A_i,\mathrm{clip}\left(\frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{old}}(o_i|q)},1-\epsilon,1+\epsilon\right)A_i\right)-\beta\cdot\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})\right),
</math>
</math>
其中：
<math>
<math>
\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})=\frac{\pi_{ref}(o_i|q)}{\pi_{\theta}(o_i|q)}-\log
\mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})=\frac{\pi_{ref}(o_i|q)}{\pi_{\theta}(o_i|q)}-\log\frac{\pi_{ref}(o_i|q)}{\pi_{\theta}(o_i|q)}-1
</math>
</math>