3,094
个编辑
更改
→RWKV Block（基本结构）
:<math>
:<math>
k_t = W_k \cdot \left(x_t \cdot (1 - w_{\text{mixk}}) + x_{t-1} \cdot w_{\text{mixk}}\right)\\
k_t = W_k \cdot \left(x_t \cdot (1 - w_{\text{mixk}}) + x_{t-1} \cdot w_{\text{mixk}}\right)
</math>
:<math>
v_t = W_v \cdot \left(x_t \cdot (1 - w_{\text{mixv}}) + x_{t-1} \cdot w_{\text{mixv}}\right)\\
v_t = W_v \cdot \left(x_t \cdot (1 - w_{\text{mixv}}) + x_{t-1} \cdot w_{\text{mixv}}\right)\\
</math>
:<math>
r_t = \sigma\left(W_r \cdot \left(x_t \cdot (1 - w_{\text{mixr}}) + x_{t-1} \cdot w_{\text{mixr}}\right)\right)\\
r_t = \sigma\left(W_r \cdot \left(x_t \cdot (1 - w_{\text{mixr}}) + x_{t-1} \cdot w_{\text{mixr}}\right)\right)\\
</math>
:<math>
s_t = \sum_{i=0}^{t} \exp\left(-\lambda (t - i)\right) \cdot k_i \odot v_i\\
s_t = \sum_{i=0}^{t} \exp\left(-\lambda (t - i)\right) \cdot k_i \odot v_i\\
</math>
:<math>
y_t = r_t \odot s_t
y_t = r_t \odot s_t
</math>
</math>