28
个编辑
更改
跳到导航
跳到搜索
第134行：
第134行： − + + + + + + + + +
→大语言模型涌现的机制解释
Lubana, E. S., Kawaguchi, K., Dick, R. P., & Tanaka, H. (2024). A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language (arXiv:2408.12578). arXiv. <nowiki>https://doi.org/10.48550/arXiv.2408.12578</nowiki>
Lubana, E. S., Kawaguchi, K., Dick, R. P., & Tanaka, H. (2024). A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language (arXiv:2408.12578). arXiv. <nowiki>https://doi.org/10.48550/arXiv.2408.12578</nowiki>
=== Quantization Model ===
=== 离散量子理论（Quantization Model） ===
The Quantization Model of Neural Scaling. <nowiki>https://arxiv.org/abs/2303.13506</nowiki>
The Quantization Model of Neural Scaling. <nowiki>https://arxiv.org/abs/2303.13506</nowiki>
=== 大模型的顿悟现象（Grokking） ===
=== 大模型的顿悟现象（Grokking） ===
机器学习中的 grokking 现象是指，训练集的收敛往往快于测试集的收敛，二者的收敛速度存在一个时间差。在这个时间差内，算法更像是在过拟合的空间中运作，然后随着迭代次数增加向泛化性上优化，即测试集准确度升高。
* Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets，<nowiki>https://arxiv.org/pdf/2201.02177</nowiki>
* Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets，<nowiki>https://arxiv.org/pdf/2201.02177</nowiki>
* Towards Understanding Grokking: An Effective Theory of Representation Learning. <nowiki>https://proceedings.neurips.cc/paper_files/paper/2022/hash/dfc310e81992d2e4cedc09ac47eff13e-Abstract-Conference.html</nowiki>
* Towards Understanding Grokking: An Effective Theory of Representation Learning. <nowiki>https://proceedings.neurips.cc/paper_files/paper/2022/hash/dfc310e81992d2e4cedc09ac47eff13e-Abstract-Conference.html</nowiki>
* A Good ML theory is Like Physics: A Physicist’s Analysis of Grokking <nowiki>https://zimingliublog.wordpress.com/2023/06/16/a-good-ml-theory-is-like-physics-a-physicists-analysis-of-grokking/</nowiki>
=== Double Descent ===
=== Double Descent ===
Deep Double Descent: Where Bigger Models and More Data Hurt. <nowiki>https://arxiv.org/abs/1912.02292</nowiki>
Deep Double Descent: Where Bigger Models and More Data Hurt. <nowiki>https://arxiv.org/abs/1912.02292</nowiki>
相关读书会：
https://pattern.swarma.org/study_group_issue/446
参考文献<references />
参考文献<references />