更改

对于解依赖数据的应用，损失必须必要地作为观测值的函数，否则模型会数据无关。通常定义为一个只能近似的[https://en.wikipedia.org/wiki/Statistic 统计量]。一个简单的例子是考虑找到最小化<math>{C}=E\left[(f(x) - y)^2\right]</math>的模型 <math>{f}</math>，对于数据对<math>\textstyle (x,y)</math> 来自分布<math>\textstyle \mathcal{D}</math>.。在实际情况下我们可能只有 <math>\textstyle N</math>从 <math>\textstyle \mathcal{D}</math>采样，这样，对于上面的例子我们只能最小化 <math>\textstyle \hat{C}=\frac{1}{N}\sum_{i=1}^N (f(x_i)-y_i)^2</math>. 因此，损失被在数据的一个样本上而不是在整个分布上最小化。

对于解依赖数据的应用，损失必须必要地作为观测值的函数，否则模型会数据无关。通常定义为一个只能近似的[https://en.wikipedia.org/wiki/Statistic 统计量]。一个简单的例子是考虑找到最小化<math>{C}=E\left[(f(x) - y)^2\right]</math>的模型 <math>{f}</math>，对于数据对<math>\textstyle (x,y)</math> 来自分布<math>\textstyle \mathcal{D}</math>.。在实际情况下我们可能只有 <math>\textstyle N</math>从 <math>\textstyle \mathcal{D}</math>采样，这样，对于上面的例子我们只能最小化 <math>\textstyle \hat{C}=\frac{1}{N}\sum_{i=1}^N (f(x_i)-y_i)^2</math>. 因此，损失被在数据的一个样本上而不是在整个分布上最小化。

当 <math>\textstyle N \rightarrow \infty</math>，必须使用【在线机器学习】的某种形式 ，其中损失随着每次观测到新的样本而减小。尽管通常当<math>\textstyle \mathcal{D}</math>固定时使用在线机器学习，它在分布随时间缓慢变化时最有用。在神经网络方法中，一些种类的在线机器学习通常被用于无限数据集。

当 <math>\textstyle N \rightarrow \infty</math>，必须使用[https://en.wikipedia.org/wiki/Online_machine_learning 在线机器学习]的某种形式 ，其中损失随着每次观测到新的样本而减小。尽管通常当<math>\textstyle \mathcal{D}</math>固定时使用在线机器学习，它在分布随时间缓慢变化时最有用。在神经网络方法中，一些种类的在线机器学习通常被用于无限数据集。

====选择一个损失函数====

====选择一个损失函数====

====反向传播====

====反向传播====

连续反向传播的基础<ref name="SCHIDHUB2"/><ref name="scholarpedia2">{{cite journal|year=2015|title=Deep Learning|url=http://www.scholarpedia.org/article/Deep_Learning|journal=Scholarpedia|volume=10|issue=11|page=32832|doi=10.4249/scholarpedia.32832|last1=Schmidhuber|first1=Jürgen|authorlink=Jürgen Schmidhuber|bibcode=2015SchpJ..1032832S}}</ref><ref name=":5">{{Cite journal|last=Dreyfus|first=Stuart E.|date=1990-09-01|title=Artificial neural networks, back propagation, and the Kelley-Bryson gradient procedure|url=http://arc.aiaa.org/doi/10.2514/3.25422|journal=Journal of Guidance, Control, and Dynamics|volume=13|issue=5|pages=926–928|doi=10.2514/3.25422|issn=0731-5090|bibcode=1990JGCD...13..926D}}</ref><ref name="mizutani2000">Eiji Mizutani, [[Stuart Dreyfus]], Kenichi Nishio (2000). On derivation of MLP backpropagation from the Kelley-Bryson optimal-control gradient formula and its application. Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 2000), Como Italy, July 2000. [http://queue.ieor.berkeley.edu/People/Faculty/dreyfus-pubs/ijcnn2k.pdf Online]</ref> 由【Kelley】<ref name="kelley1960">{{cite journal|year=1960|title=Gradient theory of optimal flight paths|url=http://arc.aiaa.org/doi/abs/10.2514/8.5282?journalCode=arsj|journal=Ars Journal|volume=30|issue=10|pages=947–954|doi=10.2514/8.5282|last1=Kelley|first1=Henry J.|authorlink=Henry J. Kelley}}</ref> 在1960和【Bryson】在1961<ref name="bryson1961">[[Arthur E. Bryson]] (1961, April). A gradient method for optimizing multi-stage allocation processes. In Proceedings of the Harvard Univ. Symposium on digital computers and their applications.</ref>使用【动态编程】的原则从【控制论】引出。1962，【Dreyfus】发表了只基于【链式法则】<ref name="dreyfus1962">{{cite journal|year=1962|title=The numerical solution of variational problems|url=https://www.researchgate.net/publication/256244271_The_numerical_solution_of_variational_problems|journal=Journal of Mathematical Analysis and Applications|volume=5|issue=1|pages=30–45|doi=10.1016/0022-247x(62)90004-5|last1=Dreyfus|first1=Stuart|authorlink=Stuart Dreyfus}}</ref>的更简单的衍生。1969，Bryson和【Ho】把它描述成一种多级动态系统优化方法。<ref>{{cite book|url={{google books |plainurl=y |id=8jZBksh-bUMC|page=578}}|title=Artificial Intelligence A Modern Approach|last2=Norvig|first2=Peter|publisher=Prentice Hall|year=2010|isbn=978-0-13-604259-4|page=578|quote=The most popular method for learning in multilayer networks is called Back-propagation.|author-link2=Peter Norvig|first1=Stuart J.|last1=Russell|author-link1=Stuart J. Russell}}</ref><ref name="Bryson1969">{{cite book|url={{google books |plainurl=y |id=1bChDAEACAAJ|page=481}}|title=Applied Optimal Control: Optimization, Estimation and Control|last=Bryson|first=Arthur Earl|publisher=Blaisdell Publishing Company or Xerox College Publishing|year=1969|page=481}}</ref>1970，【Linnainmaa】最终发表了嵌套【可微函数】<ref name="lin1970">[[Seppo Linnainmaa]] (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis (in Finnish), Univ. Helsinki, 6–7.</ref><ref name="lin1976">{{cite journal|year=1976|title=Taylor expansion of the accumulated rounding error|url=|journal=BIT Numerical Mathematics|volume=16|issue=2|pages=146–160|doi=10.1007/bf01931367|last1=Linnainmaa|first1=Seppo|authorlink=Seppo Linnainmaa}}</ref> 的离散连接网络【自动差分机】（AD）的通用方法。这对应于反向传播的现代版本，它在网络稀疏时仍有效<ref name="SCHIDHUB2"/><ref name="scholarpedia2"/><ref name="grie2012">{{Cite journal|last=Griewank|first=Andreas|date=2012|title=Who Invented the Reverse Mode of Differentiation?|url=http://www.math.uiuc.edu/documenta/vol-ismp/52_griewank-andreas-b.pdf|journal=Documenta Matematica, Extra Volume ISMP|volume=|pages=389–400|via=}}</ref><ref name="grie2008">{{cite book|url={{google books |plainurl=y |id=xoiiLaRxcbEC}}|title=Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, Second Edition|last2=Walther|first2=Andrea|publisher=SIAM|year=2008|isbn=978-0-89871-776-1|first1=Andreas|last1=Griewank}}</ref>。1973<ref name="dreyfus1973">{{cite journal|year=1973|title=The computational solution of optimal control problems with time lag|url=|journal=IEEE Transactions on Automatic Control|volume=18|issue=4|pages=383–385|doi=10.1109/tac.1973.1100330|last1=Dreyfus|first1=Stuart|authorlink=Stuart Dreyfus}}</ref> ，Dreyfus使用反向传播适配与误差梯度成比例的控制器【参数】。1974，【Werbos】提出将这个规则应用到ANN上的可能<ref name="werbos1974">[[Paul Werbos]] (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences. PhD thesis, Harvard University.</ref>，1982他将LInnainmaa的AD方法以今天广泛使用的方式应用到神经网络上<ref name="scholarpedia2"/><ref name="werbos1982">{{Cite book|url=http://werbos.com/Neural/SensitivityIFIPSeptember1981.pdf|title=System modeling and optimization|last=Werbos|first=Paul|authorlink=Paul Werbos|publisher=Springer|year=1982|isbn=|location=|pages=762–770|chapter=Applications of advances in nonlinear sensitivity analysis}}</ref>。1986, 【Rumelhart】, Hinton和【Williams】注意到这种方法可以产生有用的神经网络隐藏层到来数据的内部表征。<ref name=":4">{{Cite journal|last=Rumelhart|first=David E.|last2=Hinton|first2=Geoffrey E.|last3=Williams|first3=Ronald J.|title=Learning representations by back-propagating errors|url=http://www.nature.com/articles/Art323533a0|journal=Nature|volume=323|issue=6088|pages=533–536|doi=10.1038/323533a0|year=1986|bibcode=1986Natur.323..533R}}</ref> 1933，Wan第一个<ref name="SCHIDHUB2"/> 用反向传播赢得国际模式识别竞赛。<ref name="wan1993">Eric A. Wan (1993). "Time series prediction by using a connectionist network with internal delay lines." In ''Proceedings of the Santa Fe Institute Studies in the Sciences of Complexity'', '''15''': p. 195. Addison-Wesley Publishing Co.</ref>

连续反向传播的基础<ref name="SCHIDHUB2"/><ref name="scholarpedia2">{{cite journal|year=2015|title=Deep Learning|url=http://www.scholarpedia.org/article/Deep_Learning|journal=Scholarpedia|volume=10|issue=11|page=32832|doi=10.4249/scholarpedia.32832|last1=Schmidhuber|first1=Jürgen|authorlink=Jürgen Schmidhuber|bibcode=2015SchpJ..1032832S}}</ref><ref name=":5">{{Cite journal|last=Dreyfus|first=Stuart E.|date=1990-09-01|title=Artificial neural networks, back propagation, and the Kelley-Bryson gradient procedure|url=http://arc.aiaa.org/doi/10.2514/3.25422|journal=Journal of Guidance, Control, and Dynamics|volume=13|issue=5|pages=926–928|doi=10.2514/3.25422|issn=0731-5090|bibcode=1990JGCD...13..926D}}</ref><ref name="mizutani2000">Eiji Mizutani, [[Stuart Dreyfus]], Kenichi Nishio (2000). On derivation of MLP backpropagation from the Kelley-Bryson optimal-control gradient formula and its application. Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 2000), Como Italy, July 2000. [http://queue.ieor.berkeley.edu/People/Faculty/dreyfus-pubs/ijcnn2k.pdf Online]</ref> 由[https://en.wikipedia.org/wiki/Henry_J._Kelley Kelley]<ref name="kelley1960">{{cite journal|year=1960|title=Gradient theory of optimal flight paths|url=http://arc.aiaa.org/doi/abs/10.2514/8.5282?journalCode=arsj|journal=Ars Journal|volume=30|issue=10|pages=947–954|doi=10.2514/8.5282|last1=Kelley|first1=Henry J.|authorlink=Henry J. Kelley}}</ref> 在1960和[https://en.wikipedia.org/wiki/Arthur_E._Bryson Bryson]在1961<ref name="bryson1961">[[Arthur E. Bryson]] (1961, April). A gradient method for optimizing multi-stage allocation processes. In Proceedings of the Harvard Univ. Symposium on digital computers and their applications.</ref>使用[https://en.wikipedia.org/wiki/Chain_rule 动态编程]的原则从[https://en.wikipedia.org/wiki/Control_theory 控制论]引出。1962，[https://en.wikipedia.org/wiki/Stuart_Dreyfus Dreyfus]发表了只基于[https://en.wikipedia.org/wiki/Chain_rule 链式法则]<ref name="dreyfus1962">{{cite journal|year=1962|title=The numerical solution of variational problems|url=https://www.researchgate.net/publication/256244271_The_numerical_solution_of_variational_problems|journal=Journal of Mathematical Analysis and Applications|volume=5|issue=1|pages=30–45|doi=10.1016/0022-247x(62)90004-5|last1=Dreyfus|first1=Stuart|authorlink=Stuart Dreyfus}}</ref>的更简单的衍生。1969，Bryson和[https://en.wikipedia.org/wiki/Yu-Chi_Ho Ho]把它描述成一种多级动态系统优化方法。<ref>{{cite book|url={{google books |plainurl=y |id=8jZBksh-bUMC|page=578}}|title=Artificial Intelligence A Modern Approach|last2=Norvig|first2=Peter|publisher=Prentice Hall|year=2010|isbn=978-0-13-604259-4|page=578|quote=The most popular method for learning in multilayer networks is called Back-propagation.|author-link2=Peter Norvig|first1=Stuart J.|last1=Russell|author-link1=Stuart J. Russell}}</ref><ref name="Bryson1969">{{cite book|url={{google books |plainurl=y |id=1bChDAEACAAJ|page=481}}|title=Applied Optimal Control: Optimization, Estimation and Control|last=Bryson|first=Arthur Earl|publisher=Blaisdell Publishing Company or Xerox College Publishing|year=1969|page=481}}</ref>1970，[https://en.wikipedia.org/wiki/Seppo_Linnainmaa Linnainmaa]最终发表了嵌套[https://en.wikipedia.org/wiki/Differentiable_function 可微函数]<ref name="lin1970">[[Seppo Linnainmaa]] (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis (in Finnish), Univ. Helsinki, 6–7.</ref><ref name="lin1976">{{cite journal|year=1976|title=Taylor expansion of the accumulated rounding error|url=|journal=BIT Numerical Mathematics|volume=16|issue=2|pages=146–160|doi=10.1007/bf01931367|last1=Linnainmaa|first1=Seppo|authorlink=Seppo Linnainmaa}}</ref> 的离散连接网络[https://en.wikipedia.org/wiki/Automatic_differentiation 自动差分机]（AD）的通用方法。这对应于反向传播的现代版本，它在网络稀疏时仍有效<ref name="SCHIDHUB2"/><ref name="scholarpedia2"/><ref name="grie2012">{{Cite journal|last=Griewank|first=Andreas|date=2012|title=Who Invented the Reverse Mode of Differentiation?|url=http://www.math.uiuc.edu/documenta/vol-ismp/52_griewank-andreas-b.pdf|journal=Documenta Matematica, Extra Volume ISMP|volume=|pages=389–400|via=}}</ref><ref name="grie2008">{{cite book|url={{google books |plainurl=y |id=xoiiLaRxcbEC}}|title=Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, Second Edition|last2=Walther|first2=Andrea|publisher=SIAM|year=2008|isbn=978-0-89871-776-1|first1=Andreas|last1=Griewank}}</ref>。1973<ref name="dreyfus1973">{{cite journal|year=1973|title=The computational solution of optimal control problems with time lag|url=|journal=IEEE Transactions on Automatic Control|volume=18|issue=4|pages=383–385|doi=10.1109/tac.1973.1100330|last1=Dreyfus|first1=Stuart|authorlink=Stuart Dreyfus}}</ref> ，Dreyfus使用反向传播适配与误差梯度成比例的控制器【参数】。1974，【Werbos】提出将这个规则应用到ANN上的可能<ref name="werbos1974">[[Paul Werbos]] (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences. PhD thesis, Harvard University.</ref>，1982他将LInnainmaa的AD方法以今天广泛使用的方式应用到神经网络上<ref name="scholarpedia2"/><ref name="werbos1982">{{Cite book|url=http://werbos.com/Neural/SensitivityIFIPSeptember1981.pdf|title=System modeling and optimization|last=Werbos|first=Paul|authorlink=Paul Werbos|publisher=Springer|year=1982|isbn=|location=|pages=762–770|chapter=Applications of advances in nonlinear sensitivity analysis}}</ref>。1986, 【Rumelhart】, Hinton和【Williams】注意到这种方法可以产生有用的神经网络隐藏层到来数据的内部表征。<ref name=":4">{{Cite journal|last=Rumelhart|first=David E.|last2=Hinton|first2=Geoffrey E.|last3=Williams|first3=Ronald J.|title=Learning representations by back-propagating errors|url=http://www.nature.com/articles/Art323533a0|journal=Nature|volume=323|issue=6088|pages=533–536|doi=10.1038/323533a0|year=1986|bibcode=1986Natur.323..533R}}</ref> 1933，Wan第一个<ref name="SCHIDHUB2"/> 用反向传播赢得国际模式识别竞赛。<ref name="wan1993">Eric A. Wan (1993). "Time series prediction by using a connectionist network with internal delay lines." In ''Proceedings of the Santa Fe Institute Studies in the Sciences of Complexity'', '''15''': p. 195. Addison-Wesley Publishing Co.</ref>

反向传播的权重更新可以通过【随机梯度下降】完成，使用下面的等式：

反向传播的权重更新可以通过【随机梯度下降】完成，使用下面的等式：

===学习算法===

===学习算法===

反向传播训练算法有这三类：

反向传播训练算法有这三类：

* 【最速下降】(带参数学习速率和【动量】，【弹性反向传播】；

* [https://en.wikipedia.org/wiki/Gradient_descent 最速下降](带参数学习速率和[https://en.wikipedia.org/wiki/Gradient_descent#The_momentum_method 动量]，[https://en.wikipedia.org/wiki/Rprop 弹性反向传播]；

* 拟牛顿 （Broyden-Fletcher-Goldfarb-Shanno），【单步割线】；

* 拟牛顿 （[https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm Broyden-Fletcher-Goldfarb-Shanno]），单[https://en.wikipedia.org/wiki/Secant_method 步割线]；

* 【Levenberg-Marquardt】和【共轭梯度】(Fletcher-Reeves 更新, Polak-Ribiére 更新, Powell-Beale 重启,标度共轭梯度)。<ref>{{cite conference|author1=M. Forouzanfar |author2=H. R. Dajani |author3=V. Z. Groza |author4=M. Bolic |author5=S. Rajan  |last-author-amp=yes |date=July 2010 | title = Comparison of Feed-Forward Neural Network Training Algorithms for Oscillometric Blood Pressure Estimation | conference = 4th Int. Workshop Soft Computing Applications | publisher = IEEE| location = Arad, Romania |url=https://www.researchgate.net/profile/Mohamad_Forouzanfar/publication/224173336_Comparison_of_Feed-Forward_Neural_Network_training_algorithms_for_oscillometric_blood_pressure_estimation/links/00b7d533829c3a7484000000.pdf?ev=pub_int_doc_dl&origin=publication_detail&inViewer=true&msrp=TyT96%2BjWOHJo%2BVhkMF4IzwHPAImSd442n%2BAkEuXj9qBmQSZ495CpxqlaOYon%2BSlEzWQElBGyJmbBCiiUOV8ImeEqPFXiIRivcrWsWmlPBYU%3D }}</ref>

* [https://en.wikipedia.org/wiki/Levenberg%E2%80%93Marquardt_algorithm Levenberg-Marquardt]和[https://en.wikipedia.org/wiki/Conjugate_gradient_method 共轭梯度](Fletcher-Reeves 更新, Polak-Ribiére 更新, Powell-Beale 重启,标度共轭梯度)。<ref>{{cite conference|author1=M. Forouzanfar |author2=H. R. Dajani |author3=V. Z. Groza |author4=M. Bolic |author5=S. Rajan  |last-author-amp=yes |date=July 2010 | title = Comparison of Feed-Forward Neural Network Training Algorithms for Oscillometric Blood Pressure Estimation | conference = 4th Int. Workshop Soft Computing Applications | publisher = IEEE| location = Arad, Romania |url=https://www.researchgate.net/profile/Mohamad_Forouzanfar/publication/224173336_Comparison_of_Feed-Forward_Neural_Network_training_algorithms_for_oscillometric_blood_pressure_estimation/links/00b7d533829c3a7484000000.pdf?ev=pub_int_doc_dl&origin=publication_detail&inViewer=true&msrp=TyT96%2BjWOHJo%2BVhkMF4IzwHPAImSd442n%2BAkEuXj9qBmQSZ495CpxqlaOYon%2BSlEzWQElBGyJmbBCiiUOV8ImeEqPFXiIRivcrWsWmlPBYU%3D }}</ref>

【进化法】<ref>{{cite conference| author1 = de Rigo, D. | author2 = Castelletti, A. | author3 = Rizzoli, A. E. | author4 = Soncini-Sessa, R. | author5 = Weber, E. |date=January 2005 | title = A selective improvement technique for fastening Neuro-Dynamic Programming in Water Resources Network Management | conference = 16th IFAC World Congress | conferenceurl = http://www.nt.ntnu.no/users/skoge/prost/proceedings/ifac2005/Index.html | booktitle = Proceedings of the 16th IFAC World Congress – IFAC-PapersOnLine | editor = Pavel Zítek | volume = 16 | publisher = IFAC | location = Prague, Czech Republic | url = http://www.nt.ntnu.no/users/skoge/prost/proceedings/ifac2005/Papers/Paper4269.html

[https://en.wikipedia.org/wiki/Evolutionary_methods 进化法]<ref>{{cite conference| author1 = de Rigo, D. | author2 = Castelletti, A. | author3 = Rizzoli, A. E. | author4 = Soncini-Sessa, R. | author5 = Weber, E. |date=January 2005 | title = A selective improvement technique for fastening Neuro-Dynamic Programming in Water Resources Network Management | conference = 16th IFAC World Congress | conferenceurl = http://www.nt.ntnu.no/users/skoge/prost/proceedings/ifac2005/Index.html | booktitle = Proceedings of the 16th IFAC World Congress – IFAC-PapersOnLine | editor = Pavel Zítek | volume = 16 | publisher = IFAC | location = Prague, Czech Republic | url = http://www.nt.ntnu.no/users/skoge/prost/proceedings/ifac2005/Papers/Paper4269.html

  | accessdate = 30 December 2011 | doi = 10.3182/20050703-6-CZ-1902.02172 | isbn = 978-3-902661-75-3 }}</ref>，【基因表达式编程】<ref>{{cite web|last=Ferreira|first=C.|year=2006|title=Designing Neural Networks Using Gene Expression Programming|url= http://www.gene-expression-programming.com/webpapers/Ferreira-ASCT2006.pdf|publisher= In A. Abraham, B. de Baets, M. Köppen, and B. Nickolay, eds., Applied Soft Computing Technologies: The Challenge of Complexity, pages 517–536, Springer-Verlag}}</ref>，【模拟退火】<ref>{{cite conference| author = Da, Y. |author2=Xiurun, G. |date=July 2005 | title = An improved PSO-based ANN with simulated annealing technique | conference = New Aspects in Neurocomputing: 11th European Symposium on Artificial Neural Networks | conferenceurl = http://www.dice.ucl.ac.be/esann/proceedings/electronicproceedings.htm | editor = T. Villmann | publisher = Elsevier | doi = 10.1016/j.neucom.2004.07.002 }}<!--| accessdate = 30 December 2011 --></ref>，【期望最大化】，【非参数方法】和【粒子群算法】<ref>{{cite conference| author = Wu, J. |author2=Chen, E. |date=May 2009 | title = A Novel Nonparametric Regression Ensemble for Rainfall Forecasting Using Particle Swarm Optimization Technique Coupled with Artificial Neural Network | conference = 6th International Symposium on Neural Networks, ISNN 2009 | conferenceurl = http://www2.mae.cuhk.edu.hk/~isnn2009/ | editors = Wang, H., Shen, Y., Huang, T., Zeng, Z. | publisher = Springer | doi = 10.1007/978-3-642-01513-7-6 | isbn = 978-3-642-01215-0 }}<!--| accessdate = 1 January 2012 --></ref>是训练神经网络的其他方法。

  | accessdate = 30 December 2011 | doi = 10.3182/20050703-6-CZ-1902.02172 | isbn = 978-3-902661-75-3 }}</ref>，[https://en.wikipedia.org/wiki/Gene_expression_programming 基因表达式编程]<ref>{{cite web|last=Ferreira|first=C.|year=2006|title=Designing Neural Networks Using Gene Expression Programming|url= http://www.gene-expression-programming.com/webpapers/Ferreira-ASCT2006.pdf|publisher= In A. Abraham, B. de Baets, M. Köppen, and B. Nickolay, eds., Applied Soft Computing Technologies: The Challenge of Complexity, pages 517–536, Springer-Verlag}}</ref>，[https://en.wikipedia.org/wiki/Simulated_annealing 模拟退火]<ref>{{cite conference| author = Da, Y. |author2=Xiurun, G. |date=July 2005 | title = An improved PSO-based ANN with simulated annealing technique | conference = New Aspects in Neurocomputing: 11th European Symposium on Artificial Neural Networks | conferenceurl = http://www.dice.ucl.ac.be/esann/proceedings/electronicproceedings.htm | editor = T. Villmann | publisher = Elsevier | doi = 10.1016/j.neucom.2004.07.002 }}<!--| accessdate = 30 December 2011 --></ref>，[https://en.wikipedia.org/wiki/Expectation-maximization 期望最大化]，[https://en.wikipedia.org/wiki/Non-parametric_methods 非参数方法]和[https://en.wikipedia.org/wiki/Particle_swarm_optimization 粒子群算法]<ref>{{cite conference| author = Wu, J. |author2=Chen, E. |date=May 2009 | title = A Novel Nonparametric Regression Ensemble for Rainfall Forecasting Using Particle Swarm Optimization Technique Coupled with Artificial Neural Network | conference = 6th International Symposium on Neural Networks, ISNN 2009 | conferenceurl = http://www2.mae.cuhk.edu.hk/~isnn2009/ | editors = Wang, H., Shen, Y., Huang, T., Zeng, Z. | publisher = Springer | doi = 10.1007/978-3-642-01513-7-6 | isbn = 978-3-642-01215-0 }}<!--| accessdate = 1 January 2012 --></ref>是训练神经网络的其他方法。

==== 递推收敛学习算法（Convergent recursive learning algorithm） ====

==== 递推收敛学习算法（Convergent recursive learning algorithm） ====

这是一种为【小脑模型关节控制器】（CMAC）神经网络特别设计的学习方法。2004，递推最小二乘法被引入在线训练【CMAC】神经网络。<ref name="Qin1">Ting Qin, et al. "A learning algorithm of CMAC based on RLS." Neural Processing Letters 19.1 (2004): 49–61.</ref>这种算法可以使用任何新的输入数据，一步收敛并一步内更新所有权重。最开始，这种算法有【计算复杂度】''O''(''N''³). 基于【QR分解】，这种算法简化到''O''(''N'').<ref name="Qin2">Ting Qin, et al. "Continuous CMAC-QRLS and its systolic array." Neural Processing Letters 22.1 (2005): 1–16.</ref>

这是一种为[https://en.wikipedia.org/wiki/Cerebellar_model_articulation_controller 小脑模型关节控制器]（CMAC）神经网络特别设计的学习方法。2004，递推最小二乘法被引入在线训练CMAC神经网络。<ref name="Qin1">Ting Qin, et al. "A learning algorithm of CMAC based on RLS." Neural Processing Letters 19.1 (2004): 49–61.</ref>这种算法可以使用任何新的输入数据，一步收敛并一步内更新所有权重。最开始，这种算法有[https://en.wikipedia.org/wiki/Computational_complexity_theory 计算复杂度]''O''(''N''³). 基于[https://en.wikipedia.org/wiki/QR_decomposition QR分解]，这种算法简化到''O''(''N'').<ref name="Qin2">Ting Qin, et al. "Continuous CMAC-QRLS and its systolic array." Neural Processing Letters 22.1 (2005): 1–16.</ref>

 

== 变体 ==

== 变体 ==