更改

人工神经网络 (查看源代码)

2018年8月27日 (一) 15:58的版本

添加122字节、 2018年8月27日 (一) 15:58

第197行：第197行：

卷积神经网络 (CNN) 是一类深度前馈网络，由一或多层[https://en.wikipedia.org/wiki/Convolution 卷积]层和位于其上的全连接层（与典型ANN中的匹配）组成。它使用相等权重和池化层。特别地，最大池化<ref name="Weng19932"/>通常通过Fukushima的卷积结构组织。<ref name="FUKU1980">{{cite journal|year=1980|title=Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position|url=|journal=Biol. Cybern.|volume=36|issue=4|pages=193–202|last1=Fukushima|first1=K.}}</ref>这种结构允许CNN利用输入数据的2D结构

−

CNN适合处理视觉和其他二维数据<ref name="LECUN1989">LeCun ''et al.'', "Backpropagation Applied to Handwritten Zip Code Recognition," ''Neural Computation'', 1, pp. 541–551, 1989.</ref><ref name="lecun2016slides">[https://en.wikipedia.org/wiki/Yann_LeCun Yann LeCun] (2016). Slides on Deep Learning [https://indico.cern.ch/event/510372/ Online]</ref>，它们在图像和语音应用中展示出了优秀的结果。它们可以被标准反向传播训练。CNN比其他普通的深度前馈神经网络更容易训练且有更少的需要估计的参数。<ref name="STANCNN">{{cite ~~web~~|url=http://ufldl.stanford.edu/tutorial/supervised/ConvolutionalNeuralNetwork/|title=Unsupervised Feature Learning and Deep Learning Tutorial|publisher=}}</ref> 计算机视觉中应用的例子包括[https://en.wikipedia.org/wiki/DeepDream DeepDream]<ref name="deepdream">{{cite journal|last2=Liu|first2=Wei|last3=Jia|first3=Yangqing|last4=Sermanet|first4=Pierre|last5=Reed|first5=Scott|last6=Anguelov|first6=Dragomir|last7=Erhan|first7=Dumitru|last8=Vanhoucke|first8=Vincent|last9=Rabinovich|first9=Andrew|date=|year=2014|title=Going Deeper with Convolutions|url=|journal=Computing Research Repository|volume=|pages=1|url=https://arxiv.org/abs/1409.4842|via=|first1=Christian|last1=Szegedy}}</ref>和[https://en.wikipedia.org/wiki/Robot_navigation 机器人导航]<ref>{{cite journal | last=Ran | first=Lingyan | last2=Zhang | first2=Yanning | last3=Zhang | first3=Qilin | last4=Yang | first4=Tao | title=Convolutional Neural Network-Based Robot Navigation Using Uncalibrated Spherical Images | journal=Sensors | publisher=MDPI AG | volume=17 | issue=6 | date=2017-06-12 ~~| issn=1424-8220~~ | page=1341 | url=https://qilin-zhang.github.io/_pages/pdfs/sensors-17-01341.pdf}}</ref>

+

CNN适合处理视觉和其他二维数据<ref name="LECUN1989">LeCun ''et al.'', "Backpropagation Applied to Handwritten Zip Code Recognition," ''Neural Computation'', 1, pp. 541–551, 1989.</ref><ref name="lecun2016slides">[https://en.wikipedia.org/wiki/Yann_LeCun Yann LeCun] (2016). Slides on Deep Learning [https://indico.cern.ch/event/510372/ Online]</ref>，它们在图像和语音应用中展示出了优秀的结果。它们可以被标准反向传播训练。CNN比其他普通的深度前馈神经网络更容易训练且有更少的需要估计的参数。<ref name="STANCNN">{{cite journal|url=http://ufldl.stanford.edu/tutorial/supervised/ConvolutionalNeuralNetwork/|title=Unsupervised Feature Learning and Deep Learning Tutorial|publisher=}}</ref> 计算机视觉中应用的例子包括[https://en.wikipedia.org/wiki/DeepDream DeepDream]<ref name="deepdream">{{cite journal|last2=Liu|first2=Wei|last3=Jia|first3=Yangqing|last4=Sermanet|first4=Pierre|last5=Reed|first5=Scott|last6=Anguelov|first6=Dragomir|last7=Erhan|first7=Dumitru|last8=Vanhoucke|first8=Vincent|last9=Rabinovich|first9=Andrew|date=|year=2014|title=Going Deeper with Convolutions|url=|journal=Computing Research Repository|volume=|pages=1|url=https://arxiv.org/abs/1409.4842|via=|first1=Christian|last1=Szegedy}}</ref>和[https://en.wikipedia.org/wiki/Robot_navigation 机器人导航]<ref>{{cite journal | last=Ran | first=Lingyan | last2=Zhang | first2=Yanning | last3=Zhang | first3=Qilin | last4=Yang | first4=Tao | title=Convolutional Neural Network-Based Robot Navigation Using Uncalibrated Spherical Images | journal=Sensors | publisher=MDPI AG | volume=17 | issue=6 | date=2017-06-12 | page=1341 | url=https://qilin-zhang.github.io/_pages/pdfs/sensors-17-01341.pdf}}</ref>

。

===长短期记忆（ Long short-term memory） ===

−

长短期记忆 (LSTM) 网络是避免了[https://en.wikipedia.org/wiki/Vanishing_gradient_problem 梯度消失问题]。<ref name=":03">{{Cite journal|last=Hochreiter|first=Sepp|author-link=~~Sepp Hochreiter~~|last2=Schmidhuber|first2=Jürgen|author-link2=~~Jürgen Schmidhuber~~|date=1997-11-01|title=Long Short-Term Memory|url=http://www.mitpressjournals.org/doi/10.1162/neco.1997.9.8.1735|journal=Neural Computation|volume=9|issue=8|pages=1735–1780~~|issn=0899-7667~~|via=}}</ref> LSTM通常被称为遗忘门的循环门扩展<ref name=":10">{{Cite ~~web~~|url=https://www.researchgate.net/publication/220320057_Learning_Precise_Timing_with_LSTM_Recurrent_Networks|title=Learning Precise Timing with LSTM Recurrent Networks (PDF Download Available)|~~website~~=ResearchGate|language=en|access-date=2017-06-13|pp=115–143}}</ref>。 LSTM网络避免了反向传播误差的消失或爆炸。<ref name="HOCH19912"/> 误差可以通过在空间展开的LSTM中的无限制的虚层反向回流。也就是说，LSTM可以学习“非常深的学习”任务，<ref name="SCHIDHUB2" />这些任务需要记住上千甚至上百万离散时间步前的事件。问题特殊的LSTM形态的拓扑结构可以成为进化的LSTM，<ref>{{Cite journal|last=Bayer|first=Justin|last2=Wierstra|first2=Daan|last3=Togelius|first3=Julian|last4=Schmidhuber|first4=Jürgen|date=2009-09-14|title=Evolving Memory Cell Structures for Sequence Learning|url=https://link.springer.com/chapter/10.1007/978-3-642-04277-5_76|journal=Artificial Neural Networks – ICANN 2009|volume=5769|language=en|publisher=Springer, Berlin, Heidelberg|pages=755–764|series=Lecture Notes in Computer Science}}</ref> 能处理长延迟和混合高低频成分的信号。

+

长短期记忆 (LSTM) 网络是避免了[https://en.wikipedia.org/wiki/Vanishing_gradient_problem 梯度消失问题]。<ref name=":03">{{Cite journal|last=Hochreiter|first=Sepp|author-link=https://en.wikipedia.org/wiki/Sepp_Hochreiter|last2=Schmidhuber|first2=Jürgen|author-link2=https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber|date=1997-11-01|title=Long Short-Term Memory|url=http://www.mitpressjournals.org/doi/10.1162/neco.1997.9.8.1735|journal=Neural Computation|volume=9|issue=8|pages=1735–1780|via=}}</ref> LSTM通常被称为遗忘门的循环门扩展<ref name=":10">{{Cite journal|url=https://www.researchgate.net/publication/220320057_Learning_Precise_Timing_with_LSTM_Recurrent_Networks|title=Learning Precise Timing with LSTM Recurrent Networks (PDF Download Available)|journal=ResearchGate|language=en|access-date=2017-06-13|pp=115–143}}</ref>。 LSTM网络避免了反向传播误差的消失或爆炸。<ref name="HOCH19912"/> 误差可以通过在空间展开的LSTM中的无限制的虚层反向回流。也就是说，LSTM可以学习“非常深的学习”任务，<ref name="SCHIDHUB2" />这些任务需要记住上千甚至上百万离散时间步前的事件。问题特殊的LSTM形态的拓扑结构可以成为进化的LSTM，<ref>{{Cite journal|last=Bayer|first=Justin|last2=Wierstra|first2=Daan|last3=Togelius|first3=Julian|last4=Schmidhuber|first4=Jürgen|date=2009-09-14|title=Evolving Memory Cell Structures for Sequence Learning|url=https://link.springer.com/chapter/10.1007/978-3-642-04277-5_76|journal=Artificial Neural Networks – ICANN 2009|volume=5769|language=en|publisher=Springer, Berlin, Heidelberg|pages=755–764|series=Lecture Notes in Computer Science}}</ref> 能处理长延迟和混合高低频成分的信号。

大量LSTM RNN<ref>{{Cite journal|last=Fernández|first=Santiago|last2=Graves|first2=Alex|last3=Schmidhuber|first3=Jürgen|date=2007|title=Sequence labelling in structured domains with hierarchical recurrent neural networks|url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.79.1887|journal=In Proc. 20th Int. Joint Conf. on Artificial In℡ligence, Ijcai 2007|pages=774–779}}</ref> 使用联结主义时间分类（CTC）训练，<ref name=":12">{{Cite journal|last=Graves|first=Alex|last2=Fernández|first2=Santiago|last3=Gomez|first3=Faustino|date=2006|title=Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks|url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.75.6306|journal=In Proceedings of the International Conference on Machine Learning, ICML 2006|pages=369–376}}</ref> 给定相应输入序列，可以找到一个最大化训练集中标记序列概率的RNN权重矩阵。CTC达到了校准和识别。

−

2003，LSTM开始在传统语音识别器中具有竞争力。<ref name="graves2003">{{Cite ~~web~~|url=Ftp://ftp.idsia.ch/pub/juergen/bioadit2004.pdf|title=Biologically Plausible Speech Recognition with LSTM Neural Nets|last=Graves|first=Alex|last2=Eck|first2=Douglas|date=2003|~~website~~=1st Intl. Workshop on Biologically Inspired Approaches to Advanced Information Technology, Bio-ADIT 2004, Lausanne, Switzerland|pages=175–184|archive-url=|archive-date=|dead-url=|access-date=|last3=Beringer|first3=Nicole|last4=Schmidhuber|first4=Jürgen|authorlink4=https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber}}</ref>2007，与CTC的结合在语音数据上达到了第一个良好的结果。<ref name="fernandez2007keyword">{{Cite journal|last=Fernández|first=Santiago|last2=Graves|first2=Alex|last3=Schmidhuber|first3=Jürgen|date=2007|title=An Application of Recurrent Neural Networks to Discriminative Keyword Spotting|url=http://dl.acm.org/citation.cfm?id=1778066.1778092|journal=Proceedings of the 17th International Conference on Artificial Neural Networks|series=ICANN'07|location=Berlin, Heidelberg|publisher=Springer-Verlag|pages=220–229}}</ref>2009，一个CTC训练的LSTM成为第一个赢得模式识别比赛的RNN，当它赢得了几个连笔[https://en.wikipedia.org/wiki/Handwriting_recognition 手写识别]比赛。<ref name="SCHIDHUB2" /><ref name="graves20093"/>2014，[https://en.wikipedia.org/wiki/Baidu 百度]使用CTC训练的RNN打破了Switchboard Hub5'00语音识别在基准测试数据集上的表现，而没有使用传统语音处理方法。<ref name="hannun2014">{{cite journal|last=Hannun|first=Awni|last2=Case|first2=Carl|last3=Casper|first3=Jared|last4=Catanzaro|first4=Bryan|last5=Diamos|first5=Greg|last6=Elsen|first6=Erich|last7=Prenger|first7=Ryan|last8=Satheesh|first8=Sanjeev|last9=Sengupta|first9=Shubho|date=2014-12-17|title=Deep Speech: Scaling up end-to-end speech recognition|url=https://arxiv.org/abs/1412.5567}}</ref> LSTM也提高了大量词汇语音识别，<ref name="sak2014">{{Cite ~~web~~|url=https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43905.pdf|title=Long Short-Term Memory recurrent neural network architectures for large scale acoustic modeling|last=Sak|first=Hasim|last2=Senior|first2=Andrew|date=2014~~|website=~~|archive-url=|archive-date=|dead-url=|access-date=|last3=Beaufays|first3=Francoise}}</ref><ref name="liwu2015">{{cite journal|last=Li|first=Xiangang|last2=Wu|first2=Xihong|date=2014-10-15|title=Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition|url=https://arxiv.org/abs/1410.4281}}</ref>文本到语音合成，<ref>{{Cite ~~web~~|url=https://www.researchgate.net/publication/287741874_TTS_synthesis_with_bidirectional_LSTM_based_Recurrent_Neural_Networks|title=TTS synthesis with bidirectional LSTM based Recurrent Neural Networks|last=Fan|first=Y.|last2=Qian|first2=Y.|date=2014|~~website~~=ResearchGate|language=en|archive-url=|archive-date=|dead-url=|access-date=2017-06-13|last3=Xie|first3=F.|last4=Soong|first4=F. K.}}</ref> 对谷歌安卓<ref name="scholarpedia2"/><ref name="zen2015">{{Cite ~~web~~|url=https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43266.pdf|title=Unidirectional Long Short-Term Memory Recurrent Neural Network with Recurrent Output Layer for Low-Latency Speech Synthesis|last=Zen|first=Heiga|last2=Sak|first2=Hasim|date=2015|~~website~~=Google.com|publisher=ICASSP|pages=4470–4474|archive-url=|archive-date=|dead-url=|access-date=}}</ref>和真实图片的传声头像。<ref name="fan2015">{{Cite journal|last=Fan|first=Bo|last2=Wang|first2=Lijuan|last3=Soong|first3=Frank K.|last4=Xie|first4=Lei|date=2015|title=Photo-Real Talking Head with Deep Bidirectional LSTM|url=https://www.microsoft.com/en-us/research/wp-content/uploads/2015/04/icassp2015_fanbo_1009.pdf|journal=Proceedings of ICASSP|volume=|pages=|via=}}</ref>2015，谷歌的语音识别通过CTC训练的LSTM提高了49%的性能。<ref name="sak2015">{{Cite ~~web~~|url=http://googleresearch.blogspot.ch/2015/09/google-voice-search-faster-and-more.html|title=Google voice search: faster and more accurate|last=Sak|first=Haşim|last2=Senior|first2=Andrew|date=September 2015~~|website=~~|archive-url=|archive-date=|dead-url=|access-date=|last3=Rao|first3=Kanishka|last4=Beaufays|first4=Françoise|last5=Schalkwyk|first5=Johan}}</ref>

+

2003，LSTM开始在传统语音识别器中具有竞争力。<ref name="graves2003">{{Cite journal|url=Ftp://ftp.idsia.ch/pub/juergen/bioadit2004.pdf|title=Biologically Plausible Speech Recognition with LSTM Neural Nets|last=Graves|first=Alex|last2=Eck|first2=Douglas|date=2003|journal=1st Intl. Workshop on Biologically Inspired Approaches to Advanced Information Technology, Bio-ADIT 2004, Lausanne, Switzerland|pages=175–184|archive-url=|archive-date=|dead-url=|access-date=|last3=Beringer|first3=Nicole|last4=Schmidhuber|first4=Jürgen|authorlink4=https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber}}</ref>2007，与CTC的结合在语音数据上达到了第一个良好的结果。<ref name="fernandez2007keyword">{{Cite journal|last=Fernández|first=Santiago|last2=Graves|first2=Alex|last3=Schmidhuber|first3=Jürgen|date=2007|title=An Application of Recurrent Neural Networks to Discriminative Keyword Spotting|url=http://dl.acm.org/citation.cfm?id=1778066.1778092|journal=Proceedings of the 17th International Conference on Artificial Neural Networks|series=ICANN'07|location=Berlin, Heidelberg|publisher=Springer-Verlag|pages=220–229}}</ref>2009，一个CTC训练的LSTM成为第一个赢得模式识别比赛的RNN，当它赢得了几个连笔[https://en.wikipedia.org/wiki/Handwriting_recognition 手写识别]比赛。<ref name="SCHIDHUB2" /><ref name="graves20093"/>2014，[https://en.wikipedia.org/wiki/Baidu 百度]使用CTC训练的RNN打破了Switchboard Hub5'00语音识别在基准测试数据集上的表现，而没有使用传统语音处理方法。<ref name="hannun2014">{{cite journal|last=Hannun|first=Awni|last2=Case|first2=Carl|last3=Casper|first3=Jared|last4=Catanzaro|first4=Bryan|last5=Diamos|first5=Greg|last6=Elsen|first6=Erich|last7=Prenger|first7=Ryan|last8=Satheesh|first8=Sanjeev|last9=Sengupta|first9=Shubho|date=2014-12-17|title=Deep Speech: Scaling up end-to-end speech recognition|url=https://arxiv.org/abs/1412.5567}}</ref> LSTM也提高了大量词汇语音识别，<ref name="sak2014">{{Cite journal|url=https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43905.pdf|title=Long Short-Term Memory recurrent neural network architectures for large scale acoustic modeling|last=Sak|first=Hasim|last2=Senior|first2=Andrew|date=2014|archive-url=|archive-date=|dead-url=|access-date=|last3=Beaufays|first3=Francoise}}</ref><ref name="liwu2015">{{cite journal|last=Li|first=Xiangang|last2=Wu|first2=Xihong|date=2014-10-15|title=Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition|url=https://arxiv.org/abs/1410.4281}}</ref>文本到语音合成，<ref>{{Cite journal|url=https://www.researchgate.net/publication/287741874_TTS_synthesis_with_bidirectional_LSTM_based_Recurrent_Neural_Networks|title=TTS synthesis with bidirectional LSTM based Recurrent Neural Networks|last=Fan|first=Y.|last2=Qian|first2=Y.|date=2014|journal=ResearchGate|language=en|archive-url=|archive-date=|dead-url=|access-date=2017-06-13|last3=Xie|first3=F.|last4=Soong|first4=F. K.}}</ref> 对谷歌安卓<ref name="scholarpedia2"/><ref name="zen2015">{{Cite journal|url=https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43266.pdf|title=Unidirectional Long Short-Term Memory Recurrent Neural Network with Recurrent Output Layer for Low-Latency Speech Synthesis|last=Zen|first=Heiga|last2=Sak|first2=Hasim|date=2015|journal=Google.com|publisher=ICASSP|pages=4470–4474|archive-url=|archive-date=|dead-url=|access-date=}}</ref>和真实图片的传声头像。<ref name="fan2015">{{Cite journal|last=Fan|first=Bo|last2=Wang|first2=Lijuan|last3=Soong|first3=Frank K.|last4=Xie|first4=Lei|date=2015|title=Photo-Real Talking Head with Deep Bidirectional LSTM|url=https://www.microsoft.com/en-us/research/wp-content/uploads/2015/04/icassp2015_fanbo_1009.pdf|journal=Proceedings of ICASSP|volume=|pages=|via=}}</ref>2015，谷歌的语音识别通过CTC训练的LSTM提高了49%的性能。<ref name="sak2015">{{Cite journal|url=http://googleresearch.blogspot.ch/2015/09/google-voice-search-faster-and-more.html|title=Google voice search: faster and more accurate|last=Sak|first=Haşim|last2=Senior|first2=Andrew|date=September 2015|archive-url=|archive-date=|dead-url=|access-date=|last3=Rao|first3=Kanishka|last4=Beaufays|first4=Françoise|last5=Schalkwyk|first5=Johan}}</ref>

−

LSTM在[https://en.wikipedia.org/wiki/Natural_Language_Processing 自然语言处理]中变得受欢迎。不像之前基于[https://en.wikipedia.org/wiki/Hidden_Markov_model 隐式马尔科夫模型]和相似概念的模型，LSTM可以学习识别[https://en.wikipedia.org/wiki/Context-sensitive_languages 上下文有关语言]。<ref name="gers2001">{{cite journal|last2=Schmidhuber|first2=Jürgen|year=2001|title=LSTM Recurrent Networks Learn Simple Context Free and Context Sensitive Languages|url=|journal=IEEE Transactions on Neural Networks|volume=12|issue=6|pages=1333–1340|last1=Gers|first1=Felix A.|authorlink2=https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber}}</ref>LSTM提高了机器翻译，<ref>{{cite ~~web~~ | last=Huang | first=Jie | last2=Zhou | first2=Wengang | last3=Zhang | first3=Qilin | last4=Li | first4=Houqiang | last5=Li | first5=Weiping | title=Video-based Sign Language Recognition without Temporal Segmentation ~~| eprint=1801.10111~~ | date=2018-01-30 | url=https://arxiv.org/pdf/1801.10111.pdf}}</ref><ref name="NIPS2014">{{Cite journal|last=Sutskever|first=L.|last2=Vinyals|first2=O.|last3=Le|first3=Q.|date=2014|title=Sequence to Sequence Learning with Neural Networks|url=https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf|journal=NIPS'14 Proceedings of the 27th International Conference on Neural Information Processing Systems |volume=2 |pages=3104–3112 |url=https://arxiv.org/abs/1409.3215}}</ref>[https://en.wikipedia.org/wiki/Language_modeling 语言建模]<ref name="vinyals2016">{{cite journal|last=Jozefowicz|first=Rafal|last2=Vinyals|first2=Oriol|last3=Schuster|first3=Mike|last4=Shazeer|first4=Noam|last5=Wu|first5=Yonghui|date=2016-02-07|title=Exploring the Limits of Language Modeling|url=https://arxiv.org/abs/1602.02410}}</ref>和多语言语言处理。<ref name="gillick2015">{{cite journal|last=Gillick|first=Dan|last2=Brunk|first2=Cliff|last3=Vinyals|first3=Oriol|last4=Subramanya|first4=Amarnag|date=2015-11-30|title=Multilingual Language Processing From Bytes|url=https://arxiv.org/abs/1512.00103}}</ref>与CNN结合的LSTM提高了自动图像字幕标记。<ref name="vinyals2015">{{cite journal|last=Vinyals|first=Oriol|last2=Toshev|first2=Alexander|last3=Bengio|first3=Samy|last4=Erhan|first4=Dumitru|date=2014-11-17|title=Show and Tell: A Neural Image Caption Generator|url=https://arxiv.org/abs/1411.4555}}</ref>

+

LSTM在[https://en.wikipedia.org/wiki/Natural_Language_Processing 自然语言处理]中变得受欢迎。不像之前基于[https://en.wikipedia.org/wiki/Hidden_Markov_model 隐式马尔科夫模型]和相似概念的模型，LSTM可以学习识别[https://en.wikipedia.org/wiki/Context-sensitive_languages 上下文有关语言]。<ref name="gers2001">{{cite journal|last2=Schmidhuber|first2=Jürgen|year=2001|title=LSTM Recurrent Networks Learn Simple Context Free and Context Sensitive Languages|url=|journal=IEEE Transactions on Neural Networks|volume=12|issue=6|pages=1333–1340|last1=Gers|first1=Felix A.|authorlink2=https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber}}</ref>LSTM提高了机器翻译，<ref>{{cite journal | last=Huang | first=Jie | last2=Zhou | first2=Wengang | last3=Zhang | first3=Qilin | last4=Li | first4=Houqiang | last5=Li | first5=Weiping | title=Video-based Sign Language Recognition without Temporal Segmentation | date=2018-01-30 | url=https://arxiv.org/pdf/1801.10111.pdf}}</ref><ref name="NIPS2014">{{Cite journal|last=Sutskever|first=L.|last2=Vinyals|first2=O.|last3=Le|first3=Q.|date=2014|title=Sequence to Sequence Learning with Neural Networks|url=https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf|journal=NIPS'14 Proceedings of the 27th International Conference on Neural Information Processing Systems |volume=2 |pages=3104–3112 |url=https://arxiv.org/abs/1409.3215}}</ref>[https://en.wikipedia.org/wiki/Language_modeling 语言建模]<ref name="vinyals2016">{{cite journal|last=Jozefowicz|first=Rafal|last2=Vinyals|first2=Oriol|last3=Schuster|first3=Mike|last4=Shazeer|first4=Noam|last5=Wu|first5=Yonghui|date=2016-02-07|title=Exploring the Limits of Language Modeling|url=https://arxiv.org/abs/1602.02410}}</ref>和多语言语言处理。<ref name="gillick2015">{{cite journal|last=Gillick|first=Dan|last2=Brunk|first2=Cliff|last3=Vinyals|first3=Oriol|last4=Subramanya|first4=Amarnag|date=2015-11-30|title=Multilingual Language Processing From Bytes|url=https://arxiv.org/abs/1512.00103}}</ref>与CNN结合的LSTM提高了自动图像字幕标记。<ref name="vinyals2015">{{cite journal|last=Vinyals|first=Oriol|last2=Toshev|first2=Alexander|last3=Bengio|first3=Samy|last4=Erhan|first4=Dumitru|date=2014-11-17|title=Show and Tell: A Neural Image Caption Generator|url=https://arxiv.org/abs/1411.4555}}</ref>

=== 深度储蓄池计算（Deep reservoir computing） ===

−

深度储蓄池计算和深度回声状态网络 (deepESNs)<ref>{{Cite journal|last=Gallicchio|first=Claudio|last2=Micheli|first2=Alessio|last3=Pedrelli|first3=Luca|title=Deep reservoir computing: A critical experimental analysis|url=http://www.sciencedirect.com/science/article/pii/S0925231217307567|journal=Neurocomputing|volume=268|pages=87|year=2017}}</ref><ref>{{Cite journal|last=Gallicchio|first=Claudio|last2=Micheli|first2=Alessio|date=|title=Echo State Property of Deep Reservoir Computing Networks|url=https://link.springer.com/article/10.1007/s12559-017-9461-9|journal=Cognitive Computation|language=en|volume=9|issue=3|pages=337–350~~|issn=1866-9956~~|via=|year=2017}}</ref> 为高效训练的分层处理时序数据的模型提供了一个框架，同时使RNN的层次化构成的内在作用能够探查。

+

深度储蓄池计算和深度回声状态网络 (deepESNs)<ref>{{Cite journal|last=Gallicchio|first=Claudio|last2=Micheli|first2=Alessio|last3=Pedrelli|first3=Luca|title=Deep reservoir computing: A critical experimental analysis|url=http://www.sciencedirect.com/science/article/pii/S0925231217307567|journal=Neurocomputing|volume=268|pages=87|year=2017}}</ref><ref>{{Cite journal|last=Gallicchio|first=Claudio|last2=Micheli|first2=Alessio|date=|title=Echo State Property of Deep Reservoir Computing Networks|url=https://link.springer.com/article/10.1007/s12559-017-9461-9|journal=Cognitive Computation|language=en|volume=9|issue=3|pages=337–350|via=|year=2017}}</ref> 为高效训练的分层处理时序数据的模型提供了一个框架，同时使RNN的层次化构成的内在作用能够探查。

=== 深度置信网络（Deep belief networks） ===

第221行：第221行：

===大内存和检索神经网络===

−

大内存和检索神经网络（LAMSTAR）<ref name="book2013">{{cite book|url=https://books.google.com/books?id=W6W6CgAAQBAJ&pg=PP1|title=Principles of Artificial Neural Networks|last=Graupe|first=Daniel|publisher=World Scientific|year=2013|location=|pages=1–|ref=harv}}</ref><ref ~~name="GrPatent"~~>{{~~Patent~~|US|~~5920852 A~~|~~D. Graupe,"~~ Large memory storage and retrieval (LAMSTAR) network~~, April 1996~~}}</ref>是多层快速深度学习神经网络，可以同时使用许多滤波。这些滤波可能非线性，随机，逻辑，[https://en.wikipedia.org/wiki/Non-stationary 非固定]甚至非解析。它们是生物学动机的并且可以连续学习。

+

大内存和检索神经网络（LAMSTAR）<ref name="book2013">{{cite book|url=https://books.google.com/books?id=W6W6CgAAQBAJ&pg=PP1|title=Principles of Artificial Neural Networks|last=Graupe|first=Daniel|publisher=World Scientific|year=2013|location=|pages=1–|ref=harv}}</ref><ref>{{Cite journal|last=Graupe|first=Daniel|title=Large memory storage and retrieval(LAMSTAR) network|url=https://worldwide.espacenet.com/textdoc?DB=EPODOC&IDX=US5920852|year=2016}}</ref>是多层快速深度学习神经网络，可以同时使用许多滤波。这些滤波可能非线性，随机，逻辑，[https://en.wikipedia.org/wiki/Non-stationary 非固定]甚至非解析。它们是生物学动机的并且可以连续学习。

LAMSTAR神经网络可以作为在空间或时间或二者兼具的域内的动力神经网络。它的速度由[https://en.wikipedia.org/wiki/Hebbian 赫布（Hebbian）]连接权重提供，<ref name=book2013a>D. Graupe, "Principles of Artificial Neural Networks.3rd Edition", World Scientific Publishers, 2013, pp. 203–274.</ref>它整合多种并且通常不同的滤波（预处理函数）到它的与给定学习任务相关的很多层和函数中。这很大程度模拟了整合多种预处理器（[https://en.wikipedia.org/wiki/Cochlea 耳蜗]，[https://en.wikipedia.org/wiki/Retina 视网膜]等）和皮层（听觉，视觉等）和它们的多个域的生物学习。通过使用抑制，相关，它的深度学习能力大大增强，甚至当在任务中时，处理不完整数据的能力或“丢失的”神经元或层的能力也显著增强。由于它的连接权重，它是完全透明的。这些连接权重允许动态地决定更新和去除，并且帮助任务相关的层，滤波或单独神经元的排列。

−

LAMSTAR被应用于多个领域，包括医药<ref>{{Cite journal|last=Nigam|first=Vivek Prakash|last2=Graupe|first2=Daniel|date=2004-01-01|title=A neural-network-based detection of epilepsy|journal=Neurological Research|volume=26|issue=1|pages=55–60~~|issn=0161-6412~~}}</ref><ref name=":11">{{Cite journal|last=Waxman|first=Jonathan A.|last2=Graupe|first2=Daniel|last3=Carley|first3=David W.|date=2010-04-01|title=Automated Prediction of Apnea and Hypopnea, Using a LAMSTAR Artificial Neural Network|url=http://www.atsjournals.org/doi/abs/10.1164/rccm.200907-1146OC|journal=American Journal of Respiratory and Critical Care Medicine|volume=181|issue=7|pages=727–733~~|issn=1073-449X~~}}</ref><ref name="GrGrZh">{{cite journal|last2=Graupe|first2=M. H.|last3=Zhong|first3=Y.|last4=Jackson|first4=R. K.|year=2008|title=Blind adaptive filtering for non-invasive extraction of the fetal electrocardiogram and its non-stationarities|url=|journal=Proc. Inst. Mech. Eng. H|volume=222|issue=8|pages=1221–1234|last1=Graupe|first1=D.}}</ref>和金融预测，<ref name="book2013b">~~{{harvnb|~~Graupe|2013|pp=240–253}}</ref>在未知噪音下嘈杂语音的适应性滤波，<ref name="GrAbon">{{cite journal|last2=Abon|first2=J.|year=2002|title=A Neural Network for Blind Adaptive Filtering of Unknown Noise from Speech|url=https://www.tib.eu/en/search/id/BLCP:CN019373941/Blind-Adaptive-Filtering-of-Speech-from-Noise-of/|journal=Intelligent Engineering Systems Through Artificial Neural Networks|language=en|publisher=Technische Informationsbibliothek (TIB)|volume=12|issue=|pages=683–688|last1=Graupe|first1=D.|accessdate=2017-06-14}}</ref> 静态图像识别，<ref name="book2013c">D. Graupe, "Principles of Artificial Neural Networks.3rd Edition", World Scientific Publishers", 2013, pp. 253–274.</ref>视频图像识别，<ref name="Girado">{{cite journal|last2=Sandin|first2=D. J.|last3=DeFanti|first3=T. A.|year=2003|title=Real-time camera-based face detection using a modified LAMSTAR neural network system|url=|journal=Proc. SPIE 5015, Applications of Artificial Neural Networks in Image Processing VIII|volume=5015|issue=|pages=36|page=|last1=Girado|first1=J. I.|series=Applications of Artificial Neural Networks in Image Processing VIII}}</ref>软件安全<ref name="VenkSel">{{cite journal|last2=Selvan|first2=S.|year=2007|title=Intrusion Detection using an Improved Competitive Learning Lamstar Network|url=|journal=International Journal of Computer Science and Network Security|volume=7|issue=2|pages=255–263|last1=Venkatachalam|first1=V}}</ref> 和非线性系统的适应性控制。<ref>{{Cite ~~web~~|url=https://www.researchgate.net/publication/262316982_Control_of_unstable_nonlinear_and_nonstationary_systems_using_LAMSTAR_neural_networks|title=Control of unstable nonlinear and nonstationary systems using LAMSTAR neural networks|last=Graupe|first=D.|last2=Smollack|first2=M.|date=2007|~~website~~=ResearchGate|publisher=Proceedings of 10th IASTED on Intelligent Control, Sect.592,|pages=141–144|language=en|archive-url=|archive-date=|dead-url=|access-date=2017-06-14}}</ref> LAMSTAR比基于[https://en.wikipedia.org/wiki/ReLU ReLU]函数滤波和最大池化的CNN在20个对比研究中有明显更快的学习速度，和稍低的错误率。<ref name="book1016">{{cite book|url=https://books.google.com/books?id=e5hIDQAAQBAJ&pg=PA57|title=Deep Learning Neural Networks: Design and Case Studies|last=Graupe|first=Daniel|date=7 July 2016|publisher=World Scientific Publishing Co Inc|year=|location=|pages=57–110}}</ref>

+

LAMSTAR被应用于多个领域，包括医药<ref>{{Cite journal|last=Nigam|first=Vivek Prakash|last2=Graupe|first2=Daniel|date=2004-01-01|title=A neural-network-based detection of epilepsy|journal=Neurological Research|volume=26|issue=1|pages=55–60}}</ref><ref name=":11">{{Cite journal|last=Waxman|first=Jonathan A.|last2=Graupe|first2=Daniel|last3=Carley|first3=David W.|date=2010-04-01|title=Automated Prediction of Apnea and Hypopnea, Using a LAMSTAR Artificial Neural Network|url=http://www.atsjournals.org/doi/abs/10.1164/rccm.200907-1146OC|journal=American Journal of Respiratory and Critical Care Medicine|volume=181|issue=7|pages=727–733}}</ref><ref name="GrGrZh">{{cite journal|last2=Graupe|first2=M. H.|last3=Zhong|first3=Y.|last4=Jackson|first4=R. K.|year=2008|title=Blind adaptive filtering for non-invasive extraction of the fetal electrocardiogram and its non-stationarities|url=|journal=Proc. Inst. Mech. Eng. H|volume=222|issue=8|pages=1221–1234|last1=Graupe|first1=D.}}</ref>和金融预测，<ref name="book2013b">[https://en.wikipedia.org/wiki/Artificial_neural_network#CITEREFGraupe2013 Graupe 2013"] "pp=240–253"</ref>在未知噪音下嘈杂语音的适应性滤波，<ref name="GrAbon">{{cite journal|last2=Abon|first2=J.|year=2002|title=A Neural Network for Blind Adaptive Filtering of Unknown Noise from Speech|url=https://www.tib.eu/en/search/id/BLCP:CN019373941/Blind-Adaptive-Filtering-of-Speech-from-Noise-of/|journal=Intelligent Engineering Systems Through Artificial Neural Networks|language=en|publisher=Technische Informationsbibliothek (TIB)|volume=12|issue=|pages=683–688|last1=Graupe|first1=D.|accessdate=2017-06-14}}</ref> 静态图像识别，<ref name="book2013c">D. Graupe, "Principles of Artificial Neural Networks.3rd Edition", World Scientific Publishers", 2013, pp. 253–274.</ref>视频图像识别，<ref name="Girado">{{cite journal|last2=Sandin|first2=D. J.|last3=DeFanti|first3=T. A.|year=2003|title=Real-time camera-based face detection using a modified LAMSTAR neural network system|url=|journal=Proc. SPIE 5015, Applications of Artificial Neural Networks in Image Processing VIII|volume=5015|issue=|pages=36|page=|last1=Girado|first1=J. I.|series=Applications of Artificial Neural Networks in Image Processing VIII}}</ref>软件安全<ref name="VenkSel">{{cite journal|last2=Selvan|first2=S.|year=2007|title=Intrusion Detection using an Improved Competitive Learning Lamstar Network|url=|journal=International Journal of Computer Science and Network Security|volume=7|issue=2|pages=255–263|last1=Venkatachalam|first1=V}}</ref> 和非线性系统的适应性控制。<ref>{{Cite journal|url=https://www.researchgate.net/publication/262316982_Control_of_unstable_nonlinear_and_nonstationary_systems_using_LAMSTAR_neural_networks|title=Control of unstable nonlinear and nonstationary systems using LAMSTAR neural networks|last=Graupe|first=D.|last2=Smollack|first2=M.|date=2007|journal=ResearchGate|publisher=Proceedings of 10th IASTED on Intelligent Control, Sect.592,|pages=141–144|language=en|archive-url=|archive-date=|dead-url=|access-date=2017-06-14}}</ref> LAMSTAR比基于[https://en.wikipedia.org/wiki/ReLU ReLU]函数滤波和最大池化的CNN在20个对比研究中有明显更快的学习速度，和稍低的错误率。<ref name="book1016">{{cite book|url=https://books.google.com/books?id=e5hIDQAAQBAJ&pg=PA57|title=Deep Learning Neural Networks: Design and Case Studies|last=Graupe|first=Daniel|date=7 July 2016|publisher=World Scientific Publishing Co Inc|year=|location=|pages=57–110}}</ref>

这些应用展示了钻入数据藏在浅学习网络和人类感觉下的面貌，如预测[https://en.wikipedia.org/wiki/Sleep_apnea 睡眠呼吸中止症]，<ref name=":11" />怀孕早期从放在母亲腹部皮肤表面电极记录的胎儿心电图，<ref name="GrGrZh" /> 金融预测<ref name="book2013" />或者嘈杂语音的盲过滤<ref name="GrAbon" />

的案例。

−

LAMSTAR在1996被提议（[https://www.google.com/patents/US5920852 US Patent|5920852 A]），然后从1997到2002被Graupe和Kordylewski深入开发。<ref>{{Cite journal|last=Graupe|first=D.|last2=Kordylewski|first2=H.|date=August 1996|title=Network based on SOM (Self-Organizing-Map) modules combined with statistical decision tools|url=http://ieeexplore.ieee.org/document/594203/|journal=Proceedings of the 39th Midwest Symposium on Circuits and Systems|volume=1|pages=471–474 vol.1}}</ref><ref>{{Cite journal|last=Graupe|first=D.|last2=Kordylewski|first2=H.|date=1998-03-01|title=A Large Memory Storage and Retrieval Neural Network for Adaptive Retrieval and Diagnosis|url=http://www.worldscientific.com/doi/abs/10.1142/S0218194098000091|journal=International Journal of Software Engineering and Knowledge Engineering|volume=08|issue=1|pages=115–138~~|issn=0218-1940~~}}</ref><ref name="Kordylew">{{cite journal|last2=Graupe|first2=D|last3=Liu|first3=K.|year=2001|title=A novel large-memory neural network as an aid in medical diagnosis applications|url=|journal=IEEE Transactions on Information Technology in Biomedicine|volume=5|issue=3|pages=202–209|last1=Kordylewski|first1=H.}}</ref>一个更改的版本称为LAMSTAR2，被Schneider 和 Graupe在2008开发。<ref name="Schn">{{cite journal|last2=Graupe|year=2008|title=A modified LAMSTAR neural network and its applications|url=|journal=International journal of neural systems|volume=18|issue=4|pages=331–337|last1=Schneider|first1=N.C.}}</ref><ref name="book2013d">~~{{harvnb|~~Graupe|2013|p=217}}</ref>

+

LAMSTAR在1996被提议（[https://www.google.com/patents/US5920852 US Patent|5920852 A]），然后从1997到2002被Graupe和Kordylewski深入开发。<ref>{{Cite journal|last=Graupe|first=D.|last2=Kordylewski|first2=H.|date=August 1996|title=Network based on SOM (Self-Organizing-Map) modules combined with statistical decision tools|url=http://ieeexplore.ieee.org/document/594203/|journal=Proceedings of the 39th Midwest Symposium on Circuits and Systems|volume=1|pages=471–474 vol.1}}</ref><ref>{{Cite journal|last=Graupe|first=D.|last2=Kordylewski|first2=H.|date=1998-03-01|title=A Large Memory Storage and Retrieval Neural Network for Adaptive Retrieval and Diagnosis|url=http://www.worldscientific.com/doi/abs/10.1142/S0218194098000091|journal=International Journal of Software Engineering and Knowledge Engineering|volume=08|issue=1|pages=115–138}}</ref><ref name="Kordylew">{{cite journal|last2=Graupe|first2=D|last3=Liu|first3=K.|year=2001|title=A novel large-memory neural network as an aid in medical diagnosis applications|url=|journal=IEEE Transactions on Information Technology in Biomedicine|volume=5|issue=3|pages=202–209|last1=Kordylewski|first1=H.}}</ref>一个更改的版本称为LAMSTAR2，被Schneider 和 Graupe在2008开发。<ref name="Schn">{{cite journal|last2=Graupe|year=2008|title=A modified LAMSTAR neural network and its applications|url=|journal=International journal of neural systems|volume=18|issue=4|pages=331–337|last1=Schneider|first1=N.C.}}</ref><ref name="book2013d">[https://en.wikipedia.org/wiki/Artificial_neural_network#CITEREFGraupe2013 "Graupe 2013"] "p=217"</ref>

=== 叠加（去噪）自动编码器（Stacked (de-noising) auto-encoders） ===

第237行：第237行：

这个算法通过<math>{q_D({\tilde{\boldsymbol{x}}}|{\boldsymbol{x}})}</math>从 <math>\boldsymbol{x}</math> 到<math>\tilde{\boldsymbol{x}}</math> 的随机映射开始，这是【corrupting】步。然后【corrupted】输入 <math>\tilde{\boldsymbol{x}}</math> 传过基本自动编码过程，并被映射到隐含表示<math>\boldsymbol{y} = f_\theta(\tilde{\boldsymbol{x}}) = s(\boldsymbol{W}\tilde{\boldsymbol{x}}+b)</math>。从这个隐含表示中，我们可以重建<math>\boldsymbol{z} = g_\theta(\boldsymbol{y})</math>。在最后一步，一个最小化算法运行以使 '''''z'''''尽可能和【uncorrupted】输入<math>\boldsymbol{x}</math>近。重建误差<math>L_H(\boldsymbol{x},\boldsymbol{z})</math>可以是带有双弯曲仿射解码器的[https://en.wikipedia.org/wiki/Cross-entropy 交叉熵]损失，或者[https://en.wikipedia.org/wiki/Affine_transformation 仿射]解码器的平方误差。<ref name="ref9" />

−

为了做出一个深度结构，叠加自动编码器。<ref name="ballard1987">{{Cite ~~web~~|url=http://www.aaai.org/Papers/AAAI/1987/AAAI87-050.pdf|title=Modular learning in neural networks|last=Ballard|first=Dana H.|date=1987|~~website~~=Proceedings of AAAI|pages=279–284|archive-url=|archive-date=|dead-url=|access-date=}}</ref> 一旦第一个去噪自动编码器的编码函数<math>f_\theta</math>被学习并且用于改善输入（差的输入），第二级可以被训练。<ref name="ref9" />

+

为了做出一个深度结构，叠加自动编码器。<ref name="ballard1987">{{Cite journal|url=http://www.aaai.org/Papers/AAAI/1987/AAAI87-050.pdf|title=Modular learning in neural networks|last=Ballard|first=Dana H.|date=1987|journal=Proceedings of AAAI|pages=279–284|archive-url=|archive-date=|dead-url=|access-date=}}</ref> 一旦第一个去噪自动编码器的编码函数<math>f_\theta</math>被学习并且用于改善输入（差的输入），第二级可以被训练。<ref name="ref9" />

一旦叠加自动编码器被训练，它的输出可以被用作[https://en.wikipedia.org/wiki/Supervised_learning 监督学习]算法，如[https://en.wikipedia.org/wiki/Support_vector_machine 支持向量机]分类器或一个多分类[https://en.wikipedia.org/wiki/Logistic_regression 逻辑回归]的输入。<ref name="ref9" />

===深度叠加网络（ Deep stacking networks ）===

−

深度叠加网络 (DSN)<ref name="ref17">{{cite journal|last2=Yu|first2=Dong|last3=Platt|first3=John|date=2012|title=Scalable stacking and learning for building deep architectures|url=http://research-srv.microsoft.com/pubs/157586/DSN-ICASSP2012.pdf|journal=2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)|pages=2133–2136|last1=Deng|first1=Li}}</ref> （深度凸网络）是基于多块的简化神经网络模块的层级。在2011被Deng和Dong引入。<ref name="ref16">{{cite journal|last2=Yu|first2=Dong|date=2011|title=Deep Convex Net: A Scalable Architecture for Speech Pattern Classification|url=http://www.truebluenegotiations.com/files/deepconvexnetwork-interspeech2011-pub.pdf|journal=Proceedings of the Interspeech|pages=2285–2288|last1=Deng|first1=Li}}</ref> 它用带[https://en.wikipedia.org/wiki/Closed-form_expression 闭型解]的[https://en.wikipedia.org/wiki/Convex_optimization_problem 凸优化]表达学习，强调机制与[https://en.wikipedia.org/wiki/Ensemble_learning 层叠泛化]的相似。<ref name="ref18">{{cite journal|date=1992|title=Stacked generalization|journal=Neural Networks|volume=5|issue=2|pages=241–259|last1=David|first1=Wolpert}}</ref>每个DSN块是一个容易被[https://en.wikipedia.org/wiki/Supervised_learning 监督]式自我训练的简单模块，不需要整个块的反向传播。<ref>{{Cite journal|last=Bengio|first=Y.|date=2009-11-15|title=Learning Deep Architectures for AI|url=http://www.nowpublishers.com/article/Details/MAL-006|journal=Foundations and Trends® in Machine Learning|language=English|volume=2|issue=1|pages=1–127~~|issn=1935-8237~~}}</ref>

+

深度叠加网络 (DSN)<ref name="ref17">{{cite journal|last2=Yu|first2=Dong|last3=Platt|first3=John|date=2012|title=Scalable stacking and learning for building deep architectures|url=http://research-srv.microsoft.com/pubs/157586/DSN-ICASSP2012.pdf|journal=2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)|pages=2133–2136|last1=Deng|first1=Li}}</ref> （深度凸网络）是基于多块的简化神经网络模块的层级。在2011被Deng和Dong引入。<ref name="ref16">{{cite journal|last2=Yu|first2=Dong|date=2011|title=Deep Convex Net: A Scalable Architecture for Speech Pattern Classification|url=http://www.truebluenegotiations.com/files/deepconvexnetwork-interspeech2011-pub.pdf|journal=Proceedings of the Interspeech|pages=2285–2288|last1=Deng|first1=Li}}</ref> 它用带[https://en.wikipedia.org/wiki/Closed-form_expression 闭型解]的[https://en.wikipedia.org/wiki/Convex_optimization_problem 凸优化]表达学习，强调机制与[https://en.wikipedia.org/wiki/Ensemble_learning 层叠泛化]的相似。<ref name="ref18">{{cite journal|date=1992|title=Stacked generalization|journal=Neural Networks|volume=5|issue=2|pages=241–259|last1=David|first1=Wolpert}}</ref>每个DSN块是一个容易被[https://en.wikipedia.org/wiki/Supervised_learning 监督]式自我训练的简单模块，不需要整个块的反向传播。<ref>{{Cite journal|last=Bengio|first=Y.|date=2009-11-15|title=Learning Deep Architectures for AI|url=http://www.nowpublishers.com/article/Details/MAL-006|journal=Foundations and Trends® in Machine Learning|language=English|volume=2|issue=1|pages=1–127}}</ref>

每块由一个简化的带单隐层的[https://en.wikipedia.org/wiki/Multi-layer_perceptron 多层感知机]（MLP）组成。隐藏层 '''''h''''' 有逻辑[https://en.wikipedia.org/wiki/Sigmoid_function 双弯曲的][https://en.wikipedia.org/wiki/Logistic_function 单元]，输出层有线性单元。这些层之间的连接用权重矩阵'''''U;'''''表示，输入到隐藏层连接有权重矩阵 '''''W'''''。目标向量'''''t''''' 形成矩阵 '''''T'''''的列, 输入数据向量 '''''x'''''形成矩阵 '''''X.''''' 的列。隐藏单元的矩阵是<math>\boldsymbol{H} = \sigma(\boldsymbol{W}^T\boldsymbol{X})</math>. 。模块按顺序训练，因此底层的权重 '''''W''''' 在每一阶段已知。函数执行对应元素的[https://en.wikipedia.org/wiki/Logistic_function 逻辑双弯曲]操作。每块估计同一个最终标记类 ''y''，这个估计被原始输入'''''X''''' 串级起来，形成下一个块的扩展输入。因此第一块的输入只包含原始输入，而下游的块输入加上了前驱块的输出。然后学习上层权重矩阵 '''''U''''' ，给定网络中其他权重可以被表达为一个凸优化问题:

第256行：第256行：

=== 钉板受限玻尔兹曼机（Spike-and-slab RBMs） ===

深度学习有带[https://en.wikipedia.org/wiki/Real_number 实值]输入的需要，如在高斯受限玻尔兹曼机中一样，引出了“钉板”[https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine 受限玻尔兹曼机]，它模拟带严格[https://en.wikipedia.org/wiki/Binary_variable 二进制][https://en.wikipedia.org/wiki/Latent_variable 潜变量]的连续值输入。<ref name="ref30">{{cite journal|last2=Bergstra|first2=James|last3=Bengio|first3=Yoshua|date=2011|title=A Spike and Slab Restricted Boltzmann Machine|url=http://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2011_CourvilleBB11.pdf|journal=JMLR: Workshop and Conference Proceeding|volume=15|pages=233–241|last1=Courville|first1=Aaron}}</ref>与基本RBM和它的变体一样，钉板RBM是[https://en.wikipedia.org/wiki/Bipartite_graph 二分图]，好像GRBM一样，可见单元（输入）是实值的。

−

区别在隐藏层，每个隐藏单元有二进制的发放值【？】和实值的平滑值【？】。spike是一个离散的在零处的[https://en.wikipedia.org/wiki/Probability_mass 概率质量]，slab是一个连续域上的[https://en.wikipedia.org/wiki/Probability_density 概率密度]<ref name="ref32">{{cite ~~conference~~|last1=Courville|first1=Aaron|last2=Bergstra|first2=James|last3=Bengio|first3=Yoshua|~~chapter~~=Unsupervised Models of Images by Spike-and-Slab RBMs|~~title~~=Proceedings of the 28th International Conference on Machine Learning|volume=10|pages=1–8|date=2011|url=http://machinelearning.wustl.edu/mlpapers/paper_files/ICML2011Courville_591.pdf}}</ref>，它们的混合形成了[https://en.wikipedia.org/wiki/Prior_probability 先验]。<ref name="ref31">{{cite journal|last2=Beauchamp|first2=J|date=1988|title=Bayesian Variable Selection in Linear Regression|journal=Journal of the American Statistical Association|volume=83|issue=404|pages=1023–1032|last1=Mitchell|first1=T}}</ref>

+

区别在隐藏层，每个隐藏单元有二进制的发放值【？】和实值的平滑值【？】。spike是一个离散的在零处的[https://en.wikipedia.org/wiki/Probability_mass 概率质量]，slab是一个连续域上的[https://en.wikipedia.org/wiki/Probability_density 概率密度]<ref name="ref32">{{cite journal|last1=Courville|first1=Aaron|last2=Bergstra|first2=James|last3=Bengio|first3=Yoshua|title=Unsupervised Models of Images by Spike-and-Slab RBMs|journal=Proceedings of the 28th International Conference on Machine Learning|volume=10|pages=1–8|date=2011|url=http://machinelearning.wustl.edu/mlpapers/paper_files/ICML2011Courville_591.pdf}}</ref>，它们的混合形成了[https://en.wikipedia.org/wiki/Prior_probability 先验]。<ref name="ref31">{{cite journal|last2=Beauchamp|first2=J|date=1988|title=Bayesian Variable Selection in Linear Regression|journal=Journal of the American Statistical Association|volume=83|issue=404|pages=1023–1032|last1=Mitchell|first1=T}}</ref>

ssRBM的一个扩展是µ-ssRBM，使用[https://en.wikipedia.org/wiki/Energy_function 能量函数]中的附加项提供了额外的建模能力。这些项之一使模型形成了spike值的[https://en.wikipedia.org/wiki/Conditional_probability_distribution 条件分布]，通过给定一个观测值[https://en.wikipedia.org/wiki/Marginalizing_out 边际化出]slab值。

第281行：第281行：

=== 带单独记忆结构的网络（Networks with separate memory structures） ===

−

使用ANN整合外部记忆可以追溯到关于分布表征<ref name="Hinton, Geoffrey E 19842">{{Cite ~~web~~|url=http://repository.cmu.edu/cgi/viewcontent.cgi?article=2841&context=compsci|title=Distributed representations|last=Hinton|first=Geoffrey E.|date=1984~~|website=~~|archive-url=|archive-date=|dead-url=|access-date=}}</ref>和[https://en.wikipedia.org/wiki/Teuvo_Kohonen Kohonen]的[https://en.wikipedia.org/wiki/Teuvo_Kohonen 自组织映射]的早期研究。例如, 在[https://en.wikipedia.org/wiki/Sparse_distributed_memory 稀疏分布式记忆]或[https://en.wikipedia.org/wiki/Hierarchical_temporal_memory 层级空间记忆]中，神经网络编码的模式被用于[https://en.wikipedia.org/wiki/Content-addressable_memory 可寻址内容的记忆]的地址，使用“神经元”本质上作为地址 [https://en.wikipedia.org/wiki/Encoder 编码器]和[https://en.wikipedia.org/wiki/Binary_decoder 解码器]。然而早期这种记忆的控制器不可微。

+

使用ANN整合外部记忆可以追溯到关于分布表征<ref name="Hinton, Geoffrey E 19842">{{Cite journal|url=http://repository.cmu.edu/cgi/viewcontent.cgi?article=2841&context=compsci|title=Distributed representations|last=Hinton|first=Geoffrey E.|date=1984|archive-url=|archive-date=|dead-url=|access-date=}}</ref>和[https://en.wikipedia.org/wiki/Teuvo_Kohonen Kohonen]的[https://en.wikipedia.org/wiki/Teuvo_Kohonen 自组织映射]的早期研究。例如, 在[https://en.wikipedia.org/wiki/Sparse_distributed_memory 稀疏分布式记忆]或[https://en.wikipedia.org/wiki/Hierarchical_temporal_memory 层级空间记忆]中，神经网络编码的模式被用于[https://en.wikipedia.org/wiki/Content-addressable_memory 可寻址内容的记忆]的地址，使用“神经元”本质上作为地址 [https://en.wikipedia.org/wiki/Encoder 编码器]和[https://en.wikipedia.org/wiki/Binary_decoder 解码器]。然而早期这种记忆的控制器不可微。

====LSTM相关的可微记忆结构（LSTM-related differentiable memory structures） ====

除了[https://en.wikipedia.org/wiki/Long_short-term_memory 长短期记忆](LSTM), 其他方法也在循环函数中加入可微记忆，例如：

−

* 交替记忆网络的可微的推和弹动作，称为神经叠加机器<ref name="S. Das, C.L. Giles p. 79">S. Das, C.L. Giles, G.Z. Sun, "Learning Context Free Grammars: Limitations of a Recurrent Neural Network with an External Stack Memory," Proc. 14th Annual Conf. of the Cog. Sci. Soc., p. 79, 1992.</ref><ref name="Mozer, M. C. 1993 pp. 863-870">{{Cite ~~web~~|url=https://papers.nips.cc/paper/626-a-connectionist-symbol-manipulator-that-discovers-the-structure-of-context-free-languages|title=A connectionist symbol manipulator that discovers the structure of context-free languages|last=Mozer|first=M. C.|last2=Das|first2=S.|date=1993~~|website=~~|publisher=NIPS 5|pages=863–870|archive-url=|archive-date=|dead-url=|access-date=}}</ref>

+

* 交替记忆网络的可微的推和弹动作，称为神经叠加机器<ref name="S. Das, C.L. Giles p. 79">S. Das, C.L. Giles, G.Z. Sun, "Learning Context Free Grammars: Limitations of a Recurrent Neural Network with an External Stack Memory," Proc. 14th Annual Conf. of the Cog. Sci. Soc., p. 79, 1992.</ref><ref name="Mozer, M. C. 1993 pp. 863-870">{{Cite journal|url=https://papers.nips.cc/paper/626-a-connectionist-symbol-manipulator-that-discovers-the-structure-of-context-free-languages|title=A connectionist symbol manipulator that discovers the structure of context-free languages|last=Mozer|first=M. C.|last2=Das|first2=S.|date=1993|publisher=NIPS 5|pages=863–870|archive-url=|archive-date=|dead-url=|access-date=}}</ref>

* 控制网络的外部可微存储在其他网络的快速幂中的记忆网络。<ref name="ReferenceC">{{cite journal|year=1992|title=Learning to control fast-weight memories: An alternative to recurrent nets|url=|journal=Neural Computation|volume=4|issue=1|pages=131–139|last1=Schmidhuber|first1=J.}}</ref>

* LSTM遗忘门<ref name="F. Gers, N. Schraudolph 2002">{{cite journal|last2=Schraudolph|first2=N.|last3=Schmidhuber|first3=J.|date=|year=2002|title=Learning precise timing with LSTM recurrent networks|url=http://jmlr.org/papers/volume3/gers02a/gers02a.pdf|journal=JMLR|volume=3|issue=|pages=115–143|via=|last1=Gers|first1=F.}}</ref>

−

* 带用于寻址和在可微样式（内部存储）快速操作RNN自身权重的特殊输出单元的自我参照的RNN。<ref name="J. Schmidhuber pages 191-195">{{Cite ~~conference~~|author=[https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber Jürgen Schmidhuber]|title=An introspective network that can learn to run its own weight change algorithm|~~booktitle~~=In Proc. of the Intl. Conf. on Artificial Neural Networks, Brighton|pages=191–195|publisher=IEE|year=1993|url=ftp://ftp.idsia.ch/pub/juergen/iee93self.ps.gz}}</ref><ref name="Hochreiter, Sepp 2001">{{cite journal|last2=Younger|first2=A. Steven|last3=Conwell|first3=Peter R.|date=|year=2001|title=Learning to Learn Using Gradient Descent|url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.5.323|journal=ICANN|volume=2130|issue=|pages=87–94|via=|last1=Hochreiter|first1=Sepp}}</ref>

+

* 带用于寻址和在可微样式（内部存储）快速操作RNN自身权重的特殊输出单元的自我参照的RNN。<ref name="J. Schmidhuber pages 191-195">{{Cite journal|author=[https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber Jürgen Schmidhuber]|title=An introspective network that can learn to run its own weight change algorithm|journal=In Proc. of the Intl. Conf. on Artificial Neural Networks, Brighton|pages=191–195|publisher=IEE|year=1993|url=ftp://ftp.idsia.ch/pub/juergen/iee93self.ps.gz}}</ref><ref name="Hochreiter, Sepp 2001">{{cite journal|last2=Younger|first2=A. Steven|last3=Conwell|first3=Peter R.|date=|year=2001|title=Learning to Learn Using Gradient Descent|url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.5.323|journal=ICANN|volume=2130|issue=|pages=87–94|via=|last1=Hochreiter|first1=Sepp}}</ref>

* 学习带无界记忆的转换。<ref name="Grefenstette, Edward 1506">Grefenstette, Edward, et al. [https://arxiv.org/pdf/1506.02516.pdf "Learning to Transduce with Unbounded Memory."](2015).</ref>

第294行：第294行：

===== 神经图灵机（Neural Turing machines） =====

神经图灵机<ref name="Graves, Alex 14102">Graves, Alex, Greg Wayne, and Ivo Danihelka. [https://arxiv.org/abs/1410.5401 "Neural Turing Machines."] (2014).</ref>将LSTM网络与外部记忆资源结合，这样他们可以通过注意过程相互影响。这种组合系统和[https://en.wikipedia.org/wiki/Turing_machine 图灵机]相似但是端到端可微，允许使用[https://en.wikipedia.org/wiki/Gradient_descent 梯度下降]有效训练。初步结果表明神经图灵机可以推断简单算法，如复制，排序和从输入输出例子的联想回忆。

−

[https://en.wikipedia.org/wiki/Differentiable_neural_computer 可微神经计算机]（DNC）是一个NTM的延伸。他们在序列处理任务中表现超过神经图灵机，长短期记忆系统和记忆网络。<ref name=":02">{{Cite journal|url=https://www.wired.co.uk/article/deepmind-ai-tube-london-underground|title=DeepMind's AI learned to ride the London Underground using human-like reason and memory|last=Burgess|first=Matt|journal=WIRED UK|language=en-GB|access-date=2016-10-19}}</ref><ref>{{Cite journal|url=https://www.pcmag.com/news/348701/deepmind-ai-learns-to-navigate-london-tube|title=DeepMind AI 'Learns' to Navigate London Tube|journal=PCMAG|access-date=2016-10-19}}</ref><ref>{{Cite ~~web~~|url=https://techcrunch.com/2016/10/13/__trashed-2/|title=DeepMind's differentiable neural computer helps you navigate the subway with its memory|last=Mannes|first=John|~~website~~=TechCrunch|access-date=2016-10-19}}</ref><ref>{{Cite journal|last=Graves|first=Alex|last2=Wayne|first2=Greg|last3=Reynolds|first3=Malcolm|last4=Harley|first4=Tim|last5=Danihelka|first5=Ivo|last6=Grabska-Barwińska|first6=Agnieszka|last7=Colmenarejo|first7=Sergio Gómez|last8=Grefenstette|first8=Edward|last9=Ramalho|first9=Tiago|date=2016-10-12|title=Hybrid computing using a neural network with dynamic external memory|url=http://www.nature.com/nature/journal/vaop/ncurrent/full/nature20101.html|journal=Nature|language=en|volume=538|issue=7626~~|issn=1476-4687~~|pages=471–476}}</ref><ref>{{Cite ~~web~~|url=https://deepmind.com/blog/differentiable-neural-computers/|title=Differentiable neural computers {{!}} DeepMind|~~website~~=DeepMind|access-date=2016-10-19}}</ref>

+

[https://en.wikipedia.org/wiki/Differentiable_neural_computer 可微神经计算机]（DNC）是一个NTM的延伸。他们在序列处理任务中表现超过神经图灵机，长短期记忆系统和记忆网络。<ref name=":02">{{Cite journal|url=https://www.wired.co.uk/article/deepmind-ai-tube-london-underground|title=DeepMind's AI learned to ride the London Underground using human-like reason and memory|last=Burgess|first=Matt|journal=WIRED UK|language=en-GB|access-date=2016-10-19}}</ref><ref>{{Cite journal|url=https://www.pcmag.com/news/348701/deepmind-ai-learns-to-navigate-london-tube|title=DeepMind AI 'Learns' to Navigate London Tube|journal=PCMAG|access-date=2016-10-19}}</ref><ref>{{Cite journal|url=https://techcrunch.com/2016/10/13/__trashed-2/|title=DeepMind's differentiable neural computer helps you navigate the subway with its memory|last=Mannes|first=John|journal=TechCrunch|access-date=2016-10-19}}</ref><ref>{{Cite journal|last=Graves|first=Alex|last2=Wayne|first2=Greg|last3=Reynolds|first3=Malcolm|last4=Harley|first4=Tim|last5=Danihelka|first5=Ivo|last6=Grabska-Barwińska|first6=Agnieszka|last7=Colmenarejo|first7=Sergio Gómez|last8=Grefenstette|first8=Edward|last9=Ramalho|first9=Tiago|date=2016-10-12|title=Hybrid computing using a neural network with dynamic external memory|url=http://www.nature.com/nature/journal/vaop/ncurrent/full/nature20101.html|journal=Nature|language=en|volume=538|issue=7626|pages=471–476}}</ref><ref>{{Cite journal|url=https://deepmind.com/blog/differentiable-neural-computers/|title=Differentiable neural computers {{!}} DeepMind|journal=DeepMind|access-date=2016-10-19}}</ref>

==== 语义哈希（Semantic hashing ）====

第307行：第307行：

==== 编码解码网络（Encoder–decoder networks ）====

−

编码解码框架是基于从高度[https://en.wikipedia.org/wiki/Structured_prediction 结构化]输入到高度结构化输出的映射的神经网络。这种方法在[https://en.wikipedia.org/wiki/Machine_translation 机器翻译]<ref>{{Cite ~~web~~|url=http://www.aclweb.org/anthology/D13-1176|title=Recurrent continuous translation models|last=Kalchbrenner|first=N.|last2=Blunsom|first2=P.|date=2013~~|website=~~|publisher=EMNLP'2013|archive-url=|archive-date=|dead-url=|access-date=}}</ref><ref>{{Cite ~~web~~|url=https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf|title=Sequence to sequence learning with neural networks|last=Sutskever|first=I.|last2=Vinyals|first2=O.|date=2014~~|website=~~|publisher=NIPS'2014|archive-url=|archive-date=|dead-url=|access-date=|last3=Le|first3=Q. V.}}</ref><ref>{{Cite journal|last=Cho|first=K.|last2=van Merrienboer|first2=B.|last3=Gulcehre|first3=C.|last4=Bougares|first4=F.|last5=Schwenk|first5=H.|last6=Bengio|first6=Y.|date=October 2014|title=Learning phrase representations using RNN encoder-decoder for statistical machine translation|journal=Proceedings of the Empiricial Methods in Natural Language Processing|volume=1406|pages=arXiv:1406.1078|via=|url=https://arxiv.org/abs/1406.1078}}</ref>的背景下被提出，它的输入和输出是使用两种自然语言写成的句子。在这个工作中，LSTM RNN或CNN被用作编码机，来总结源语句，这个总结被条件RNN【语言模型】解码来产生翻译。<ref>Cho, Kyunghyun, Aaron Courville, and Yoshua Bengio.[https://arxiv.org/abs/1507.01053 "Describing Multimedia Content using Attention-based Encoder–Decoder Networks." ](2015).</ref> 这些系统共享建立的模块：门限RNN，CNN，和训练的注意机制。

+

编码解码框架是基于从高度[https://en.wikipedia.org/wiki/Structured_prediction 结构化]输入到高度结构化输出的映射的神经网络。这种方法在[https://en.wikipedia.org/wiki/Machine_translation 机器翻译]<ref>{{Cite journal|url=http://www.aclweb.org/anthology/D13-1176|title=Recurrent continuous translation models|last=Kalchbrenner|first=N.|last2=Blunsom|first2=P.|date=2013|publisher=EMNLP'2013|archive-url=|archive-date=|dead-url=|access-date=}}</ref><ref>{{Cite journal|url=https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf|title=Sequence to sequence learning with neural networks|last=Sutskever|first=I.|last2=Vinyals|first2=O.|date=2014|publisher=NIPS'2014|archive-url=|archive-date=|dead-url=|access-date=|last3=Le|first3=Q. V.}}</ref><ref>{{Cite journal|last=Cho|first=K.|last2=van Merrienboer|first2=B.|last3=Gulcehre|first3=C.|last4=Bougares|first4=F.|last5=Schwenk|first5=H.|last6=Bengio|first6=Y.|date=October 2014|title=Learning phrase representations using RNN encoder-decoder for statistical machine translation|journal=Proceedings of the Empiricial Methods in Natural Language Processing|volume=1406|pages=arXiv:1406.1078|via=|url=https://arxiv.org/abs/1406.1078}}</ref>的背景下被提出，它的输入和输出是使用两种自然语言写成的句子。在这个工作中，LSTM RNN或CNN被用作编码机，来总结源语句，这个总结被条件RNN【语言模型】解码来产生翻译。<ref>Cho, Kyunghyun, Aaron Courville, and Yoshua Bengio.[https://arxiv.org/abs/1507.01053 "Describing Multimedia Content using Attention-based Encoder–Decoder Networks." ](2015).</ref> 这些系统共享建立的模块：门限RNN，CNN，和训练的注意机制。

=== 多层核机器（Multilayer kernel machine） ===

匿名用户

http://c2.com/cgi/wiki?$1>Cynthia