更改
跳到导航
跳到搜索
第193行：
第193行： − + − + − + − + − + − LSTM在【自然语言处理】中变得受欢迎。不像之前基于【隐式马尔科夫模型】和相似概念的模型，LSTM可以学习识别【上下文有关语言】。<ref name="gers2001">{{cite journal|last2=Schmidhuber|first2=Jürgen|year=2001|title=LSTM Recurrent Networks Learn Simple Context Free and Context Sensitive Languages|url=|journal=IEEE Transactions on Neural Networks|volume=12|issue=6|pages=1333–1340|doi=10.1109/72.963769|last1=Gers|first1=Felix A.|authorlink2=Jürgen Schmidhuber}}</ref>LSTM提高了机器翻译，<ref>{{cite web | last=Huang | first=Jie | last2=Zhou | first2=Wengang | last3=Zhang | first3=Qilin | last4=Li | first4=Houqiang | last5=Li | first5=Weiping | title=Video-based Sign Language Recognition without Temporal Segmentation | eprint=1801.10111 | date=2018-01-30 | url=https://arxiv.org/pdf/1801.10111.pdf}}</ref><ref name="NIPS2014">{{Cite journal|last=Sutskever|first=L.|last2=Vinyals|first2=O.|last3=Le|first3=Q.|date=2014|title=Sequence to Sequence Learning with Neural Networks|url=https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf|journal=NIPS'14 Proceedings of the 27th International Conference on Neural Information Processing Systems |volume=2 |pages=3104–3112 |bibcode=2014arXiv1409.3215S |arxiv=1409.3215 |class=cs.CL}}</ref>【语言建模】<ref name="vinyals2016">{{cite arxiv|last=Jozefowicz|first=Rafal|last2=Vinyals|first2=Oriol|last3=Schuster|first3=Mike|last4=Shazeer|first4=Noam|last5=Wu|first5=Yonghui|date=2016-02-07|title=Exploring the Limits of Language Modeling|eprint=1602.02410|class=cs.CL}}</ref>和多语言语言处理。<ref name="gillick2015">{{cite arxiv|last=Gillick|first=Dan|last2=Brunk|first2=Cliff|last3=Vinyals|first3=Oriol|last4=Subramanya|first4=Amarnag|date=2015-11-30|title=Multilingual Language Processing From Bytes|eprint=1512.00103|class=cs.CL}}</ref>与CNN结合的LSTM提高了自动图像字幕标记。<ref name="vinyals2015">{{cite arxiv|last=Vinyals|first=Oriol|last2=Toshev|first2=Alexander|last3=Bengio|first3=Samy|last4=Erhan|first4=Dumitru|date=2014-11-17|title=Show and Tell: A Neural Image Caption Generator|eprint=1411.4555|class=cs.CV}}</ref>+
→变体
== 变体 ==
== 变体 ==
=== 数据处理的群方法（Group method of data handling） ===
=== 数据处理的群方法（Group method of data handling） ===
数据处理的群方法(GMDH) 突出了全自动结构和参数化模型优化。结点激活函数是允许加法和乘法操作的【Kolmogorov】-Gabor多项式。它使用八层的深度前馈多层感知机<ref name="ivak1971">{{Cite journal|last=Ivakhnenko|first=Alexey|date=1971|title=Polynomial theory of complex systems|url=|journal=IEEE Transactions on Systems, Man and Cybernetics (4)|issue=4|pages=364–378|doi=10.1109/TSMC.1971.4308320|pmid=|access-date=}}</ref> ，是一个逐层增长的【监督学习】网络，其中每层使用【回归分析】训练。使用验证集检测无用的项，通过【正则化】消除。结果网络的尺寸和深度取决于任务。<ref name="kondo2008">{{cite journal|last2=Ueno|first2=J.|date=|year=2008|title=Multi-layered GMDH-type neural network self-selecting optimum neural network architecture and its application to 3-dimensional medical image recognition of blood vessels|url=https://www.researchgate.net/publication/228402366_GMDH-Type_Neural_Network_Self-Selecting_Optimum_Neural_Network_Architecture_and_Its_Application_to_3-Dimensional_Medical_Image_Recognition_of_the_Lungs|journal=International Journal of Innovative Computing, Information and Control|volume=4|issue=1|pages=175–187|via=|last1=Kondo|first1=T.}}</ref>
数据处理的群方法(GMDH) 突出了全自动结构和参数化模型优化。结点激活函数是允许加法和乘法操作的[https://en.wikipedia.org/wiki/Andrey_Kolmogorov Kolmogorov]-Gabor多项式。它使用八层的深度前馈多层感知机<ref name="ivak1971">{{Cite journal|last=Ivakhnenko|first=Alexey|date=1971|title=Polynomial theory of complex systems|url=|journal=IEEE Transactions on Systems, Man and Cybernetics (4)|issue=4|pages=364–378|doi=10.1109/TSMC.1971.4308320|pmid=|access-date=}}</ref> ，是一个逐层增长的[https://en.wikipedia.org/wiki/Supervised_learning 监督学习]网络，其中每层使用[https://en.wikipedia.org/wiki/Regression_analysis 回归分析]训练。使用验证集检测无用的项，通过[https://en.wikipedia.org/wiki/Regression_analysis 正则化]消除。结果网络的尺寸和深度取决于任务。<ref name="kondo2008">{{cite journal|last2=Ueno|first2=J.|date=|year=2008|title=Multi-layered GMDH-type neural network self-selecting optimum neural network architecture and its application to 3-dimensional medical image recognition of blood vessels|url=https://www.researchgate.net/publication/228402366_GMDH-Type_Neural_Network_Self-Selecting_Optimum_Neural_Network_Architecture_and_Its_Application_to_3-Dimensional_Medical_Image_Recognition_of_the_Lungs|journal=International Journal of Innovative Computing, Information and Control|volume=4|issue=1|pages=175–187|via=|last1=Kondo|first1=T.}}</ref>
=== 卷积神经网络（Convolutional neural networks） ===
=== 卷积神经网络（Convolutional neural networks） ===
卷积神经网络 (CNN) 是一类深度前馈网络，由一或多层【卷积】层和位于其上的全连接层（与典型ANN中的匹配）组成。它使用相等权重和池化层。特别地，最大池化<ref name="Weng19932"/>通常通过Fukushima的卷积结构组织。<ref name="FUKU1980">{{cite journal|year=1980|title=Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position|url=|journal=Biol. Cybern.|volume=36|issue=4|pages=193–202|doi=10.1007/bf00344251|pmid=7370364|last1=Fukushima|first1=K.}}</ref>这种结构允许CNN利用输入数据的2D结构
卷积神经网络 (CNN) 是一类深度前馈网络，由一或多层[https://en.wikipedia.org/wiki/Convolution 卷积]层和位于其上的全连接层（与典型ANN中的匹配）组成。它使用相等权重和池化层。特别地，最大池化<ref name="Weng19932"/>通常通过Fukushima的卷积结构组织。<ref name="FUKU1980">{{cite journal|year=1980|title=Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position|url=|journal=Biol. Cybern.|volume=36|issue=4|pages=193–202|doi=10.1007/bf00344251|pmid=7370364|last1=Fukushima|first1=K.}}</ref>这种结构允许CNN利用输入数据的2D结构
CNN适合处理视觉和其他二维数据<ref name="LECUN1989">LeCun ''et al.'', "Backpropagation Applied to Handwritten Zip Code Recognition," ''Neural Computation'', 1, pp. 541–551, 1989.</ref><ref name="lecun2016slides">[[Yann LeCun]] (2016). Slides on Deep Learning [https://indico.cern.ch/event/510372/ Online]</ref>，它们在图像和语音应用中展示出了优秀的结果。它们可以被标准反向传播训练。CNN比其他普通的深度前馈神经网络更容易训练且有更少的需要估计的参数。<ref name="STANCNN">{{cite web|url=http://ufldl.stanford.edu/tutorial/supervised/ConvolutionalNeuralNetwork/|title=Unsupervised Feature Learning and Deep Learning Tutorial|publisher=}}</ref> 计算机视觉中应用的例子包括【DeepDream】<ref name="deepdream">{{cite journal|last2=Liu|first2=Wei|last3=Jia|first3=Yangqing|last4=Sermanet|first4=Pierre|last5=Reed|first5=Scott|last6=Anguelov|first6=Dragomir|last7=Erhan|first7=Dumitru|last8=Vanhoucke|first8=Vincent|last9=Rabinovich|first9=Andrew|date=|year=2014|title=Going Deeper with Convolutions|url=|journal=Computing Research Repository|volume=|pages=1|arxiv=1409.4842|doi=10.1109/CVPR.2015.7298594|via=|first1=Christian|last1=Szegedy|isbn=978-1-4673-6964-0}}</ref>和【机器人导航】<ref>{{cite journal | last=Ran | first=Lingyan | last2=Zhang | first2=Yanning | last3=Zhang | first3=Qilin | last4=Yang | first4=Tao | title=Convolutional Neural Network-Based Robot Navigation Using Uncalibrated Spherical Images | journal=Sensors | publisher=MDPI AG | volume=17 | issue=6 | date=2017-06-12 | issn=1424-8220 | doi=10.3390/s17061341 | page=1341 | url=https://qilin-zhang.github.io/_pages/pdfs/sensors-17-01341.pdf}}</ref>
CNN适合处理视觉和其他二维数据<ref name="LECUN1989">LeCun ''et al.'', "Backpropagation Applied to Handwritten Zip Code Recognition," ''Neural Computation'', 1, pp. 541–551, 1989.</ref><ref name="lecun2016slides">[[Yann LeCun]] (2016). Slides on Deep Learning [https://indico.cern.ch/event/510372/ Online]</ref>，它们在图像和语音应用中展示出了优秀的结果。它们可以被标准反向传播训练。CNN比其他普通的深度前馈神经网络更容易训练且有更少的需要估计的参数。<ref name="STANCNN">{{cite web|url=http://ufldl.stanford.edu/tutorial/supervised/ConvolutionalNeuralNetwork/|title=Unsupervised Feature Learning and Deep Learning Tutorial|publisher=}}</ref> 计算机视觉中应用的例子包括[https://en.wikipedia.org/wiki/DeepDream DeepDream]<ref name="deepdream">{{cite journal|last2=Liu|first2=Wei|last3=Jia|first3=Yangqing|last4=Sermanet|first4=Pierre|last5=Reed|first5=Scott|last6=Anguelov|first6=Dragomir|last7=Erhan|first7=Dumitru|last8=Vanhoucke|first8=Vincent|last9=Rabinovich|first9=Andrew|date=|year=2014|title=Going Deeper with Convolutions|url=|journal=Computing Research Repository|volume=|pages=1|arxiv=1409.4842|doi=10.1109/CVPR.2015.7298594|via=|first1=Christian|last1=Szegedy|isbn=978-1-4673-6964-0}}</ref>和[https://en.wikipedia.org/wiki/Robot_navigation 机器人导航]<ref>{{cite journal | last=Ran | first=Lingyan | last2=Zhang | first2=Yanning | last3=Zhang | first3=Qilin | last4=Yang | first4=Tao | title=Convolutional Neural Network-Based Robot Navigation Using Uncalibrated Spherical Images | journal=Sensors | publisher=MDPI AG | volume=17 | issue=6 | date=2017-06-12 | issn=1424-8220 | doi=10.3390/s17061341 | page=1341 | url=https://qilin-zhang.github.io/_pages/pdfs/sensors-17-01341.pdf}}</ref>
。
。
===长短期记忆（ Long short-term memory） ===
===长短期记忆（ Long short-term memory） ===
长短期记忆 (LSTM) 网络是避免了【梯度消失问题】。<ref name=":03">{{Cite journal|last=Hochreiter|first=Sepp|author-link=Sepp Hochreiter|last2=Schmidhuber|first2=Jürgen|author-link2=Jürgen Schmidhuber|date=1997-11-01|title=Long Short-Term Memory|url=http://www.mitpressjournals.org/doi/10.1162/neco.1997.9.8.1735|journal=Neural Computation|volume=9|issue=8|pages=1735–1780|doi=10.1162/neco.1997.9.8.1735|issn=0899-7667|via=}}</ref> LSTM通常被称为遗忘门的循环门扩展<ref name=":10">{{Cite web|url=https://www.researchgate.net/publication/220320057_Learning_Precise_Timing_with_LSTM_Recurrent_Networks|title=Learning Precise Timing with LSTM Recurrent Networks (PDF Download Available)|website=ResearchGate|language=en|access-date=2017-06-13|pp=115–143}}</ref>。 LSTM网络避免了反向传播误差的消失或爆炸。<ref name="HOCH19912"/> 误差可以通过在空间展开的LSTM中的无限制的虚层反向回流 。也就是说，LSTM可以学习“非常深的学习”任务，<ref name="SCHIDHUB2" />这些任务需要记住上千甚至上百万离散时间步前的事件。问题特殊的LSTM形态的拓扑结构可以成为进化的LSTM，<ref>{{Cite journal|last=Bayer|first=Justin|last2=Wierstra|first2=Daan|last3=Togelius|first3=Julian|last4=Schmidhuber|first4=Jürgen|date=2009-09-14|title=Evolving Memory Cell Structures for Sequence Learning|url=https://link.springer.com/chapter/10.1007/978-3-642-04277-5_76|journal=Artificial Neural Networks – ICANN 2009|volume=5769|language=en|publisher=Springer, Berlin, Heidelberg|pages=755–764|doi=10.1007/978-3-642-04277-5_76|series=Lecture Notes in Computer Science|isbn=978-3-642-04276-8}}</ref> 能处理长延迟和混合高低频成分的信号。
长短期记忆 (LSTM) 网络是避免了[https://en.wikipedia.org/wiki/Vanishing_gradient_problem 梯度消失问题]。<ref name=":03">{{Cite journal|last=Hochreiter|first=Sepp|author-link=Sepp Hochreiter|last2=Schmidhuber|first2=Jürgen|author-link2=Jürgen Schmidhuber|date=1997-11-01|title=Long Short-Term Memory|url=http://www.mitpressjournals.org/doi/10.1162/neco.1997.9.8.1735|journal=Neural Computation|volume=9|issue=8|pages=1735–1780|doi=10.1162/neco.1997.9.8.1735|issn=0899-7667|via=}}</ref> LSTM通常被称为遗忘门的循环门扩展<ref name=":10">{{Cite web|url=https://www.researchgate.net/publication/220320057_Learning_Precise_Timing_with_LSTM_Recurrent_Networks|title=Learning Precise Timing with LSTM Recurrent Networks (PDF Download Available)|website=ResearchGate|language=en|access-date=2017-06-13|pp=115–143}}</ref>。 LSTM网络避免了反向传播误差的消失或爆炸。<ref name="HOCH19912"/> 误差可以通过在空间展开的LSTM中的无限制的虚层反向回流 。也就是说，LSTM可以学习“非常深的学习”任务，<ref name="SCHIDHUB2" />这些任务需要记住上千甚至上百万离散时间步前的事件。问题特殊的LSTM形态的拓扑结构可以成为进化的LSTM，<ref>{{Cite journal|last=Bayer|first=Justin|last2=Wierstra|first2=Daan|last3=Togelius|first3=Julian|last4=Schmidhuber|first4=Jürgen|date=2009-09-14|title=Evolving Memory Cell Structures for Sequence Learning|url=https://link.springer.com/chapter/10.1007/978-3-642-04277-5_76|journal=Artificial Neural Networks – ICANN 2009|volume=5769|language=en|publisher=Springer, Berlin, Heidelberg|pages=755–764|doi=10.1007/978-3-642-04277-5_76|series=Lecture Notes in Computer Science|isbn=978-3-642-04276-8}}</ref> 能处理长延迟和混合高低频成分的信号。
大量LSTM RNN<ref>{{Cite journal|last=Fernández|first=Santiago|last2=Graves|first2=Alex|last3=Schmidhuber|first3=Jürgen|date=2007|title=Sequence labelling in structured domains with hierarchical recurrent neural networks|url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.79.1887|journal=In Proc. 20th Int. Joint Conf. on Artificial In℡ligence, Ijcai 2007|pages=774–779}}</ref> 使用联结主义时间分类（CTC）训练，<ref name=":12">{{Cite journal|last=Graves|first=Alex|last2=Fernández|first2=Santiago|last3=Gomez|first3=Faustino|date=2006|title=Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks|url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.75.6306|journal=In Proceedings of the International Conference on Machine Learning, ICML 2006|pages=369–376}}</ref> 给定相应输入序列，可以找到一个最大化训练集中标记序列概率的RNN权重矩阵。CTC达到了校准和识别。
大量LSTM RNN<ref>{{Cite journal|last=Fernández|first=Santiago|last2=Graves|first2=Alex|last3=Schmidhuber|first3=Jürgen|date=2007|title=Sequence labelling in structured domains with hierarchical recurrent neural networks|url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.79.1887|journal=In Proc. 20th Int. Joint Conf. on Artificial In℡ligence, Ijcai 2007|pages=774–779}}</ref> 使用联结主义时间分类（CTC）训练，<ref name=":12">{{Cite journal|last=Graves|first=Alex|last2=Fernández|first2=Santiago|last3=Gomez|first3=Faustino|date=2006|title=Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks|url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.75.6306|journal=In Proceedings of the International Conference on Machine Learning, ICML 2006|pages=369–376}}</ref> 给定相应输入序列，可以找到一个最大化训练集中标记序列概率的RNN权重矩阵。CTC达到了校准和识别。
2003，LSTM开始在传统语音识别器中具有竞争力。<ref name="graves2003">{{Cite web|url=Ftp://ftp.idsia.ch/pub/juergen/bioadit2004.pdf|title=Biologically Plausible Speech Recognition with LSTM Neural Nets|last=Graves|first=Alex|last2=Eck|first2=Douglas|date=2003|website=1st Intl. Workshop on Biologically Inspired Approaches to Advanced Information Technology, Bio-ADIT 2004, Lausanne, Switzerland|pages=175–184|archive-url=|archive-date=|dead-url=|access-date=|last3=Beringer|first3=Nicole|last4=Schmidhuber|first4=Jürgen|authorlink4=Jürgen Schmidhuber}}</ref>2007，与CTC的结合在语音数据上达到了第一个良好的结果。<ref name="fernandez2007keyword">{{Cite journal|last=Fernández|first=Santiago|last2=Graves|first2=Alex|last3=Schmidhuber|first3=Jürgen|date=2007|title=An Application of Recurrent Neural Networks to Discriminative Keyword Spotting|url=http://dl.acm.org/citation.cfm?id=1778066.1778092|journal=Proceedings of the 17th International Conference on Artificial Neural Networks|series=ICANN'07|location=Berlin, Heidelberg|publisher=Springer-Verlag|pages=220–229|isbn=3540746935}}</ref>2009，一个CTC训练的LSTM成为第一个赢得模式识别比赛的RNN，当它赢得了几个连笔【手写识别】比赛。<ref name="SCHIDHUB2" /><ref name="graves20093"/>2014，【百度】使用CTC训练的RNN打破了Switchboard Hub5'00语音识别在基准测试数据集上的表现，而没有使用传统语音处理方法。<ref name="hannun2014">{{cite arxiv|last=Hannun|first=Awni|last2=Case|first2=Carl|last3=Casper|first3=Jared|last4=Catanzaro|first4=Bryan|last5=Diamos|first5=Greg|last6=Elsen|first6=Erich|last7=Prenger|first7=Ryan|last8=Satheesh|first8=Sanjeev|last9=Sengupta|first9=Shubho|date=2014-12-17|title=Deep Speech: Scaling up end-to-end speech recognition|eprint=1412.5567|class=cs.CL}}</ref> LSTM也提高了大量词汇语音识别，<ref name="sak2014">{{Cite web|url=https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43905.pdf|title=Long Short-Term Memory recurrent neural network architectures for large scale acoustic modeling|last=Sak|first=Hasim|last2=Senior|first2=Andrew|date=2014|website=|archive-url=|archive-date=|dead-url=|access-date=|last3=Beaufays|first3=Francoise}}</ref><ref name="liwu2015">{{cite arxiv|last=Li|first=Xiangang|last2=Wu|first2=Xihong|date=2014-10-15|title=Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition|eprint=1410.4281|class=cs.CL}}</ref>文本到语音合成，<ref>{{Cite web|url=https://www.researchgate.net/publication/287741874_TTS_synthesis_with_bidirectional_LSTM_based_Recurrent_Neural_Networks|title=TTS synthesis with bidirectional LSTM based Recurrent Neural Networks|last=Fan|first=Y.|last2=Qian|first2=Y.|date=2014|website=ResearchGate|language=en|archive-url=|archive-date=|dead-url=|access-date=2017-06-13|last3=Xie|first3=F.|last4=Soong|first4=F. K.}}</ref> 对谷歌安卓<ref name="scholarpedia2"/><ref name="zen2015">{{Cite web|url=https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43266.pdf|title=Unidirectional Long Short-Term Memory Recurrent Neural Network with Recurrent Output Layer for Low-Latency Speech Synthesis|last=Zen|first=Heiga|last2=Sak|first2=Hasim|date=2015|website=Google.com|publisher=ICASSP|pages=4470–4474|archive-url=|archive-date=|dead-url=|access-date=}}</ref>和真实图片的传声头像。<ref name="fan2015">{{Cite journal|last=Fan|first=Bo|last2=Wang|first2=Lijuan|last3=Soong|first3=Frank K.|last4=Xie|first4=Lei|date=2015|title=Photo-Real Talking Head with Deep Bidirectional LSTM|url=https://www.microsoft.com/en-us/research/wp-content/uploads/2015/04/icassp2015_fanbo_1009.pdf|journal=Proceedings of ICASSP|volume=|pages=|via=}}</ref>2015，谷歌的语音识别通过CTC训练的LSTM提高了49%的性能。<ref name="sak2015">{{Cite web|url=http://googleresearch.blogspot.ch/2015/09/google-voice-search-faster-and-more.html|title=Google voice search: faster and more accurate|last=Sak|first=Haşim|last2=Senior|first2=Andrew|date=September 2015|website=|archive-url=|archive-date=|dead-url=|access-date=|last3=Rao|first3=Kanishka|last4=Beaufays|first4=Françoise|last5=Schalkwyk|first5=Johan}}</ref>
2003，LSTM开始在传统语音识别器中具有竞争力。<ref name="graves2003">{{Cite web|url=Ftp://ftp.idsia.ch/pub/juergen/bioadit2004.pdf|title=Biologically Plausible Speech Recognition with LSTM Neural Nets|last=Graves|first=Alex|last2=Eck|first2=Douglas|date=2003|website=1st Intl. Workshop on Biologically Inspired Approaches to Advanced Information Technology, Bio-ADIT 2004, Lausanne, Switzerland|pages=175–184|archive-url=|archive-date=|dead-url=|access-date=|last3=Beringer|first3=Nicole|last4=Schmidhuber|first4=Jürgen|authorlink4=Jürgen Schmidhuber}}</ref>2007，与CTC的结合在语音数据上达到了第一个良好的结果。<ref name="fernandez2007keyword">{{Cite journal|last=Fernández|first=Santiago|last2=Graves|first2=Alex|last3=Schmidhuber|first3=Jürgen|date=2007|title=An Application of Recurrent Neural Networks to Discriminative Keyword Spotting|url=http://dl.acm.org/citation.cfm?id=1778066.1778092|journal=Proceedings of the 17th International Conference on Artificial Neural Networks|series=ICANN'07|location=Berlin, Heidelberg|publisher=Springer-Verlag|pages=220–229|isbn=3540746935}}</ref>2009，一个CTC训练的LSTM成为第一个赢得模式识别比赛的RNN，当它赢得了几个连笔[https://en.wikipedia.org/wiki/Handwriting_recognition 手写识别]比赛。<ref name="SCHIDHUB2" /><ref name="graves20093"/>2014，[https://en.wikipedia.org/wiki/Baidu 百度]使用CTC训练的RNN打破了Switchboard Hub5'00语音识别在基准测试数据集上的表现，而没有使用传统语音处理方法。<ref name="hannun2014">{{cite arxiv|last=Hannun|first=Awni|last2=Case|first2=Carl|last3=Casper|first3=Jared|last4=Catanzaro|first4=Bryan|last5=Diamos|first5=Greg|last6=Elsen|first6=Erich|last7=Prenger|first7=Ryan|last8=Satheesh|first8=Sanjeev|last9=Sengupta|first9=Shubho|date=2014-12-17|title=Deep Speech: Scaling up end-to-end speech recognition|eprint=1412.5567|class=cs.CL}}</ref> LSTM也提高了大量词汇语音识别，<ref name="sak2014">{{Cite web|url=https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43905.pdf|title=Long Short-Term Memory recurrent neural network architectures for large scale acoustic modeling|last=Sak|first=Hasim|last2=Senior|first2=Andrew|date=2014|website=|archive-url=|archive-date=|dead-url=|access-date=|last3=Beaufays|first3=Francoise}}</ref><ref name="liwu2015">{{cite arxiv|last=Li|first=Xiangang|last2=Wu|first2=Xihong|date=2014-10-15|title=Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition|eprint=1410.4281|class=cs.CL}}</ref>文本到语音合成，<ref>{{Cite web|url=https://www.researchgate.net/publication/287741874_TTS_synthesis_with_bidirectional_LSTM_based_Recurrent_Neural_Networks|title=TTS synthesis with bidirectional LSTM based Recurrent Neural Networks|last=Fan|first=Y.|last2=Qian|first2=Y.|date=2014|website=ResearchGate|language=en|archive-url=|archive-date=|dead-url=|access-date=2017-06-13|last3=Xie|first3=F.|last4=Soong|first4=F. K.}}</ref> 对谷歌安卓<ref name="scholarpedia2"/><ref name="zen2015">{{Cite web|url=https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43266.pdf|title=Unidirectional Long Short-Term Memory Recurrent Neural Network with Recurrent Output Layer for Low-Latency Speech Synthesis|last=Zen|first=Heiga|last2=Sak|first2=Hasim|date=2015|website=Google.com|publisher=ICASSP|pages=4470–4474|archive-url=|archive-date=|dead-url=|access-date=}}</ref>和真实图片的传声头像。<ref name="fan2015">{{Cite journal|last=Fan|first=Bo|last2=Wang|first2=Lijuan|last3=Soong|first3=Frank K.|last4=Xie|first4=Lei|date=2015|title=Photo-Real Talking Head with Deep Bidirectional LSTM|url=https://www.microsoft.com/en-us/research/wp-content/uploads/2015/04/icassp2015_fanbo_1009.pdf|journal=Proceedings of ICASSP|volume=|pages=|via=}}</ref>2015，谷歌的语音识别通过CTC训练的LSTM提高了49%的性能。<ref name="sak2015">{{Cite web|url=http://googleresearch.blogspot.ch/2015/09/google-voice-search-faster-and-more.html|title=Google voice search: faster and more accurate|last=Sak|first=Haşim|last2=Senior|first2=Andrew|date=September 2015|website=|archive-url=|archive-date=|dead-url=|access-date=|last3=Rao|first3=Kanishka|last4=Beaufays|first4=Françoise|last5=Schalkwyk|first5=Johan}}</ref>
LSTM在[https://en.wikipedia.org/wiki/Natural_Language_Processing 自然语言处理]中变得受欢迎。不像之前基于[https://en.wikipedia.org/wiki/Hidden_Markov_model 隐式马尔科夫模型]和相似概念的模型，LSTM可以学习识别[https://en.wikipedia.org/wiki/Context-sensitive_languages 上下文有关语言]。<ref name="gers2001">{{cite journal|last2=Schmidhuber|first2=Jürgen|year=2001|title=LSTM Recurrent Networks Learn Simple Context Free and Context Sensitive Languages|url=|journal=IEEE Transactions on Neural Networks|volume=12|issue=6|pages=1333–1340|doi=10.1109/72.963769|last1=Gers|first1=Felix A.|authorlink2=Jürgen Schmidhuber}}</ref>LSTM提高了机器翻译，<ref>{{cite web | last=Huang | first=Jie | last2=Zhou | first2=Wengang | last3=Zhang | first3=Qilin | last4=Li | first4=Houqiang | last5=Li | first5=Weiping | title=Video-based Sign Language Recognition without Temporal Segmentation | eprint=1801.10111 | date=2018-01-30 | url=https://arxiv.org/pdf/1801.10111.pdf}}</ref><ref name="NIPS2014">{{Cite journal|last=Sutskever|first=L.|last2=Vinyals|first2=O.|last3=Le|first3=Q.|date=2014|title=Sequence to Sequence Learning with Neural Networks|url=https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf|journal=NIPS'14 Proceedings of the 27th International Conference on Neural Information Processing Systems |volume=2 |pages=3104–3112 |bibcode=2014arXiv1409.3215S |arxiv=1409.3215 |class=cs.CL}}</ref>[https://en.wikipedia.org/wiki/Language_modeling 语言建模]<ref name="vinyals2016">{{cite arxiv|last=Jozefowicz|first=Rafal|last2=Vinyals|first2=Oriol|last3=Schuster|first3=Mike|last4=Shazeer|first4=Noam|last5=Wu|first5=Yonghui|date=2016-02-07|title=Exploring the Limits of Language Modeling|eprint=1602.02410|class=cs.CL}}</ref>和多语言语言处理。<ref name="gillick2015">{{cite arxiv|last=Gillick|first=Dan|last2=Brunk|first2=Cliff|last3=Vinyals|first3=Oriol|last4=Subramanya|first4=Amarnag|date=2015-11-30|title=Multilingual Language Processing From Bytes|eprint=1512.00103|class=cs.CL}}</ref>与CNN结合的LSTM提高了自动图像字幕标记。<ref name="vinyals2015">{{cite arxiv|last=Vinyals|first=Oriol|last2=Toshev|first2=Alexander|last3=Bengio|first3=Samy|last4=Erhan|first4=Dumitru|date=2014-11-17|title=Show and Tell: A Neural Image Caption Generator|eprint=1411.4555|class=cs.CV}}</ref>