| 第7行： |
第7行： |
| | | | |
| | | | |
| − | @TODO
| + | [[File:Colored neural network.svg|thumb|300px|人工神经网络是一组互相连接的节点，与【大脑】中的大量【神经元】类似。这里，每个圆形节点代表一个【人工神经元】 ，一个箭头表示从一个人工神经元的输出连接到另一个的输入。]] |
| | + | '''人工神经网络'' ('''ANNs''') 或 '''【联结主义】 系统''' 是构成动物【大脑】的【生物神经网络】含糊启发的计算系统。这种系统通过考虑例子“学习”执行任务，通常不需要用任何具体的任务规则编程。例如，在【图像识别】，他们可能通过分析被手动【标记】成“有猫”和“无猫”的示例图像来学习识别包含猫的图像并利用结果识别其他图像中的猫。他们不需要任何关于猫的先验知识做这个任务，例如它们有毛，尾巴，胡须和猫科动物的脸。它们自动地从它们处理的学习材料中产生识别特征。 |
| | | | |
| − | [[File:Colored neural network.svg|thumb|300px|An artificial neural network is an interconnected group of nodes, akin to the vast network of [[neuron]]s in a [[brain]]. Here, each circular node represents an [[artificial neuron]] and an arrow represents a connection from the output of one artificial neuron to the input of another.]]
| + | 人工神经网络是基于一些称为【人工神经元】的相连单元或节点，它们宽松地模拟了一个生物的【大脑】中的【神经元】。每个连接好像一个生物的【大脑】中的【突触】，它们可以将信号从一个人工神经元传递到另一个。一个接收信号的人工神经元可以处理它然后发信号给它连接到的额外的人工神经元。 |
| − | '''人工神经网络''' ('''ANNs''') 或 '''[https://en.wikipedia.org/wiki/Connectionism 联结主义]系统''' 是 computing systems vaguely inspired by the [[biological neural network]]s that constitute animal [[brain]]s.<ref>{{Cite web|url=https://www.frontiersin.org/research-topics/4817/artificial-neural-networks-as-models-of-neural-information-processing|title=Artificial Neural Networks as Models of Neural Information Processing {{!}} Frontiers Research Topic|language=en|access-date=2018-02-20}}</ref> Such systems "learn" to perform tasks by considering examples, generally without being programmed with any task-specific rules. For example, in [[image recognition]], they might learn to identify images that contain cats by analyzing example images that have been manually [[Labeled data|labeled]] as "cat" or "no cat" and using the results to identify cats in other images. They do this without any prior knowledge about cats, e.g., that they have fur, tails, whiskers and cat-like faces. Instead, they automatically generate identifying characteristics from the learning material that they process.
| |
| − | | |
| − | An ANN is based on a collection of connected units or nodes called [[artificial neuron]]s which loosely model the [[neuron]]s in a biological [[brain]]. Each connection, like the [[Synapse|synapses]] in a biological [[brain]], can transmit a signal from one artificial neuron to another. An artificial neuron that receives a signal can process it and then signal additional artificial neurons connected to it.
| |
| | | | |
| | 在通常的人工神经网络实现中，在人工神经元之间连接的信号是一个实数，每个人工神经元的输出则由它输入之和的某个非线性函数计算。人工神经元之间的连接成为“边”。人工神经元和边通常有一个随学习进行而调整的[https://en.wikipedia.org/wiki/Weighting 权重]。这个权重增加或减少一个连接处的信号强度。人工神经元可能有一个阈值以便信号只有在聚集的信号越过这个阈值时才发出。一般地，人工神经元聚集成层，不同的层可能对它们的输入执行不同种类的转换。信号从第一层（输入层）传递到最后一层（输出层），可能会多次穿过这些层。 | | 在通常的人工神经网络实现中，在人工神经元之间连接的信号是一个实数，每个人工神经元的输出则由它输入之和的某个非线性函数计算。人工神经元之间的连接成为“边”。人工神经元和边通常有一个随学习进行而调整的[https://en.wikipedia.org/wiki/Weighting 权重]。这个权重增加或减少一个连接处的信号强度。人工神经元可能有一个阈值以便信号只有在聚集的信号越过这个阈值时才发出。一般地，人工神经元聚集成层，不同的层可能对它们的输入执行不同种类的转换。信号从第一层（输入层）传递到最后一层（输出层），可能会多次穿过这些层。 |
| 第20行： |
第18行： |
| | | | |
| | == 历史 == | | == 历史 == |
| − | [[Warren McCulloch]] and [[Walter Pitts]]<ref>{{cite journal|last=McCulloch|first=Warren|author2=Walter Pitts|title=A Logical Calculus of Ideas Immanent in Nervous Activity|journal=Bulletin of Mathematical Biophysics|year=1943|volume=5|pages=115–133|doi=10.1007/BF02478259|issue=4}}</ref> (1943) created a computational model for neural networks based on [[mathematics]] and [[algorithm]]s called threshold logic. This model paved the way for neural network research to split into two approaches. One approach focused on biological processes in the brain while the other focused on the application of neural networks to [[artificial intelligence]]. This work led to work on nerve networks and their link to [[Finite state machine|finite automata]].<ref>{{Cite news|url=https://www.degruyter.com/view/books/9781400882618/9781400882618-002/9781400882618-002.xml|title=Representation of Events in Nerve Nets and Finite Automata|last=Kleene|first=S.C.|date=|work=Annals of Mathematics Studies|access-date=2017-06-17|archive-url=|archive-date=|dead-url=|publisher=Princeton University Press|year=1956|issue=34|pages=3–41|language=en}}</ref>
| + | 【Warren McCulloch】 和 【Walter Pitts】<ref>构造了一个关于基于【数学】和【算法】的神经网络计算模型，称为阈值逻辑。这个模型为神经网络研究铺平了分为两边的道路。一个关注大脑中的生物学过程，而另一个关注神经网络向【人工智能】的应用。这个工作引领了神经网络的工作以及他们与【有限状态机（Finite state machine）】的联系<ref> |
| | | | |
| | === 赫布学习（Hebbian learning）=== | | === 赫布学习（Hebbian learning）=== |
| | + | 在19世纪40年代晚期，【】<> 基于【神经可塑性】的机制构造了一个学习假设，被称为【赫布学习】。赫布学习是【无监督学习（unsupervised learning）】。这形成了【长程增强效应】模型。在1948年，研究者开始将这些想法和【B类图灵机】应用到计算模型上。 |
| | | | |
| − | In the late 1940s, [[Donald O. Hebb|D.O. Hebb]]<ref>{{cite book|url={{google books |plainurl=y |id=ddB4AgAAQBAJ}}|title=The Organization of Behavior|last=Hebb|first=Donald|publisher=Wiley|year=1949|isbn=978-1-135-63190-1|location=New York|pages=}}</ref> created a learning hypothesis based on the mechanism of [[Neuroplasticity|neural plasticity]] that became known as [[Hebbian learning]]. Hebbian learning is [[unsupervised learning]]. This evolved into models for [[long term potentiation]]. Researchers started applying these ideas to computational models in 1948 with [[unorganized machine|Turing's B-type machines]].
| + | Farley 和 【】<ref>首先使用计算机器，后来称作“计算器”，来模拟赫布网络。其他神经网络计算机器被【】创造.<ref> |
| | | | |
| − | Farley and [[Wesley A. Clark|Clark]]<ref>{{cite journal|last=Farley|first=B.G.|author2=W.A. Clark|title=Simulation of Self-Organizing Systems by Digital Computer|journal=IRE Transactions on Information Theory|year=1954|volume=4|pages=76–84|doi=10.1109/TIT.1954.1057468|issue=4}}</ref> (1954) first used computational machines, then called "calculators", to simulate a Hebbian network. Other neural network computational machines were created by [[Nathaniel Rochester (computer scientist)|Rochester]], Holland, Habit and Duda (1956).<ref>{{cite journal|last=Rochester|first=N. |author2=J.H. Holland |author3=L.H. Habit |author4=W.L. Duda|title=Tests on a cell assembly theory of the action of the brain, using a large digital computer|journal=IRE Transactions on Information Theory|year=1956|volume=2|pages=80–93|doi=10.1109/TIT.1956.1056810|issue=3}}</ref>
| + | 【】<ref>创造了【感知机】，这是一种模式识别算法。Rosenblatt 使用数学符号描述了不在基本感知中的电路，如那时无法被神经网络处理的异或电路。<> |
| | | | |
| − | [[Frank Rosenblatt|Rosenblatt]]<ref>{{cite journal|last=Rosenblatt|first=F.|title=The Perceptron: A Probabilistic Model For Information Storage And Organization In The Brain|journal=Psychological Review|year=1958|volume=65|pages=386–408|doi=10.1037/h0042519|pmid=13602029|issue=6|citeseerx=10.1.1.588.3775}}</ref> (1958) created the [[perceptron]], an algorithm for pattern recognition. With mathematical notation, Rosenblatt described circuitry not in the basic perceptron, such as the [[exclusive-or]] circuit that could not be processed by neural networks at the time.<ref name="Werbos 1975">{{cite book|url={{google books |plainurl=y |id=z81XmgEACAAJ}}|title=Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences|last=Werbos|first=P.J.|publisher=|year=1975|isbn=|location=|pages=}}</ref>
| + | 1959年，【】【】和【】在初级视皮层发现了两种类型的细胞：简单细胞（simple cell）和复杂细胞（complex cell），并基于他们的发现提出了一个生物学模型, |
| | | | |
| − | In 1959, a biological model proposed by [[Nobel laureate]]s [[David H. Hubel|Hubel]] and [[Torsten Wiesel|Wiesel]] was based on their discovery of two types of cells in the [[primary visual cortex]]: [[simple cell]]s and [[complex cell]]s.<ref>{{cite book|url=https://books.google.com/books?id=8YrxWojxUA4C&pg=PA106|title=Brain and visual perception: the story of a 25-year collaboration|publisher=Oxford University Press US|year=2005|isbn=978-0-19-517618-6|page=106|author=David H. Hubel and Torsten N. Wiesel}}</ref>
| + | 第一个有多层的功能网络由【】和【】在1965年发表，它成为了【数据处理的组方法】 |
| | | | |
| − | The first functional networks with many layers were published by [[Alexey Grigorevich Ivakhnenko|Ivakhnenko]] and Lapa in 1965, becoming the [[Group method of data handling|Group Method of Data Handling]].<ref name="SCHIDHUB2">{{cite journal|last=Schmidhuber|first=J.|year=2015|title=Deep Learning in Neural Networks: An Overview|journal=Neural Networks|volume=61|pages=85–117|arxiv=1404.7828|doi=10.1016/j.neunet.2014.09.003|pmid=25462637}}</ref><ref name="ivak1965">{{cite book|url={{google books |plainurl=y |id=FhwVNQAACAAJ}}|title=Cybernetic Predicting Devices|last=Ivakhnenko|first=A. G.|publisher=CCM Information Corporation|year=1973}}</ref><ref name="ivak1967">{{cite book|url={{google books |plainurl=y |id=rGFgAAAAMAAJ}}|title=Cybernetics and forecasting techniques|last2=Grigorʹevich Lapa|first2=Valentin|publisher=American Elsevier Pub. Co.|year=1967|first1=A. G.|last1=Ivakhnenko}}</ref>
| + | 在发现了两个执行神经网络的计算机器关键问题的【】和【】研究的【机器学习】后，神经网络的研究停滞了。第一个是基本感知机不能处理异或电路。第二个是计算机没有足够的处理能力来有效地处理大型神经网络需要的任务。神经网络研究减慢知道计算机达到了远远更好的处理能力。 |
| | | | |
| − | Neural network research stagnated after [[machine learning]] research by [[Marvin Minsky|Minsky]] and [[Seymour Papert|Papert]] (1969),<ref>{{cite book|url={{google books |plainurl=y |id=Ow1OAQAAIAAJ}}|title=Perceptrons: An Introduction to Computational Geometry|last=Minsky|first=Marvin|first2=Seymour|publisher=MIT Press|year=1969|isbn=0-262-63022-2|location=|pages=|author2=Papert}}</ref> who discovered two key issues with the computational machines that processed neural networks. The first was that basic perceptrons were incapable of processing the exclusive-or circuit. The second was that computers didn't have enough processing power to effectively handle the work required by large neural networks. Neural network research slowed until computers achieved far greater processing power.
| + | 更多的【人工智能】专注于【算法】执行的高层面（符号的）模型，以知识体现在如果-那么规则中的【专家系统】为特征。直到19世纪80年代末期，研究扩展到低层面（次符号的）sub-symbolic【机器学习】，以知识体现在一个【认知模型】的参数中为特征。 |
| | | | |
| − | Much of [[artificial intelligence]] had focused on high-level (symbolic) models that are processed by using [[algorithms]], characterized for example by [[expert system]]s with knowledge embodied in ''if-then'' rules, until in the late 1980s research expanded to low-level (sub-symbolic) [[machine learning]], characterized by knowledge embodied in the parameters of a [[cognitive model]].{{Citation needed|date=August 2017|reason=Reliable source needed for the whole sentence}}
| + | === 反向传播（Backpropagation） === |
| | | | |
| − | === 反向传播（Backpropagation） ===
| + | 【werbos】的反向传播算法重新燃起了人们对于神经网络和学习的兴趣，它有效地解决了异或问题并且更普遍地加速了多层网络的训练。反向传播通过修改每个节点的权重，反向分散了贯穿层中的误差项。 |
| − | A key trigger for renewed interest in neural networks and learning was [[Paul Werbos|Werbos]]'s (1975) [[backpropagation]] algorithm that effectively solved the exclusive-or problem and more generally accelerated the training of multi-layer networks. Backpropagation distributed the error term back up through the layers, by modifying the weights at each node.<ref name="Werbos 1975" />
| |
| | | | |
| − | In the mid-1980s, parallel distributed processing became popular under the name [[connectionism]]. [[David E. Rumelhart|Rumelhart]] and [[James McClelland (psychologist)|McClelland]] (1986) described the use of connectionism to simulate neural processes.<ref>{{cite book|url={{google books |plainurl=y |id=davmLgzusB8C}}|title=Parallel Distributed Processing: Explorations in the Microstructure of Cognition|last=Rumelhart|first=D.E|first2=James|publisher=MIT Press|year=1986|isbn=978-0-262-63110-5|location=Cambridge|pages=|author2=McClelland}}</ref>
| + | 在19世纪80年代中期，并行分布处理以【联结主义】的名义变得受欢迎，【】和【】描述了联结主义模拟神经过程的作用。 |
| | | | |
| − | [[Support vector machine]]s and other, much simpler methods such as [[linear classifier]]s gradually overtook neural networks in machine learning popularity. However, using neural networks transformed some domains, such as the prediction of protein structures.<ref>{{cite article|id=Qian1988|title=
| + | 【支持向量机（Support vector machine）】和其他更简单的方法如【线性分类器】在机器学习中的受欢迎程度逐步超过了神经网络。然而，使用神经网络改变了一些领域，例如蛋白质结构的预测。 |
| − | Predicting the secondary structure of globular proteins using neural network models. |last=Qian|first=N.|last2=Sejnowski|first2=T.J.|journal=Journal of Molecular Biology|volume=202|pages=865-884|year=1988}}</ref><ref>{{cite article|id=Rost1993|title=
| |
| − | Prediction of protein secondary structure at better than 70% accuracy |last=Rost|first=B.|last2=Sander|first2=C.|journal=Journal of Molecular Biology|volume=232|pages=584-599|year=1993}}</ref>
| |
| | | | |
| − | In 1992, [[Convolutional neural network#Pooling layer|max-pooling]] was introduced to help with least shift invariance and tolerance to deformation to aid in 3D object recognition.<ref name="Weng1992">J. Weng, N. Ahuja and T. S. Huang, "[http://www.cse.msu.edu/~weng/research/CresceptronIJCNN1992.pdf Cresceptron: a self-organizing neural network which grows adaptively]," ''Proc. International Joint Conference on Neural Networks'', Baltimore, Maryland, vol I, pp. 576–581, June, 1992.</ref><ref name="Weng19932">J. Weng, N. Ahuja and T. S. Huang, "[http://www.cse.msu.edu/~weng/research/CresceptronICCV1993.pdf Learning recognition and segmentation of 3-D objects from 2-D images]," ''Proc. 4th International Conf. Computer Vision'', Berlin, Germany, pp. 121–128, May, 1993.</ref><ref name="Weng1997">J. Weng, N. Ahuja and T. S. Huang, "[http://www.cse.msu.edu/~weng/research/CresceptronIJCV.pdf Learning recognition and segmentation using the Cresceptron]," ''International Journal of Computer Vision'', vol. 25, no. 2, pp. 105–139, Nov. 1997.</ref>
| + | 1992年【最大池化】被引入帮助最小转移不变性和最大容忍性来变形，有助于3D物体识别。2010年，通过【最大池化】训练的反向传播训练被GPU加速，显示出超过其他池化变体的性能。 |
| − | In 2010, Backpropagation training through [[Convolutional neural network#Pooling layer|max-pooling]] was accelerated by GPUs and shown to perform better than other pooling variants.<ref name="Scherer2010">Dominik Scherer, Andreas C. Müller, and Sven Behnke: "[https://www.ais.uni-bonn.de/papers/icann2010_maxpool.pdf Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition]," ''In 20th International Conference Artificial Neural Networks (ICANN)'', pp. 92–101, 2010. {{doi|10.1007/978-3-642-15825-4_10}}.</ref>
| |
| | | | |
| − | The [[vanishing gradient problem]] affects many-layered [[Feedforward neural network|feedforward networks]] that used backpropagation and also [[recurrent neural network]]s (RNNs).<ref name="HOCH19912">S. Hochreiter., "[http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf Untersuchungen zu dynamischen neuronalen Netzen]," ''Diploma thesis. Institut f. Informatik, Technische Univ. Munich. Advisor: J. Schmidhuber'', 1991.</ref><ref name="HOCH2001">{{cite book|url={{google books |plainurl=y |id=NWOcMVA64aAC}}|title=A Field Guide to Dynamical Recurrent Networks|last=Hochreiter|first=S.|last2=et al.|date=15 January 2001|publisher=John Wiley & Sons|year=|isbn=978-0-7803-5369-5|location=|pages=|chapter=Gradient flow in recurrent nets: the difficulty of learning long-term dependencies|editor-last2=Kremer|editor-first2=Stefan C.|editor-first1=John F.|editor-last1=Kolen}}</ref> As errors propagate from layer to layer, they shrink exponentially with the number of layers, impeding the tuning of neuron weights that is based on those errors, particularly affecting deep networks.
| + | 【梯度消失问题】影响使用反向传播的多层【前馈神经网络】 和【循环神经网络】（RNN）。由于误差从一层到另一层传播，它们随着层数指数级缩小，阻碍依赖这些误差的的神经元权重的调整，尤其影响深度网络。 |
| | | | |
| − | To overcome this problem, [[Jürgen Schmidhuber|Schmidhuber]] adopted a multi-level hierarchy of networks (1992) pre-trained one level at a time by [[unsupervised learning]] and fine-tuned by [[backpropagation]].<ref name="SCHMID1992">J. Schmidhuber., "Learning complex, extended sequences using the principle of history compression," ''Neural Computation'', 4, pp. 234–242, 1992.</ref> Behnke (2003) relied only on the sign of the gradient ([[Rprop]])<ref>{{cite book|url=http://www.ais.uni-bonn.de/books/LNCS2766.pdf|title=Hierarchical Neural Networks for Image Interpretation.|publisher=Springer|year=2003|series=Lecture Notes in Computer Science|volume=2766|author=Sven Behnke}}</ref> on problems such as image reconstruction and face localization.
| + | 为了解决这个问题，【】采用了一种多层网络结构，通过【无监督学习】每次预训练一级然后使用反向传播很好地调整。Behnke在例如图像重建和人脸定位中只依赖梯度符号。 |
| | | | |
| − | [[Geoffrey Hinton|Hinton]] et al. (2006) proposed learning a high-level representation using successive layers of binary or real-valued [[latent variable]]s with a [[restricted Boltzmann machine]]<ref name="smolensky1986">{{cite book|url=http://portal.acm.org/citation.cfm?id=104290|title=Parallel Distributed Processing: Explorations in the Microstructure of Cognition|year=1986|editors=D. E. Rumelhart, J. L. McClelland, & the PDP Research Group|volume=1|pages=194–281|chapter=Information processing in dynamical systems: Foundations of harmony theory.|last1=Smolensky|first1=P.|authorlink1=Paul Smolensky}}</ref> to model each layer. Once sufficiently many layers have been learned, the deep architecture may be used as a [[generative model]] by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations.<ref name="hinton2006">{{cite journal|last2=Osindero|first2=S.|last3=Teh|first3=Y.|year=2006|title=A fast learning algorithm for deep belief nets|url=http://www.cs.toronto.edu/~hinton/absps/fastnc.pdf|journal=[[Neural Computation (journal)|Neural Computation]]|volume=18|issue=7|pages=1527–1554|doi=10.1162/neco.2006.18.7.1527|pmid=16764513|last1=Hinton|first1=G. E.|authorlink1=Geoffrey Hinton}}</ref><ref>{{Cite journal|year=2009|title=Deep belief networks|url=http://www.scholarpedia.org/article/Deep_belief_networks|journal=Scholarpedia|volume=4|issue=5|pages=5947|doi=10.4249/scholarpedia.5947|pmc=|pmid=|last1=Hinton|first1=G.|bibcode=2009SchpJ...4.5947H}}</ref> In 2012, [[Andrew Ng|Ng]] and [[Jeff Dean (computer scientist)|Dean]] created a network that learned to recognize higher-level concepts, such as cats, only from watching unlabeled images taken from [[YouTube]] videos.<ref name="ng2012">{{cite arXiv|eprint=1112.6209|first2=Jeff|last2=Dean|title=Building High-level Features Using Large Scale Unsupervised Learning|last1=Ng|first1=Andrew|year=2012|class=cs.LG}}</ref>
| + | 【Hinton】 提出了使用连续层的二进制或潜变量实数【受限玻尔兹曼机】来模拟每一层学习一种高级别表征。一旦很多层被充分学习，这种深度结构可能像【生成模型】一样被使用，通过在下采样（一个古老的方法）模型时从顶层特征激活处复制数据。2012年【Ng】和【Dean】创造了一个只通过看【YouTube】视频中未标记的图像学习识别例如猫这样更高层概念的网络。 |
| | | | |
| − | Earlier challenges in training deep neural networks were successfully addressed with methods such as [[unsupervised pre-training]], while available computing power increased through the use of [[GPU]]s and [[distributed computing]]. Neural networks were deployed on a large scale, particularly in image and visual recognition problems. This became known as "[[deep learning]]".{{citation needed|date=June 2018}}
| + | 在训练深度神经网络中早期的挑战被成功地用【无监督预训练】等方法处理，与此同时可见的计算性能通过GPU和分布计算的使用提升。神经网络被部署在大规模，尤其是在图像和视觉识别问题上。这被称为“【深度学习】” |
| | | | |
| | ===基于硬件的设计（Hardware-based designs）=== | | ===基于硬件的设计（Hardware-based designs）=== |
| − | Computational devices were created in [[CMOS]], for both biophysical simulation and [[neuromorphic computing]]. [[Nanodevice]]s<ref>{{cite journal | last1 = Yang | first1 = J. J. | last2 = Pickett | first2 = M. D. | last3 = Li | first3 = X. M. | last4 = Ohlberg | first4 = D. A. A. | last5 = Stewart | first5 = D. R. | last6 = Williams | first6 = R. S. | year = 2008 | title = Memristive switching mechanism for metal/oxide/metal nanodevices| url = | journal = Nat. Nanotechnol. | volume = 3 | issue = 7| pages = 429–433 | doi = 10.1038/nnano.2008.160 }}</ref> for very large scale [[principal component]]s analyses and [[convolution]] may create a new class of neural computing because they are fundamentally [[Analog signal|analog]] rather than [[Digital data|digital]] (even though the first implementations may use digital devices).<ref>{{cite journal | last1 = Strukov | first1 = D. B. | last2 = Snider | first2 = G. S. | last3 = Stewart | first3 = D. R. | last4 = Williams | first4 = R. S. | year = 2008 | title = The missing memristor found| url = | journal = Nature | volume = 453 | issue = 7191| pages = 80–83 | doi=10.1038/nature06932 | pmid=18451858| bibcode = 2008Natur.453...80S }}</ref> Ciresan and colleagues (2010)<ref name=":3">{{Cite journal|last=Cireşan|first=Dan Claudiu|last2=Meier|first2=Ueli|last3=Gambardella|first3=Luca Maria|last4=Schmidhuber|first4=Jürgen|date=2010-09-21|title=Deep, Big, Simple Neural Nets for Handwritten Digit Recognition|url=http://www.mitpressjournals.org/doi/10.1162/NECO_a_00052|journal=Neural Computation|volume=22|issue=12|pages=3207–3220|doi=10.1162/neco_a_00052|issn=0899-7667}}</ref> in Schmidhuber's group showed that despite the vanishing gradient problem, GPUs makes [[back-propagation]] feasible for many-layered feedforward neural networks.
| + | 用于生物学模拟和神经形态计算的计算设备在【CMOS】创建。用于很大规模【主成分】分析和【卷积】的纳米元件可能创造一类新的神经计算，因为它们根本上是【模拟的】而不是【数字的】（尽管第一个实现使用数字设备）。在Schmidhuber 组的.Ciresan 和 colleagues表明，尽管有梯度消失问题，GPU使【反向传播】对多层前馈神经网络更可行。 |
| − | | |
| − | === Contests ===
| |
| − | Between 2009 and 2012, [[recurrent neural network]]s and deep feedforward neural networks developed in [[Jürgen Schmidhuber|Schmidhuber]]'s research group won eight international competitions in [[pattern recognition]] and [[machine learning]].<ref>[http://www.kurzweilai.net/how-bio-inspired-deep-learning-keeps-winning-competitions 2012 Kurzweil AI Interview] with [[Jürgen Schmidhuber]] on the eight competitions won by his Deep Learning team 2009–2012</ref><ref>{{Cite web|url=http://www.kurzweilai.net/how-bio-inspired-deep-learning-keeps-winning-competitions|title=How bio-inspired deep learning keeps winning competitions {{!}} KurzweilAI|last=|first=|date=|website=www.kurzweilai.net|language=en-US|archive-url=|archive-date=|dead-url=|access-date=2017-06-16}}</ref> For example, the bi-directional and [[multi-dimensional]] [[long short-term memory]] (LSTM)<ref>Graves, Alex; and Schmidhuber, Jürgen; ''[http://www.idsia.ch/~juergen/nips2009.pdf Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks]'', in Bengio, Yoshua; Schuurmans, Dale; Lafferty, John; Williams, Chris K. I.; and Culotta, Aron (eds.), ''Advances in Neural Information Processing Systems 22 (NIPS'22), 7–10 December 2009, Vancouver, BC'', Neural Information Processing Systems (NIPS) Foundation, 2009, pp. 545–552.
| |
| − | </ref><ref name="graves 855" /><ref name="graves20093">{{Cite journal|last2=Schmidhuber|first2=Jürgen|date=2009|editor-last=Bengio|editor-first=Yoshua|title=Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks|url=https://papers.nips.cc/paper/3449-offline-handwriting-recognition-with-multidimensional-recurrent-neural-networks|journal=Neural Information Processing Systems (NIPS) Foundation|volume=|pages=545–552|via=|editor-last2=Schuurmans|editor-first2=Dale|editor-last3=Lafferty|editor-first3=John|editor-last4=Williams|editor-first4=Chris editor-K. I.|editor-last5=Culotta|editor-first5=Aron|last1=Graves|first1=Alex}}</ref><ref>{{Cite journal|last=Graves|first=A.|last2=Liwicki|first2=M.|last3=Fernández|first3=S.|last4=Bertolami|first4=R.|last5=Bunke|first5=H.|last6=Schmidhuber|first6=J.|date=May 2009|title=A Novel Connectionist System for Unconstrained Handwriting Recognition|url=http://ieeexplore.ieee.org/document/4531750/|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|volume=31|issue=5|pages=855–868|doi=10.1109/tpami.2008.137|issn=0162-8828}}</ref> of [[Alex Graves (computer scientist)|Graves]] et al. won three competitions in connected handwriting recognition at the 2009 [[International Conference on Document Analysis and Recognition]] (ICDAR), without any prior knowledge about the three languages to be learned.<ref name="graves20093"/><ref name="graves 855">{{cite journal|last2=Liwicki|first2=M.|last3=Fernandez|first3=S.|last4=Bertolami|first4=R.|last5=Bunke|first5=H.|last6=Schmidhuber|first6=J.|year=2009|title=A Novel Connectionist System for Improved Unconstrained Handwriting Recognition|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|volume=31|issue=5|pages=855–868|doi=10.1109/tpami.2008.137|last1=Graves|first1=A.| url = http://www.idsia.ch/~juergen/tpami_2008.pdf | format = PDF}}</ref>
| |
| | | | |
| − | Ciresan and colleagues won [[pattern recognition]] contests, including the IJCNN 2011 Traffic Sign Recognition Competition,<ref name=":72">{{Cite journal|last=Cireşan|first=Dan|last2=Meier|first2=Ueli|last3=Masci|first3=Jonathan|last4=Schmidhuber|first4=Jürgen|date=August 2012|title=Multi-column deep neural network for traffic sign classification|url=http://www.sciencedirect.com/science/article/pii/S0893608012000524|journal=Neural Networks|series=Selected Papers from IJCNN 2011|volume=32|pages=333–338|doi=10.1016/j.neunet.2012.02.023}}</ref> the ISBI 2012 Segmentation of Neuronal Structures in Electron Microscopy Stacks challenge<ref name=":8"/> and others. Their neural networks were the first pattern recognizers to achieve human-competitive or even superhuman performance<ref name=":92">{{Cite journal|last=Ciresan|first=Dan|last2=Meier|first2=U.|last3=Schmidhuber|first3=J.|date=June 2012|title=Multi-column deep neural networks for image classification|url=http://ieeexplore.ieee.org/document/6248110/|journal=2012 IEEE Conference on Computer Vision and Pattern Recognition|volume=|pages=3642–3649|doi=10.1109/cvpr.2012.6248110|via=|isbn=978-1-4673-1228-8|arxiv=1202.2745}}</ref> on benchmarks such as traffic sign recognition (IJCNN 2012), or the [[MNIST database|MNIST handwritten digits problem]].
| + | === 竞赛 === |
| | + | 在2009~2012年间，【循环神经网络】和【】的研究组发展的深度前馈神经网络赢得了八个在【模式识别】和【机器学习】的国际竞赛。例如，【Graves】的双向和多维【长短期记忆】（LSTM）在2009【文件分析和识别国际会议】上的连笔手写识别中赢得了三个比赛，而没有任何关于要学习的那三种语言的先验知识。 |
| | | | |
| − | Researchers demonstrated (2010) that deep neural networks interfaced to a [[hidden Markov model]] with context-dependent states that define the neural network output layer can drastically reduce errors in large-vocabulary speech recognition tasks such as voice search.
| + | Ciresan 和同事赢得了【模式识别】比赛，包括IJCNN2011交通信号识别比赛，ISBI2012电子显微镜栈中的神经结构分割挑战和其他比赛。他们的神经网络是第一个在基准测试数据集中达到能挑战甚至超过人类表现的模式识别模型。这些基准数据集例如交通信号识别（ijcnn2012）或者【MNIST手写数字问题】 |
| | + | |
| | + | 研究人员演示了深度神经网络接口下的【隐式马尔科夫模型】，它依赖上下文定义神经网络输出层的状态，可以降低在大量词汇语音识别——例如语音搜索——中的误差。【？】 |
| | | | |
| − | GPU-based implementations<ref name=":6">{{Cite journal|last=Ciresan|first=D. C.|last2=Meier|first2=U.|last3=Masci|first3=J.|last4=Gambardella|first4=L. M.|last5=Schmidhuber|first5=J.|date=2011|editor-last=|title=Flexible, High Performance Convolutional Neural Networks for Image Classification|url=http://ijcai.org/papers11/Papers/IJCAI11-210.pdf|journal=International Joint Conference on Artificial Intelligence|volume=|pages=|doi=10.5591/978-1-57735-516-8/ijcai11-210|via=}}</ref> of this approach won many pattern recognition contests, including the IJCNN 2011 Traffic Sign Recognition Competition,<ref name=":72"/> the ISBI 2012 Segmentation of neuronal structures in EM stacks challenge,<ref name=":8">{{Cite book|url=http://papers.nips.cc/paper/4741-deep-neural-networks-segment-neuronal-membranes-in-electron-microscopy-images.pdf|title=Advances in Neural Information Processing Systems 25|last=Ciresan|first=Dan|last2=Giusti|first2=Alessandro|last3=Gambardella|first3=Luca M.|last4=Schmidhuber|first4=Juergen|date=2012|publisher=Curran Associates, Inc.|editor-last=Pereira|editor-first=F.|pages=2843–2851|editor-last2=Burges|editor-first2=C. J. C.|editor-last3=Bottou|editor-first3=L.|editor-last4=Weinberger|editor-first4=K. Q.}}</ref> the [[ImageNet Competition]]<ref name="krizhevsky2012">{{cite journal|last2=Sutskever|first2=Ilya|last3=Hinton|first3=Geoffry|date=2012|title=ImageNet Classification with Deep Convolutional Neural Networks|url=https://www.cs.toronto.edu/~kriz/imagenet_classification_with_deep_convolutional.pdf|journal=NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada|last1=Krizhevsky|first1=Alex}}</ref> and others.
| + | 这种方法基于GPU的实现赢得了很多模式识别竞赛，包括IJCNN2011交通信号识别比赛，ISBI2012电子显微镜栈中的神经结构分割挑战和【ImageNet竞赛】以及其他比赛。 |
| | | | |
| − | Deep, highly nonlinear neural architectures similar to the [[neocognitron]]<ref name="K. Fukushima. Neocognitron 1980">{{cite journal|year=1980|title=Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position|journal=Biological Cybernetics|volume=36|issue=4|pages=93–202|doi=10.1007/BF00344251|pmid=7370364|author=Fukushima, K.}}</ref> and the "standard architecture of vision",<ref>{{cite journal|last2=Poggio|first2=T|year=1999|title=Hierarchical models of object recognition in cortex|journal=Nature Neuroscience|volume=2|issue=11|pages=1019–1025|doi=10.1038/14819|last1=Riesenhuber|first1=M}}</ref> inspired by [[Simple cell|simple]] and [[complex cell]]s, were pre-trained by unsupervised methods by Hinton.<ref name=":1">{{Cite journal|last=Hinton|first=Geoffrey|date=2009-05-31|title=Deep belief networks|url=http://www.scholarpedia.org/article/Deep_belief_networks|journal=Scholarpedia|language=en|volume=4|issue=5|pages=5947|doi=10.4249/scholarpedia.5947|issn=1941-6016|bibcode=2009SchpJ...4.5947H}}</ref><ref name="hinton2006" /> A team from his lab won a 2012 contest sponsored by [[Merck & Co.|Merck]] to design software to help find molecules that might identify new drugs.<ref>{{cite news|url=https://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-deep-learning-a-part-of-artificial-intelligence.html|title=Scientists See Promise in Deep-Learning Programs|last=Markoff|first=John|date=November 23, 2012|author=|newspaper=New York Times}}</ref>
| + | 被【简单】和【复杂细胞】启发的，与【新认知机】相似的深度的高度非线性神经结构和“标准视觉结构”，被Hinton提出的无监督方法预训练。他实验室的一个团队赢得了一个2012年的竞赛，这个竞赛由【Merck】资助来设计可以帮助找到能识别新药物分子的软件。 |
| | | | |
| | === 卷积网络（Convolutional networks） === | | === 卷积网络（Convolutional networks） === |