130
个编辑
更改
跳到导航
跳到搜索
第1行：
第1行： − + − 注意力机制的本质是，从关注全部到关注重点。+ − 比如，当我们阅读文章或观看图像时，视觉系统会自动将注意力集中在关键信息上，而非均匀处理所有细节。+ − − 注意力机制的计算过程包含三个核心组件： − − 查询（Query, Q）：表示当前需要关注什么；键（Key, K）：表示可以被关注的信息；值（Value, V）：表示实际的信息内容 − − 注意力机制计算一个权重分布，该分布表示了查询与每个键之间的相关性。然后，将这些权重与对应的值进行加权求和，得到最终的注意力输出。这个输出可以被视为模型在处理当前查询时，从输入序列中“提取”出的最相关信息。 − 根据不同的分类标准，注意力机制可以分为多种类型。+ − − 按注意力范围分类： − − − − 交叉注意力（Cross-Attention）：不同序列之间的注意力，如编码器-解码器结构中的注意力。 − − − − 工作流程(2,3)：+ + + − （1）查询 (Query), 键(Key), 值 (Value) 的生成+ − + − − − − （2）注意力分数矩阵 + − （3）归一化点乘矩阵+ − − （4）加权输出+ − + − 使用归一化后的注意力分数对V向量进行加权求和，作为自注意力机制的输出。+ − + − 整体流程可以概括为缩放点乘注意力 (Scaled dot-product attention)，用公式表示为：<syntaxhighlight lang="python3" line="1"> 第59行：
第41行： − + − 多头注意力机制将输入序列的表示拆分成多个子空间（头），然后在每个子空间内独立地计算注意力权重，最后将各个子空间的结果拼接起来。多头注意力机制使得模型可以在不同的表示子空间中捕获到不同的上下文信息，增强对复杂数据内在结构的建模能力(3-5)。+ + + 第79行：
第63行： − [[文件:多头注意力机制.png|缩略图|多头注意力机制|替代=|居中]] − + + + + + + + + + + + + + + − * '''自然语言处理:'''+ − ** '''机器翻译:''' 注意力机制使得模型在翻译长句子时，能够关注到源语言句子中与目标语言当前词语相关的部分。通过引入注意力机制，我们将源语言中每个位置的信息都保存下来。在解码过程中生成每一个目标语言的单词时，我们都通过注意力机制直接从源语言的信息中选择相关的信息作为辅助。 − ** '''文本摘要:''' 模型可以学习关注文档中的关键句子或段落，生成更准确、更简洁的摘要。 − ** '''问答系统:''' 注意力机制帮助模型定位问题相关的文档段落，从而给出更准确的答案。 − * '''计算机视觉:''' − ** '''图像描述生成:''' 在生成描述的每一个单词时，注意力机制来选择一些来自于图像的相关信息，关注图像的不同区域。 − ** '''目标检测:''' 模型可以使用注意力机制来聚焦于图像中可能存在目标的区域，提高检测的准确性和效率。 − ** '''图像分类:''' 注意力机制可以帮助模型关注图像中对分类决策至关重要的区域。
无编辑摘要
[[文件:注意力机制的理解.png|缩略图|注意力机制的理解]]
[[文件:注意力机制.png|缩略图|598x598像素|注意力机制<ref>Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020.</ref>。注意力机制的基本原理可以类比为人类的视觉注意过程。当我们观察一个场景时，会自动将注意力集中在最重要或最相关的区域上。]]
在认知科学中，由于信息处理的瓶颈，人类会选择性地关注所有信息的一部分，同时忽略其他可见的信息。通过借鉴人类聚焦关键信息的观察与思考模式，在人工智能领域，注意力机制（Attention Mechanism）是深度学习领域的一项重要技术创新。它允许模型在处理信息时动态地关注最相关的部分，而不是平等对待所有输入信息。注意力机制可以增强可解释性：通过可视化注意力权重，帮助理解模型的决策过程。
在认知科学中，由于信息处理的瓶颈，人类会选择性地关注所有信息的一部分，同时忽略其他可见的信息。通过借鉴人类聚焦关键信息的观察与思考模式，在人工智能领域，注意力机制（Attention Mechanism）是深度学习领域的一项重要技术创新。它允许模型在处理信息时动态地关注最相关的部分，而不是平等对待所有输入信息。注意力机制可以增强可解释性：通过可视化注意力权重，帮助理解模型的决策过程。
= 注意力机制原理 =
= 注意力机制原理 =
注意力机制的本质是，从关注全部到关注重点。比如，当我们阅读文章或观看图像时，视觉系统会自动将注意力集中在关键信息上，而非均匀处理所有细节。注意力机制是一种在神经网络中模拟人类注意力行为的技术。其核心思想是让模型在处理输入序列时，能够有选择性地关注输入序列中与当前任务相关的部分，并分配不同的权重。权重越高，表示模型对该部分输入的信息给予了更多的关注。
注意力机制的计算过程包含三个核心组件：查询（Query, Q）：表示当前需要关注什么；键（Key, K）：表示可以被关注的信息；值（Value, V）：表示实际的信息内容。注意力机制计算一个权重分布，该分布表示了查询与每个键之间的相关性。然后，将这些权重与对应的值进行加权求和，得到最终的注意力输出。这个输出可以被视为模型在处理当前查询时，从输入序列中“提取”出的最相关信息。
= 注意力机制的类型 =
= 注意力机制的类型 =
根据不同的分类标准，注意力机制可以分为多种类型。按注意力范围分类：'''自注意力（Self-Attention）'''：序列内部元素之间的注意力，如Transformer中的自注意力。'''交叉注意力（Cross-Attention）'''：不同序列之间的注意力，如编码器-解码器结构中的注意力。[[文件:Self-attention.png|缩略图|self-attention<ref>Zhang, Y. ''et al.'' Attention is all you need: utilizing attention in AI-enabled drug discovery. ''Briefings in Bioinformatics'' '''25''', bbad467, doi:10.1093/bib/bbad467 (2024).</ref>|替代=|344x344像素]]
自注意力（Self-Attention）：序列内部元素之间的注意力，如Transformer中的自注意力。
[[文件:Self-attention.png|居中|缩略图|self-attention]]
<span class="lark-record-clipboard" data-lark-record-format="docx/record" data-lark-record-data="{"isCut":false,"rootId":"BGqidHR1WoXkk9xC7bqcFw2anEg","parentId":"BGqidHR1WoXkk9xC7bqcFw2anEg","blockIds":[30,31,32,33],"recordIds":["LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50"],"recordMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"id":"LU0zdBh0GoHHQHxMP0HcIK5JnUc","snapshot":{"hidden":false,"text":{"initialAttributedTexts":{"text":{"0":"不同的分类标准，注意力机制可以分为多种类型。"},"attribs":{"0":"*0+m"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}},"folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"locked":false,"revisions":[],"author":"7472320058063110148","children":[],"align":""}},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"id":"Pckgd0iWNo2OoixOqMlcp6cmnBb","snapshot":{"author":"7472320058063110148","type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"revisions":[],"locked":false,"hidden":false,"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"text":{"0":"按注意力范围分类："},"attribs":{"0":"*0+9"}}},"align":"left","folded":false}},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"id":"GHjLdxVaJoyjyOxgRsvcCtk0n0f","snapshot":{"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+1g"},"text":{"0":"自注意力（Self-Attention）：序列内部元素之间的注意力，如Transformer中的自注意力"}}},"folded":false,"type":"text","hidden":false,"revisions":[],"locked":false,"author":"7472320058063110148","align":"left","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[]}},"FStJdh0X2oliUFxY5dzc5Imhn50":{"id":"FStJdh0X2oliUFxY5dzc5Imhn50","snapshot":{"revisions":[],"locked":false,"children":[],"align":"left","folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"hidden":false,"author":"7472320058063110148","text":{"initialAttributedTexts":{"text":{"0":"交叉注意力（Cross-Attention）：不同序列之间的注意力，如编码器-解码器结构中的注意力"},"attribs":{"0":"*0+1d"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}}}},"BGqidHR1WoXkk9xC7bqcFw2anEg":{"id":"BGqidHR1WoXkk9xC7bqcFw2anEg","snapshot":{"locked":false,"hidden":false,"children":["M30AdNYWJo7l4PxDzw7cb0PSn6d","AoxddNG8foQ2sHxMCbDcd9eYnI6","L3prdeCKlouYiPx1XUicfWhxn8d","Thypd4vwTorBglx3jyzcffPgnXc","MmUqdfpEIoYnq1xhP7EcOt06nne","S7lcdvJPFoCbs1xKeMxcipdwndh","EYxJdlFLvoFtLgxHfC4cG4cRnZe","LjyDdfeNvo3Ej7xCVchcEEOEnwe","ABwTdssRgo6pI5xQZNhc5RaEnqg","Ow4LdIsRyoTBojxwgJhcdpSZnLf","USqqdMtj2o9cWUxldVycCssGnzc","DCXOd8qxDo4SPFxiFpMcXwoSnzb","N98ndsupYowzUXxTooacbcwgnYd","M8TAdsdBOo8aU2xnA0ncGY8onsf","YArtdPu0yoVqEHxInyXcUMxfnQb","TLdUdxwAJoqw94xrJRRcZNtgnEe","Wat0d1h3UoX3wkx7OfochVu5n2d","CuxmdxG53o9gc4xAlf9cPpzUnkh","KpuFdVVmeoE2SWxkh6zca3kdnie","Mx2udbKoOoJpk3x4MhacHowpnEe","CAFZd1ulcoAZqGxpCkJcH2ldnpc","QayjdFjYcoA1Fzx9YCBc8ITDnjb","Kn2edmu9moNzxlxyi1McboK2nob","Ouy6dUfikouy53xdm7JcBm1InMg","FpIIdFzLGoYe9Tx101GcahZqnbf","UmIZdue72oYt5xxz3kFcKVWZnwb","Nz4Xd9ZCHoODRNxIKmucaiNfnFf","Joo9dO5FDoOGmaxOiRcc1SXLnob","LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50","ViZudgkuDoBCIsxx8mrckuP7n3b","KwT0dSpnwo5npexvkEZcD2iXnLb","VDM3duxTPo5EBBxEWVAcnzGXnxd","HYjQdfSXpoLh8ZxzAPXczH6inGe","XG2SdE7sqo5iyAxJ8Ryc6OGGnof","RJAWdcI8MogTgDxczd9ceWSHnud","ZSHTd8goGoui1qxk4w0csxsKnfd","LmvxdWGsdo9KgNx80EZciEV0nre","MA1adBNOFoVuWgx4nFvcT6cenef","PsZkdQ6j0otjyIxGlclcztEenBh","Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd","KDJudzs3joriUlxi7TFcjJzAnae","VqftdjOWToCr4bxngPwcnlUgnhc","KISNd7eZqoDilYxnFjUcgaOsnVh","CzhPdgsoQofm8hxJ6Akckfctn4e","EZhxdAK2toAfOOxAy0uclImunOh","Iq5ydWe5xoRSVsxVjHVcykzcnZc","LizadPNYloSMBwxCKHqcI2Qznyg","ZXmXdPCeKo0OYHxJVRecVkeYnkf","DvosdDAsdo4glKxpskPclLsUn2c","ECWdd3WzzoJAWSxC8onceqtEnVh","I0Gud48rioj7suxXW6uclNinnMd","RXKxdSPP5orlIrxPewucrNoXnSf","P1YwdZnPloUmiJxk5fXc06eKn6Q","BwyMdpdZyonHALxCWffcOMlDn3c","DMJpdunfGoL22vxb4g9cBiihnth","D1v1d3HdUot3PUxvpXrcZRLGnUg","ZcEPdC501o8kvQxXvr8c4jFLnog","Zh8Jd3sH6odSMDxqDYocEyCPn2c","K3lQdJbx6o7vBvxQOV4cki9ZnNh","TWLediEoqoneynxGNX0cPvoGn4f","RNirdrgRooPv2ex9OcDcNwOdnCc","CIrQdoawpon11DxDfyVcrwVfnQh","NeWsd97RcoZ2Wwx2BbDcM0yznYb","CMC8dittNoOQ2yxW2jpcIZKanKg","TX9qdmyGkoHUtOxzaBccdmipnEd","HIfSdkagOoaAp2xTVFocIqHonSJ","PU6bdY3Koo7AfdxpmASc4SoCnKc","ZiVAdFShEoaTprxh85Fc7Ro5nnd","Ou4YdSo8Yo8UiIxxwt2cYt1RnOb","RRAvdbOYLoVJA3xLO9PcGltZnmd","VTnMde8WYoAnp1xNqTmc6aKbncf","KC5mdqWSgoNHVVxIF8DczyB2nkh","UqyudArJBodu8ZxCrNmczfkfnMg","B8J5dOUg8oyrypxvQhOcNV4Pnqh","VasHdkCnGohyNQx99sucoXJjnKd","SJCrdAKfUoqk3YxXEkQcaBQ9nVd","WsPPdG3xSolHcbxhzjBcdoobnRg","IwY5dBmqlonoQExwHTnc9VlInmh","CRVcdgIAJojwM9xwI9eczPIPn3g","EHU0dEykNoFuinxJL6NcXvoRnNe","PN3edbQABodx4ixyGUYcxtMZnTe","NSvwdYbXWoOIzuxwCLhc6ny1n2b","POXgdXbI1o69vMxZwUfcObrZnCc"],"doc_info":{"editors":["7472320058063110148","7489058965368913939"],"options":["editors","edit_time"],"deleted_editors":[],"option_modified":null},"author":"7472320058063110148","text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+b"},"text":{"0":"词条作业--注意力机制"}}},"align":"","type":"page","parent_id":"","comments":null,"revisions":null}}},"payloadMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"level":1},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"level":1},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"level":1},"FStJdh0X2oliUFxY5dzc5Imhn50":{"level":1}},"extra":{"channel":"saas","pasteRandomId":"8e7c43b3-d19d-41b0-848f-af317a313ef4","mention_page_title":{},"external_mention_url":{},"isEqualBlockSelection":false},"isKeepQuoteContainer":false,"selection":[{"id":30,"type":"text","selection":{"start":2,"end":24},"recordId":"LU0zdBh0GoHHQHxMP0HcIK5JnUc"},{"id":31,"type":"text","selection":{"start":0,"end":9},"recordId":"Pckgd0iWNo2OoixOqMlcp6cmnBb"},{"id":32,"type":"text","selection":{"start":0,"end":52},"recordId":"GHjLdxVaJoyjyOxgRsvcCtk0n0f"},{"id":33,"type":"text","selection":{"start":0,"end":49},"recordId":"FStJdh0X2oliUFxY5dzc5Imhn50"}],"pasteFlag":"763a5488-2fa9-4107-aa84-183dd617ebd3"}"></span>自注意力机制通过动态权重分配，允许序列中每个元素与其他元素进行交互，从而动态聚合全局上下文信息，精确建模元素间的依赖关系。Transformer模型在诸多自然语言处理任务中展现出的卓越性能源于其全局自注意力机制(1)。
<span class="lark-record-clipboard" data-lark-record-format="docx/record" data-lark-record-data="{"isCut":false,"rootId":"BGqidHR1WoXkk9xC7bqcFw2anEg","parentId":"BGqidHR1WoXkk9xC7bqcFw2anEg","blockIds":[30,31,32,33],"recordIds":["LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50"],"recordMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"id":"LU0zdBh0GoHHQHxMP0HcIK5JnUc","snapshot":{"hidden":false,"text":{"initialAttributedTexts":{"text":{"0":"不同的分类标准，注意力机制可以分为多种类型。"},"attribs":{"0":"*0+m"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}},"folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"locked":false,"revisions":[],"author":"7472320058063110148","children":[],"align":""}},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"id":"Pckgd0iWNo2OoixOqMlcp6cmnBb","snapshot":{"author":"7472320058063110148","type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"revisions":[],"locked":false,"hidden":false,"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"text":{"0":"按注意力范围分类："},"attribs":{"0":"*0+9"}}},"align":"left","folded":false}},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"id":"GHjLdxVaJoyjyOxgRsvcCtk0n0f","snapshot":{"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+1g"},"text":{"0":"自注意力（Self-Attention）：序列内部元素之间的注意力，如Transformer中的自注意力"}}},"folded":false,"type":"text","hidden":false,"revisions":[],"locked":false,"author":"7472320058063110148","align":"left","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[]}},"FStJdh0X2oliUFxY5dzc5Imhn50":{"id":"FStJdh0X2oliUFxY5dzc5Imhn50","snapshot":{"revisions":[],"locked":false,"children":[],"align":"left","folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"hidden":false,"author":"7472320058063110148","text":{"initialAttributedTexts":{"text":{"0":"交叉注意力（Cross-Attention）：不同序列之间的注意力，如编码器-解码器结构中的注意力"},"attribs":{"0":"*0+1d"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}}}},"BGqidHR1WoXkk9xC7bqcFw2anEg":{"id":"BGqidHR1WoXkk9xC7bqcFw2anEg","snapshot":{"locked":false,"hidden":false,"children":["M30AdNYWJo7l4PxDzw7cb0PSn6d","AoxddNG8foQ2sHxMCbDcd9eYnI6","L3prdeCKlouYiPx1XUicfWhxn8d","Thypd4vwTorBglx3jyzcffPgnXc","MmUqdfpEIoYnq1xhP7EcOt06nne","S7lcdvJPFoCbs1xKeMxcipdwndh","EYxJdlFLvoFtLgxHfC4cG4cRnZe","LjyDdfeNvo3Ej7xCVchcEEOEnwe","ABwTdssRgo6pI5xQZNhc5RaEnqg","Ow4LdIsRyoTBojxwgJhcdpSZnLf","USqqdMtj2o9cWUxldVycCssGnzc","DCXOd8qxDo4SPFxiFpMcXwoSnzb","N98ndsupYowzUXxTooacbcwgnYd","M8TAdsdBOo8aU2xnA0ncGY8onsf","YArtdPu0yoVqEHxInyXcUMxfnQb","TLdUdxwAJoqw94xrJRRcZNtgnEe","Wat0d1h3UoX3wkx7OfochVu5n2d","CuxmdxG53o9gc4xAlf9cPpzUnkh","KpuFdVVmeoE2SWxkh6zca3kdnie","Mx2udbKoOoJpk3x4MhacHowpnEe","CAFZd1ulcoAZqGxpCkJcH2ldnpc","QayjdFjYcoA1Fzx9YCBc8ITDnjb","Kn2edmu9moNzxlxyi1McboK2nob","Ouy6dUfikouy53xdm7JcBm1InMg","FpIIdFzLGoYe9Tx101GcahZqnbf","UmIZdue72oYt5xxz3kFcKVWZnwb","Nz4Xd9ZCHoODRNxIKmucaiNfnFf","Joo9dO5FDoOGmaxOiRcc1SXLnob","LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50","ViZudgkuDoBCIsxx8mrckuP7n3b","KwT0dSpnwo5npexvkEZcD2iXnLb","VDM3duxTPo5EBBxEWVAcnzGXnxd","HYjQdfSXpoLh8ZxzAPXczH6inGe","XG2SdE7sqo5iyAxJ8Ryc6OGGnof","RJAWdcI8MogTgDxczd9ceWSHnud","ZSHTd8goGoui1qxk4w0csxsKnfd","LmvxdWGsdo9KgNx80EZciEV0nre","MA1adBNOFoVuWgx4nFvcT6cenef","PsZkdQ6j0otjyIxGlclcztEenBh","Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd","KDJudzs3joriUlxi7TFcjJzAnae","VqftdjOWToCr4bxngPwcnlUgnhc","KISNd7eZqoDilYxnFjUcgaOsnVh","CzhPdgsoQofm8hxJ6Akckfctn4e","EZhxdAK2toAfOOxAy0uclImunOh","Iq5ydWe5xoRSVsxVjHVcykzcnZc","LizadPNYloSMBwxCKHqcI2Qznyg","ZXmXdPCeKo0OYHxJVRecVkeYnkf","DvosdDAsdo4glKxpskPclLsUn2c","ECWdd3WzzoJAWSxC8onceqtEnVh","I0Gud48rioj7suxXW6uclNinnMd","RXKxdSPP5orlIrxPewucrNoXnSf","P1YwdZnPloUmiJxk5fXc06eKn6Q","BwyMdpdZyonHALxCWffcOMlDn3c","DMJpdunfGoL22vxb4g9cBiihnth","D1v1d3HdUot3PUxvpXrcZRLGnUg","ZcEPdC501o8kvQxXvr8c4jFLnog","Zh8Jd3sH6odSMDxqDYocEyCPn2c","K3lQdJbx6o7vBvxQOV4cki9ZnNh","TWLediEoqoneynxGNX0cPvoGn4f","RNirdrgRooPv2ex9OcDcNwOdnCc","CIrQdoawpon11DxDfyVcrwVfnQh","NeWsd97RcoZ2Wwx2BbDcM0yznYb","CMC8dittNoOQ2yxW2jpcIZKanKg","TX9qdmyGkoHUtOxzaBccdmipnEd","HIfSdkagOoaAp2xTVFocIqHonSJ","PU6bdY3Koo7AfdxpmASc4SoCnKc","ZiVAdFShEoaTprxh85Fc7Ro5nnd","Ou4YdSo8Yo8UiIxxwt2cYt1RnOb","RRAvdbOYLoVJA3xLO9PcGltZnmd","VTnMde8WYoAnp1xNqTmc6aKbncf","KC5mdqWSgoNHVVxIF8DczyB2nkh","UqyudArJBodu8ZxCrNmczfkfnMg","B8J5dOUg8oyrypxvQhOcNV4Pnqh","VasHdkCnGohyNQx99sucoXJjnKd","SJCrdAKfUoqk3YxXEkQcaBQ9nVd","WsPPdG3xSolHcbxhzjBcdoobnRg","IwY5dBmqlonoQExwHTnc9VlInmh","CRVcdgIAJojwM9xwI9eczPIPn3g","EHU0dEykNoFuinxJL6NcXvoRnNe","PN3edbQABodx4ixyGUYcxtMZnTe","NSvwdYbXWoOIzuxwCLhc6ny1n2b","POXgdXbI1o69vMxZwUfcObrZnCc"],"doc_info":{"editors":["7472320058063110148","7489058965368913939"],"options":["editors","edit_time"],"deleted_editors":[],"option_modified":null},"author":"7472320058063110148","text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+b"},"text":{"0":"词条作业--注意力机制"}}},"align":"","type":"page","parent_id":"","comments":null,"revisions":null}}},"payloadMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"level":1},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"level":1},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"level":1},"FStJdh0X2oliUFxY5dzc5Imhn50":{"level":1}},"extra":{"channel":"saas","pasteRandomId":"8e7c43b3-d19d-41b0-848f-af317a313ef4","mention_page_title":{},"external_mention_url":{},"isEqualBlockSelection":false},"isKeepQuoteContainer":false,"selection":[{"id":30,"type":"text","selection":{"start":2,"end":24},"recordId":"LU0zdBh0GoHHQHxMP0HcIK5JnUc"},{"id":31,"type":"text","selection":{"start":0,"end":9},"recordId":"Pckgd0iWNo2OoixOqMlcp6cmnBb"},{"id":32,"type":"text","selection":{"start":0,"end":52},"recordId":"GHjLdxVaJoyjyOxgRsvcCtk0n0f"},{"id":33,"type":"text","selection":{"start":0,"end":49},"recordId":"FStJdh0X2oliUFxY5dzc5Imhn50"}],"pasteFlag":"763a5488-2fa9-4107-aa84-183dd617ebd3"}"></span>
=== 自注意力机制 ===
自注意力机制通过动态权重分配，允许序列中每个元素与其他元素进行交互，从而动态聚合全局上下文信息，精确建模元素间的依赖关系。Transformer模型在诸多自然语言处理任务中展现出的卓越性能源于其全局自注意力机制<ref>Feng, Z. & Li, Y. in ''2024 5th International Conference on Computer Engineering and Application (ICCEA).'' 449-454.</ref>。
具体的步骤为<ref>Han, K. ''et al.'' A Survey on Vision Transformer. ''IEEE Transactions on Pattern Analysis and Machine Intelligence'' '''45''', 87-110, doi:10.1109/TPAMI.2022.3152247 (2023).</ref><ref>Vaswani, A. ''et al.'' in ''ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 30 (NIPS 2017)'' Vol. 30 (2017).</ref>：
===== 1.查询 (Query), 键(Key), 值 (Value) 的生成 =====
将输入序列转化形成的嵌入向量（可以是词向量，也可以是词向量加上位置编码向量）分别与3个不同的可学习权重矩阵W(Q), W(K), W(V)相乘生成Q, K, V向量。其中Q, K向量的维度为d(k)，V向量的维度为d(v)。
将输入序列转化形成的嵌入向量（可以是词向量，也可以是词向量加上位置编码向量）分别与3个不同的可学习权重矩阵W(Q), W(K), W(V)相乘生成Q, K, V向量。其中Q, K向量的维度为d(k)，V向量的维度为d(v)。在Transformer中，
==== 2.注意力分数矩阵 ====
将Q向量与所有K向量点乘（matrix multiplication, MatMul）得到注意力分数矩阵。注意实际应用中，如模型在生成第i个词时，只能依赖目标序列的前 i-1 个词，而不能看到目标词。通过添加 mask，可以将目标序列中当前位置之后的词隐藏起来，防止模型在生成过程中提前获取到未来的信息，从而保证模型在训练过程中的自回归特性，使其能够更好地模拟实际的序列生成过程。
将Q向量与所有K向量点乘（matrix multiplication, MatMul）得到注意力分数矩阵。注意实际应用中，如模型在生成第i个词时，只能依赖目标序列的前 i-1 个词，而不能看到目标词。通过添加 mask，可以将目标序列中当前位置之后的词隐藏起来，防止模型在生成过程中提前获取到未来的信息，从而保证模型在训练过程中的自回归特性，使其能够更好地模拟实际的序列生成过程。
==== 3.归一化点乘矩阵 ====
由于点乘操作，S值可能过大或过小，在归一化之前通常需要除以一个缩放因子来稳定梯度。之后应用Softmax 函数对注意力分数矩阵进行归一化。
由于点乘操作，S值可能过大或过小，在归一化之前通常需要除以一个缩放因子来稳定梯度。之后应用Softmax 函数对注意力分数矩阵进行归一化。
==== 4.加权输出 ====
使用归一化后的注意力分数对V向量进行加权求和，作为自注意力机制的输出。整体流程可以概括为缩放点乘注意力 (Scaled dot-product attention)，用公式表示为：
[[文件:公式2.png|居中|缩略图]]
具体的代码实现可参考：<syntaxhighlight lang="python3" line="1">
import torch
import torch
from torch import nn
from torch import nn
context = torch.matmul(attn, V)
context = torch.matmul(attn, V)
return context, attn
return context, attn
</syntaxhighlight><span class="lark-record-clipboard" data-lark-record-format="docx/record" data-lark-record-data="{"isCut":false,"rootId":"BGqidHR1WoXkk9xC7bqcFw2anEg","parentId":"BGqidHR1WoXkk9xC7bqcFw2anEg","blockIds":[44,45,46,47,48],"recordIds":["Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd"],"recordMap":{"Nbj2dHyQRouQiaxLJBScuFcDnph":{"id":"Nbj2dHyQRouQiaxLJBScuFcDnph","snapshot":{"author":"7489058965368913939","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","hidden":false,"revisions":[],"locked":false,"children":[],"text":{"initialAttributedTexts":{"text":{"0":"（3）归一化点乘矩阵"},"attribs":{"0":"*0+a"}},"apool":{"numToAttrib":{"0":["author","7489058965368913939"]},"nextNum":1}},"align":"","folded":false,"type":"text","comments":[]}},"SFGbdp0C7ooLUIxfVRHcI9WWnic":{"id":"SFGbdp0C7ooLUIxfVRHcI9WWnic","snapshot":{"children":[],"text":{"apool":{"numToAttrib":{"0":["author","7489058965368913939"]},"nextNum":1},"initialAttributedTexts":{"attribs":{"0":"*0+1x"},"text":{"0":"由于点乘操作，S值可能过大或过小，在归一化之前通常需要除以一个缩放因子来稳定梯度。之后应用Softmax 函数对注意力分数矩阵进行归一化。"}}},"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"author":"7489058965368913939","folded":false,"revisions":[],"locked":false,"hidden":false,"align":""}},"NicZdBnNGoIX7FxecnAc66nunac":{"id":"NicZdBnNGoIX7FxecnAc66nunac","snapshot":{"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","locked":false,"hidden":false,"folded":false,"comments":[],"revisions":[],"author":"7489058965368913939","children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7489058965368913939"]}},"initialAttributedTexts":{"attribs":{"0":"*0+7"},"text":{"0":"（4）加权输出"}}},"align":""}},"Q7kxdIZhMopEDXxZ6CEcqztEn3b":{"id":"Q7kxdIZhMopEDXxZ6CEcqztEn3b","snapshot":{"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7489058965368913939"]}},"initialAttributedTexts":{"attribs":{"0":"*0+z"},"text":{"0":"使用归一化后的注意力分数对V向量进行加权求和，作为自注意力机制的输出。"}}},"align":"","revisions":[],"parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"locked":false,"hidden":false,"author":"7489058965368913939","children":[],"folded":false,"type":"text"}},"H5zmdoYSDonC2vxNVhicCXBMnFd":{"id":"H5zmdoYSDonC2vxNVhicCXBMnFd","snapshot":{"children":[],"text":{"initialAttributedTexts":{"text":{"0":"整体流程可以概括为缩放点乘注意力 (Scaled dot-product attention)，用公式表示为："},"attribs":{"0":"*0+1j"}},"apool":{"numToAttrib":{"0":["author","7489058965368913939"]},"nextNum":1}},"align":"","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","author":"7489058965368913939","revisions":[],"locked":false,"hidden":false,"folded":false,"type":"text","comments":[]}},"BGqidHR1WoXkk9xC7bqcFw2anEg":{"id":"BGqidHR1WoXkk9xC7bqcFw2anEg","snapshot":{"locked":false,"hidden":false,"children":["M30AdNYWJo7l4PxDzw7cb0PSn6d","AoxddNG8foQ2sHxMCbDcd9eYnI6","L3prdeCKlouYiPx1XUicfWhxn8d","Thypd4vwTorBglx3jyzcffPgnXc","MmUqdfpEIoYnq1xhP7EcOt06nne","S7lcdvJPFoCbs1xKeMxcipdwndh","EYxJdlFLvoFtLgxHfC4cG4cRnZe","LjyDdfeNvo3Ej7xCVchcEEOEnwe","ABwTdssRgo6pI5xQZNhc5RaEnqg","Ow4LdIsRyoTBojxwgJhcdpSZnLf","USqqdMtj2o9cWUxldVycCssGnzc","DCXOd8qxDo4SPFxiFpMcXwoSnzb","N98ndsupYowzUXxTooacbcwgnYd","M8TAdsdBOo8aU2xnA0ncGY8onsf","YArtdPu0yoVqEHxInyXcUMxfnQb","TLdUdxwAJoqw94xrJRRcZNtgnEe","Wat0d1h3UoX3wkx7OfochVu5n2d","CuxmdxG53o9gc4xAlf9cPpzUnkh","KpuFdVVmeoE2SWxkh6zca3kdnie","Mx2udbKoOoJpk3x4MhacHowpnEe","CAFZd1ulcoAZqGxpCkJcH2ldnpc","QayjdFjYcoA1Fzx9YCBc8ITDnjb","Kn2edmu9moNzxlxyi1McboK2nob","Ouy6dUfikouy53xdm7JcBm1InMg","FpIIdFzLGoYe9Tx101GcahZqnbf","UmIZdue72oYt5xxz3kFcKVWZnwb","Nz4Xd9ZCHoODRNxIKmucaiNfnFf","Joo9dO5FDoOGmaxOiRcc1SXLnob","LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50","ViZudgkuDoBCIsxx8mrckuP7n3b","KwT0dSpnwo5npexvkEZcD2iXnLb","VDM3duxTPo5EBBxEWVAcnzGXnxd","HYjQdfSXpoLh8ZxzAPXczH6inGe","XG2SdE7sqo5iyAxJ8Ryc6OGGnof","RJAWdcI8MogTgDxczd9ceWSHnud","ZSHTd8goGoui1qxk4w0csxsKnfd","LmvxdWGsdo9KgNx80EZciEV0nre","MA1adBNOFoVuWgx4nFvcT6cenef","PsZkdQ6j0otjyIxGlclcztEenBh","Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd","KDJudzs3joriUlxi7TFcjJzAnae","VqftdjOWToCr4bxngPwcnlUgnhc","KISNd7eZqoDilYxnFjUcgaOsnVh","CzhPdgsoQofm8hxJ6Akckfctn4e","EZhxdAK2toAfOOxAy0uclImunOh","Iq5ydWe5xoRSVsxVjHVcykzcnZc","LizadPNYloSMBwxCKHqcI2Qznyg","ZXmXdPCeKo0OYHxJVRecVkeYnkf","DvosdDAsdo4glKxpskPclLsUn2c","ECWdd3WzzoJAWSxC8onceqtEnVh","I0Gud48rioj7suxXW6uclNinnMd","RXKxdSPP5orlIrxPewucrNoXnSf","P1YwdZnPloUmiJxk5fXc06eKn6Q","BwyMdpdZyonHALxCWffcOMlDn3c","DMJpdunfGoL22vxb4g9cBiihnth","D1v1d3HdUot3PUxvpXrcZRLGnUg","ZcEPdC501o8kvQxXvr8c4jFLnog","Zh8Jd3sH6odSMDxqDYocEyCPn2c","K3lQdJbx6o7vBvxQOV4cki9ZnNh","TWLediEoqoneynxGNX0cPvoGn4f","RNirdrgRooPv2ex9OcDcNwOdnCc","CIrQdoawpon11DxDfyVcrwVfnQh","NeWsd97RcoZ2Wwx2BbDcM0yznYb","CMC8dittNoOQ2yxW2jpcIZKanKg","TX9qdmyGkoHUtOxzaBccdmipnEd","HIfSdkagOoaAp2xTVFocIqHonSJ","PU6bdY3Koo7AfdxpmASc4SoCnKc","ZiVAdFShEoaTprxh85Fc7Ro5nnd","Ou4YdSo8Yo8UiIxxwt2cYt1RnOb","RRAvdbOYLoVJA3xLO9PcGltZnmd","VTnMde8WYoAnp1xNqTmc6aKbncf","KC5mdqWSgoNHVVxIF8DczyB2nkh","UqyudArJBodu8ZxCrNmczfkfnMg","B8J5dOUg8oyrypxvQhOcNV4Pnqh","VasHdkCnGohyNQx99sucoXJjnKd","SJCrdAKfUoqk3YxXEkQcaBQ9nVd","WsPPdG3xSolHcbxhzjBcdoobnRg","IwY5dBmqlonoQExwHTnc9VlInmh","CRVcdgIAJojwM9xwI9eczPIPn3g","EHU0dEykNoFuinxJL6NcXvoRnNe","PN3edbQABodx4ixyGUYcxtMZnTe","NSvwdYbXWoOIzuxwCLhc6ny1n2b","POXgdXbI1o69vMxZwUfcObrZnCc"],"doc_info":{"editors":["7472320058063110148","7489058965368913939"],"options":["editors","edit_time"],"deleted_editors":[],"option_modified":null},"author":"7472320058063110148","text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+b"},"text":{"0":"词条作业--注意力机制"}}},"align":"","type":"page","parent_id":"","comments":null,"revisions":null}}},"payloadMap":{"Nbj2dHyQRouQiaxLJBScuFcDnph":{"level":1},"SFGbdp0C7ooLUIxfVRHcI9WWnic":{"level":1},"NicZdBnNGoIX7FxecnAc66nunac":{"level":1},"Q7kxdIZhMopEDXxZ6CEcqztEn3b":{"level":1},"H5zmdoYSDonC2vxNVhicCXBMnFd":{"level":1}},"extra":{"channel":"saas","pasteRandomId":"94e291cb-ab77-4893-aee3-ec4aabe88184","mention_page_title":{},"external_mention_url":{},"isEqualBlockSelection":true},"isKeepQuoteContainer":false,"selection":[{"id":44,"type":"text","selection":{"start":0,"end":10},"recordId":"Nbj2dHyQRouQiaxLJBScuFcDnph"},{"id":45,"type":"text","selection":{"start":0,"end":69},"recordId":"SFGbdp0C7ooLUIxfVRHcI9WWnic"},{"id":46,"type":"text","selection":{"start":0,"end":7},"recordId":"NicZdBnNGoIX7FxecnAc66nunac"},{"id":47,"type":"text","selection":{"start":0,"end":35},"recordId":"Q7kxdIZhMopEDXxZ6CEcqztEn3b"},{"id":48,"type":"text","selection":{"start":0,"end":55},"recordId":"H5zmdoYSDonC2vxNVhicCXBMnFd"}],"pasteFlag":"763a5488-2fa9-4107-aa84-183dd617ebd3"}"></span>多头注意力是利用多个查询，来平行地计算从输入信息中选取多个信息。每个注意力关注输入信息的不同部分。
</syntaxhighlight>
=== 多头注意力机制 ===
[[文件:多头注意力机制.png|缩略图|多头注意力机制<ref>Zhang, Y. ''et al.'' Attention is all you need: utilizing attention in AI-enabled drug discovery. ''Briefings in Bioinformatics'' '''25''', bbad467, doi:10.1093/bib/bbad467 (2024).</ref>|替代=]]多头注意力是利用多个查询，来平行地计算从输入信息中选取多个信息。每个注意力关注输入信息的不同部分。它将输入序列的表示拆分成多个子空间（头），然后在每个子空间内独立地计算注意力权重，最后将各个子空间的结果拼接起来。多头注意力机制使得模型可以在不同的表示子空间中捕获到不同的上下文信息，增强对复杂数据内在结构的建模能力(3-5)。
工作流程:
工作流程:
将所有头的输出拼接后通过线性层融合作为多头注意力机制的输出：
将所有头的输出拼接后通过线性层融合作为多头注意力机制的输出：
= 注意力机制的应用 =
= 注意力机制的应用 =
注意力机制在各种人工智能领域都得到了广泛的应用：
注意力机制在各种人工智能领域都得到了广泛的应用：
[[文件:注意力机制在自然语言处理中的应用.png|缩略图|注意力机制在自然语言处理中的应用<ref name=":0">Andrea, G. ''et al''. Attention in Natural Language Processing. ''IEEE Transactions on Neural Networks and Learning Systems'' '''32''', 4291-4308, doi: 10.1109/tnnls.2020.3019893 (2021).</ref>]]
'''自然语言处理:'''
'''机器翻译:''' 注意力机制使得模型在翻译长句子时，能够关注到源语言句子中与目标语言当前词语相关的部分。通过引入注意力机制，我们将源语言中每个位置的信息都保存下来。在解码过程中生成每一个目标语言的单词时，我们都通过注意力机制直接从源语言的信息中选择相关的信息作为辅助。
'''文本摘要:''' 模型可以学习关注文档中的关键句子或段落，生成更准确、更简洁的摘要。
'''问答系统:''' 注意力机制帮助模型定位问题相关的文档段落，从而给出更准确的答案
[[文件:注意力机制用于计算机视觉.png|缩略图|注意力机制用于计算机视觉<ref name=":0" />]]
'''计算机视觉:'''
'''图像描述生成:''' 在生成描述的每一个单词时，注意力机制来选择一些来自于图像的相关信息，关注图像的不同区域。
'''目标检测:''' 模型可以使用注意力机制来聚焦于图像中可能存在目标的区域，提高检测的准确性和效率。
'''图像分类:''' 注意力机制可以帮助模型关注图像中对分类决策至关重要的区域。