130
个编辑
更改
注意力机制
[[文件:注意力机制的理解.png|缩略图|注意力机制的理解]]
在认知科学中，由于信息处理的瓶颈，人类会选择性地关注所有信息的一部分，同时忽略其他可见的信息。通过借鉴人类聚焦关键信息的观察与思考模式，在人工智能领域，注意力机制（Attention Mechanism）是深度学习领域的一项重要技术创新。它允许模型在处理信息时动态地关注最相关的部分，而不是平等对待所有输入信息。注意力机制可以增强可解释性：通过可视化注意力权重，帮助理解模型的决策过程。
=== 注意力机制原理 ===
注意力机制的本质是，从关注全部到关注重点。
比如，当我们阅读文章或观看图像时，视觉系统会自动将注意力集中在关键信息上，而非均匀处理所有细节。
注意力机制的计算过程包含三个核心组件：
查询（Query, Q）：表示当前需要关注什么；键（Key, K）：表示可以被关注的信息；值（Value, V）：表示实际的信息内容
注意力机制计算一个权重分布，该分布表示了查询与每个键之间的相关性。然后，将这些权重与对应的值进行加权求和，得到最终的注意力输出。这个输出可以被视为模型在处理当前查询时，从输入序列中“提取”出的最相关信息。
=== 注意力机制的类型 ===
根据不同的分类标准，注意力机制可以分为多种类型。
按注意力范围分类：
自注意力（Self-Attention）：序列内部元素之间的注意力，如Transformer中的自注意力。
[[文件:Self-attention.png|居中|缩略图|self-attention]]
交叉注意力（Cross-Attention）：不同序列之间的注意力，如编码器-解码器结构中的注意力。
<span class="lark-record-clipboard" data-lark-record-format="docx/record" data-lark-record-data="{"isCut":false,"rootId":"BGqidHR1WoXkk9xC7bqcFw2anEg","parentId":"BGqidHR1WoXkk9xC7bqcFw2anEg","blockIds":[30,31,32,33],"recordIds":["LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50"],"recordMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"id":"LU0zdBh0GoHHQHxMP0HcIK5JnUc","snapshot":{"hidden":false,"text":{"initialAttributedTexts":{"text":{"0":"不同的分类标准，注意力机制可以分为多种类型。"},"attribs":{"0":"*0+m"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}},"folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"locked":false,"revisions":[],"author":"7472320058063110148","children":[],"align":""}},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"id":"Pckgd0iWNo2OoixOqMlcp6cmnBb","snapshot":{"author":"7472320058063110148","type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"revisions":[],"locked":false,"hidden":false,"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"text":{"0":"按注意力范围分类："},"attribs":{"0":"*0+9"}}},"align":"left","folded":false}},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"id":"GHjLdxVaJoyjyOxgRsvcCtk0n0f","snapshot":{"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+1g"},"text":{"0":"自注意力（Self-Attention）：序列内部元素之间的注意力，如Transformer中的自注意力"}}},"folded":false,"type":"text","hidden":false,"revisions":[],"locked":false,"author":"7472320058063110148","align":"left","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[]}},"FStJdh0X2oliUFxY5dzc5Imhn50":{"id":"FStJdh0X2oliUFxY5dzc5Imhn50","snapshot":{"revisions":[],"locked":false,"children":[],"align":"left","folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"hidden":false,"author":"7472320058063110148","text":{"initialAttributedTexts":{"text":{"0":"交叉注意力（Cross-Attention）：不同序列之间的注意力，如编码器-解码器结构中的注意力"},"attribs":{"0":"*0+1d"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}}}},"BGqidHR1WoXkk9xC7bqcFw2anEg":{"id":"BGqidHR1WoXkk9xC7bqcFw2anEg","snapshot":{"locked":false,"hidden":false,"children":["M30AdNYWJo7l4PxDzw7cb0PSn6d","AoxddNG8foQ2sHxMCbDcd9eYnI6","L3prdeCKlouYiPx1XUicfWhxn8d","Thypd4vwTorBglx3jyzcffPgnXc","MmUqdfpEIoYnq1xhP7EcOt06nne","S7lcdvJPFoCbs1xKeMxcipdwndh","EYxJdlFLvoFtLgxHfC4cG4cRnZe","LjyDdfeNvo3Ej7xCVchcEEOEnwe","ABwTdssRgo6pI5xQZNhc5RaEnqg","Ow4LdIsRyoTBojxwgJhcdpSZnLf","USqqdMtj2o9cWUxldVycCssGnzc","DCXOd8qxDo4SPFxiFpMcXwoSnzb","N98ndsupYowzUXxTooacbcwgnYd","M8TAdsdBOo8aU2xnA0ncGY8onsf","YArtdPu0yoVqEHxInyXcUMxfnQb","TLdUdxwAJoqw94xrJRRcZNtgnEe","Wat0d1h3UoX3wkx7OfochVu5n2d","CuxmdxG53o9gc4xAlf9cPpzUnkh","KpuFdVVmeoE2SWxkh6zca3kdnie","Mx2udbKoOoJpk3x4MhacHowpnEe","CAFZd1ulcoAZqGxpCkJcH2ldnpc","QayjdFjYcoA1Fzx9YCBc8ITDnjb","Kn2edmu9moNzxlxyi1McboK2nob","Ouy6dUfikouy53xdm7JcBm1InMg","FpIIdFzLGoYe9Tx101GcahZqnbf","UmIZdue72oYt5xxz3kFcKVWZnwb","Nz4Xd9ZCHoODRNxIKmucaiNfnFf","Joo9dO5FDoOGmaxOiRcc1SXLnob","LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50","ViZudgkuDoBCIsxx8mrckuP7n3b","KwT0dSpnwo5npexvkEZcD2iXnLb","VDM3duxTPo5EBBxEWVAcnzGXnxd","HYjQdfSXpoLh8ZxzAPXczH6inGe","XG2SdE7sqo5iyAxJ8Ryc6OGGnof","RJAWdcI8MogTgDxczd9ceWSHnud","ZSHTd8goGoui1qxk4w0csxsKnfd","LmvxdWGsdo9KgNx80EZciEV0nre","MA1adBNOFoVuWgx4nFvcT6cenef","PsZkdQ6j0otjyIxGlclcztEenBh","Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd","KDJudzs3joriUlxi7TFcjJzAnae","VqftdjOWToCr4bxngPwcnlUgnhc","KISNd7eZqoDilYxnFjUcgaOsnVh","CzhPdgsoQofm8hxJ6Akckfctn4e","EZhxdAK2toAfOOxAy0uclImunOh","Iq5ydWe5xoRSVsxVjHVcykzcnZc","LizadPNYloSMBwxCKHqcI2Qznyg","ZXmXdPCeKo0OYHxJVRecVkeYnkf","DvosdDAsdo4glKxpskPclLsUn2c","ECWdd3WzzoJAWSxC8onceqtEnVh","I0Gud48rioj7suxXW6uclNinnMd","RXKxdSPP5orlIrxPewucrNoXnSf","P1YwdZnPloUmiJxk5fXc06eKn6Q","BwyMdpdZyonHALxCWffcOMlDn3c","DMJpdunfGoL22vxb4g9cBiihnth","D1v1d3HdUot3PUxvpXrcZRLGnUg","ZcEPdC501o8kvQxXvr8c4jFLnog","Zh8Jd3sH6odSMDxqDYocEyCPn2c","K3lQdJbx6o7vBvxQOV4cki9ZnNh","TWLediEoqoneynxGNX0cPvoGn4f","RNirdrgRooPv2ex9OcDcNwOdnCc","CIrQdoawpon11DxDfyVcrwVfnQh","NeWsd97RcoZ2Wwx2BbDcM0yznYb","CMC8dittNoOQ2yxW2jpcIZKanKg","TX9qdmyGkoHUtOxzaBccdmipnEd","HIfSdkagOoaAp2xTVFocIqHonSJ","PU6bdY3Koo7AfdxpmASc4SoCnKc","ZiVAdFShEoaTprxh85Fc7Ro5nnd","Ou4YdSo8Yo8UiIxxwt2cYt1RnOb","RRAvdbOYLoVJA3xLO9PcGltZnmd","VTnMde8WYoAnp1xNqTmc6aKbncf","KC5mdqWSgoNHVVxIF8DczyB2nkh","UqyudArJBodu8ZxCrNmczfkfnMg","B8J5dOUg8oyrypxvQhOcNV4Pnqh","VasHdkCnGohyNQx99sucoXJjnKd","SJCrdAKfUoqk3YxXEkQcaBQ9nVd","WsPPdG3xSolHcbxhzjBcdoobnRg","IwY5dBmqlonoQExwHTnc9VlInmh","CRVcdgIAJojwM9xwI9eczPIPn3g","EHU0dEykNoFuinxJL6NcXvoRnNe","PN3edbQABodx4ixyGUYcxtMZnTe","NSvwdYbXWoOIzuxwCLhc6ny1n2b","POXgdXbI1o69vMxZwUfcObrZnCc"],"doc_info":{"editors":["7472320058063110148","7489058965368913939"],"options":["editors","edit_time"],"deleted_editors":[],"option_modified":null},"author":"7472320058063110148","text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+b"},"text":{"0":"词条作业--注意力机制"}}},"align":"","type":"page","parent_id":"","comments":null,"revisions":null}}},"payloadMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"level":1},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"level":1},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"level":1},"FStJdh0X2oliUFxY5dzc5Imhn50":{"level":1}},"extra":{"channel":"saas","pasteRandomId":"8e7c43b3-d19d-41b0-848f-af317a313ef4","mention_page_title":{},"external_mention_url":{},"isEqualBlockSelection":false},"isKeepQuoteContainer":false,"selection":[{"id":30,"type":"text","selection":{"start":2,"end":24},"recordId":"LU0zdBh0GoHHQHxMP0HcIK5JnUc"},{"id":31,"type":"text","selection":{"start":0,"end":9},"recordId":"Pckgd0iWNo2OoixOqMlcp6cmnBb"},{"id":32,"type":"text","selection":{"start":0,"end":52},"recordId":"GHjLdxVaJoyjyOxgRsvcCtk0n0f"},{"id":33,"type":"text","selection":{"start":0,"end":49},"recordId":"FStJdh0X2oliUFxY5dzc5Imhn50"}],"pasteFlag":"763a5488-2fa9-4107-aa84-183dd617ebd3"}"></span>自注意力机制通过动态权重分配，允许序列中每个元素与其他元素进行交互，从而动态聚合全局上下文信息，精确建模元素间的依赖关系。Transformer模型在诸多自然语言处理任务中展现出的卓越性能源于其全局自注意力机制(1)。
<span class="lark-record-clipboard" data-lark-record-format="docx/record" data-lark-record-data="{"isCut":false,"rootId":"BGqidHR1WoXkk9xC7bqcFw2anEg","parentId":"BGqidHR1WoXkk9xC7bqcFw2anEg","blockIds":[30,31,32,33],"recordIds":["LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50"],"recordMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"id":"LU0zdBh0GoHHQHxMP0HcIK5JnUc","snapshot":{"hidden":false,"text":{"initialAttributedTexts":{"text":{"0":"不同的分类标准，注意力机制可以分为多种类型。"},"attribs":{"0":"*0+m"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}},"folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"locked":false,"revisions":[],"author":"7472320058063110148","children":[],"align":""}},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"id":"Pckgd0iWNo2OoixOqMlcp6cmnBb","snapshot":{"author":"7472320058063110148","type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"revisions":[],"locked":false,"hidden":false,"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"text":{"0":"按注意力范围分类："},"attribs":{"0":"*0+9"}}},"align":"left","folded":false}},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"id":"GHjLdxVaJoyjyOxgRsvcCtk0n0f","snapshot":{"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+1g"},"text":{"0":"自注意力（Self-Attention）：序列内部元素之间的注意力，如Transformer中的自注意力"}}},"folded":false,"type":"text","hidden":false,"revisions":[],"locked":false,"author":"7472320058063110148","align":"left","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[]}},"FStJdh0X2oliUFxY5dzc5Imhn50":{"id":"FStJdh0X2oliUFxY5dzc5Imhn50","snapshot":{"revisions":[],"locked":false,"children":[],"align":"left","folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"hidden":false,"author":"7472320058063110148","text":{"initialAttributedTexts":{"text":{"0":"交叉注意力（Cross-Attention）：不同序列之间的注意力，如编码器-解码器结构中的注意力"},"attribs":{"0":"*0+1d"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}}}},"BGqidHR1WoXkk9xC7bqcFw2anEg":{"id":"BGqidHR1WoXkk9xC7bqcFw2anEg","snapshot":{"locked":false,"hidden":false,"children":["M30AdNYWJo7l4PxDzw7cb0PSn6d","AoxddNG8foQ2sHxMCbDcd9eYnI6","L3prdeCKlouYiPx1XUicfWhxn8d","Thypd4vwTorBglx3jyzcffPgnXc","MmUqdfpEIoYnq1xhP7EcOt06nne","S7lcdvJPFoCbs1xKeMxcipdwndh","EYxJdlFLvoFtLgxHfC4cG4cRnZe","LjyDdfeNvo3Ej7xCVchcEEOEnwe","ABwTdssRgo6pI5xQZNhc5RaEnqg","Ow4LdIsRyoTBojxwgJhcdpSZnLf","USqqdMtj2o9cWUxldVycCssGnzc","DCXOd8qxDo4SPFxiFpMcXwoSnzb","N98ndsupYowzUXxTooacbcwgnYd","M8TAdsdBOo8aU2xnA0ncGY8onsf","YArtdPu0yoVqEHxInyXcUMxfnQb","TLdUdxwAJoqw94xrJRRcZNtgnEe","Wat0d1h3UoX3wkx7OfochVu5n2d","CuxmdxG53o9gc4xAlf9cPpzUnkh","KpuFdVVmeoE2SWxkh6zca3kdnie","Mx2udbKoOoJpk3x4MhacHowpnEe","CAFZd1ulcoAZqGxpCkJcH2ldnpc","QayjdFjYcoA1Fzx9YCBc8ITDnjb","Kn2edmu9moNzxlxyi1McboK2nob","Ouy6dUfikouy53xdm7JcBm1InMg","FpIIdFzLGoYe9Tx101GcahZqnbf","UmIZdue72oYt5xxz3kFcKVWZnwb","Nz4Xd9ZCHoODRNxIKmucaiNfnFf","Joo9dO5FDoOGmaxOiRcc1SXLnob","LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50","ViZudgkuDoBCIsxx8mrckuP7n3b","KwT0dSpnwo5npexvkEZcD2iXnLb","VDM3duxTPo5EBBxEWVAcnzGXnxd","HYjQdfSXpoLh8ZxzAPXczH6inGe","XG2SdE7sqo5iyAxJ8Ryc6OGGnof","RJAWdcI8MogTgDxczd9ceWSHnud","ZSHTd8goGoui1qxk4w0csxsKnfd","LmvxdWGsdo9KgNx80EZciEV0nre","MA1adBNOFoVuWgx4nFvcT6cenef","PsZkdQ6j0otjyIxGlclcztEenBh","Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd","KDJudzs3joriUlxi7TFcjJzAnae","VqftdjOWToCr4bxngPwcnlUgnhc","KISNd7eZqoDilYxnFjUcgaOsnVh","CzhPdgsoQofm8hxJ6Akckfctn4e","EZhxdAK2toAfOOxAy0uclImunOh","Iq5ydWe5xoRSVsxVjHVcykzcnZc","LizadPNYloSMBwxCKHqcI2Qznyg","ZXmXdPCeKo0OYHxJVRecVkeYnkf","DvosdDAsdo4glKxpskPclLsUn2c","ECWdd3WzzoJAWSxC8onceqtEnVh","I0Gud48rioj7suxXW6uclNinnMd","RXKxdSPP5orlIrxPewucrNoXnSf","P1YwdZnPloUmiJxk5fXc06eKn6Q","BwyMdpdZyonHALxCWffcOMlDn3c","DMJpdunfGoL22vxb4g9cBiihnth","D1v1d3HdUot3PUxvpXrcZRLGnUg","ZcEPdC501o8kvQxXvr8c4jFLnog","Zh8Jd3sH6odSMDxqDYocEyCPn2c","K3lQdJbx6o7vBvxQOV4cki9ZnNh","TWLediEoqoneynxGNX0cPvoGn4f","RNirdrgRooPv2ex9OcDcNwOdnCc","CIrQdoawpon11DxDfyVcrwVfnQh","NeWsd97RcoZ2Wwx2BbDcM0yznYb","CMC8dittNoOQ2yxW2jpcIZKanKg","TX9qdmyGkoHUtOxzaBccdmipnEd","HIfSdkagOoaAp2xTVFocIqHonSJ","PU6bdY3Koo7AfdxpmASc4SoCnKc","ZiVAdFShEoaTprxh85Fc7Ro5nnd","Ou4YdSo8Yo8UiIxxwt2cYt1RnOb","RRAvdbOYLoVJA3xLO9PcGltZnmd","VTnMde8WYoAnp1xNqTmc6aKbncf","KC5mdqWSgoNHVVxIF8DczyB2nkh","UqyudArJBodu8ZxCrNmczfkfnMg","B8J5dOUg8oyrypxvQhOcNV4Pnqh","VasHdkCnGohyNQx99sucoXJjnKd","SJCrdAKfUoqk3YxXEkQcaBQ9nVd","WsPPdG3xSolHcbxhzjBcdoobnRg","IwY5dBmqlonoQExwHTnc9VlInmh","CRVcdgIAJojwM9xwI9eczPIPn3g","EHU0dEykNoFuinxJL6NcXvoRnNe","PN3edbQABodx4ixyGUYcxtMZnTe","NSvwdYbXWoOIzuxwCLhc6ny1n2b","POXgdXbI1o69vMxZwUfcObrZnCc"],"doc_info":{"editors":["7472320058063110148","7489058965368913939"],"options":["editors","edit_time"],"deleted_editors":[],"option_modified":null},"author":"7472320058063110148","text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+b"},"text":{"0":"词条作业--注意力机制"}}},"align":"","type":"page","parent_id":"","comments":null,"revisions":null}}},"payloadMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"level":1},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"level":1},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"level":1},"FStJdh0X2oliUFxY5dzc5Imhn50":{"level":1}},"extra":{"channel":"saas","pasteRandomId":"8e7c43b3-d19d-41b0-848f-af317a313ef4","mention_page_title":{},"external_mention_url":{},"isEqualBlockSelection":false},"isKeepQuoteContainer":false,"selection":[{"id":30,"type":"text","selection":{"start":2,"end":24},"recordId":"LU0zdBh0GoHHQHxMP0HcIK5JnUc"},{"id":31,"type":"text","selection":{"start":0,"end":9},"recordId":"Pckgd0iWNo2OoixOqMlcp6cmnBb"},{"id":32,"type":"text","selection":{"start":0,"end":52},"recordId":"GHjLdxVaJoyjyOxgRsvcCtk0n0f"},{"id":33,"type":"text","selection":{"start":0,"end":49},"recordId":"FStJdh0X2oliUFxY5dzc5Imhn50"}],"pasteFlag":"763a5488-2fa9-4107-aa84-183dd617ebd3"}"></span>
工作流程(2,3)：
（1）查询 (Query), 键(Key), 值 (Value) 的生成
将输入序列转化形成的嵌入向量（可以是词向量，也可以是词向量加上位置编码向量）分别与3个不同的可学习权重矩阵W(Q), W(K), W(V)相乘生成Q, K, V向量。其中Q, K向量的维度为d(k)，V向量的维度为d(v)。在Transformer中，
（2）注意力分数矩阵
将Q向量与所有K向量点乘（matrix multiplication, MatMul）得到注意力分数矩阵。注意实际应用中，如模型在生成第i个词时，只能依赖目标序列的前 i-1 个词，而不能看到目标词。通过添加 mask，可以将目标序列中当前位置之后的词隐藏起来，防止模型在生成过程中提前获取到未来的信息，从而保证模型在训练过程中的自回归特性，使其能够更好地模拟实际的序列生成过程。
（3）归一化点乘矩阵
由于点乘操作，S值可能过大或过小，在归一化之前通常需要除以一个缩放因子来稳定梯度。之后应用Softmax 函数对注意力分数矩阵进行归一化。
（4）加权输出
使用归一化后的注意力分数对V向量进行加权求和，作为自注意力机制的输出。
整体流程可以概括为缩放点乘注意力 (Scaled dot-product attention)，用公式表示为：<syntaxhighlight lang="python3" line="1">
import torch
from torch import nn
class ScaledDotProductAttention(nn.Module):
def __init__(self, d_k):
super().init()
self.sqrt_d_k = torch.sqrt(torch.tensor(d_k, dtype=torch.float32))
def forward(self, Q, K, V, attn_mask=None):
scores = torch.matmul(Q, K.transpose(-1, -2)) / self.sqrt_d_kif attn_mask is not None:
scores = scores.masked_fill(attn_mask == 0, -1e9)
attn = torch.softmax(scores, dim=-1)
context = torch.matmul(attn, V)
return context, attn
</syntaxhighlight><span class="lark-record-clipboard" data-lark-record-format="docx/record" data-lark-record-data="{"isCut":false,"rootId":"BGqidHR1WoXkk9xC7bqcFw2anEg","parentId":"BGqidHR1WoXkk9xC7bqcFw2anEg","blockIds":[44,45,46,47,48],"recordIds":["Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd"],"recordMap":{"Nbj2dHyQRouQiaxLJBScuFcDnph":{"id":"Nbj2dHyQRouQiaxLJBScuFcDnph","snapshot":{"author":"7489058965368913939","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","hidden":false,"revisions":[],"locked":false,"children":[],"text":{"initialAttributedTexts":{"text":{"0":"（3）归一化点乘矩阵"},"attribs":{"0":"*0+a"}},"apool":{"numToAttrib":{"0":["author","7489058965368913939"]},"nextNum":1}},"align":"","folded":false,"type":"text","comments":[]}},"SFGbdp0C7ooLUIxfVRHcI9WWnic":{"id":"SFGbdp0C7ooLUIxfVRHcI9WWnic","snapshot":{"children":[],"text":{"apool":{"numToAttrib":{"0":["author","7489058965368913939"]},"nextNum":1},"initialAttributedTexts":{"attribs":{"0":"*0+1x"},"text":{"0":"由于点乘操作，S值可能过大或过小，在归一化之前通常需要除以一个缩放因子来稳定梯度。之后应用Softmax 函数对注意力分数矩阵进行归一化。"}}},"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"author":"7489058965368913939","folded":false,"revisions":[],"locked":false,"hidden":false,"align":""}},"NicZdBnNGoIX7FxecnAc66nunac":{"id":"NicZdBnNGoIX7FxecnAc66nunac","snapshot":{"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","locked":false,"hidden":false,"folded":false,"comments":[],"revisions":[],"author":"7489058965368913939","children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7489058965368913939"]}},"initialAttributedTexts":{"attribs":{"0":"*0+7"},"text":{"0":"（4）加权输出"}}},"align":""}},"Q7kxdIZhMopEDXxZ6CEcqztEn3b":{"id":"Q7kxdIZhMopEDXxZ6CEcqztEn3b","snapshot":{"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7489058965368913939"]}},"initialAttributedTexts":{"attribs":{"0":"*0+z"},"text":{"0":"使用归一化后的注意力分数对V向量进行加权求和，作为自注意力机制的输出。"}}},"align":"","revisions":[],"parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"locked":false,"hidden":false,"author":"7489058965368913939","children":[],"folded":false,"type":"text"}},"H5zmdoYSDonC2vxNVhicCXBMnFd":{"id":"H5zmdoYSDonC2vxNVhicCXBMnFd","snapshot":{"children":[],"text":{"initialAttributedTexts":{"text":{"0":"整体流程可以概括为缩放点乘注意力 (Scaled dot-product attention)，用公式表示为："},"attribs":{"0":"*0+1j"}},"apool":{"numToAttrib":{"0":["author","7489058965368913939"]},"nextNum":1}},"align":"","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","author":"7489058965368913939","revisions":[],"locked":false,"hidden":false,"folded":false,"type":"text","comments":[]}},"BGqidHR1WoXkk9xC7bqcFw2anEg":{"id":"BGqidHR1WoXkk9xC7bqcFw2anEg","snapshot":{"locked":false,"hidden":false,"children":["M30AdNYWJo7l4PxDzw7cb0PSn6d","AoxddNG8foQ2sHxMCbDcd9eYnI6","L3prdeCKlouYiPx1XUicfWhxn8d","Thypd4vwTorBglx3jyzcffPgnXc","MmUqdfpEIoYnq1xhP7EcOt06nne","S7lcdvJPFoCbs1xKeMxcipdwndh","EYxJdlFLvoFtLgxHfC4cG4cRnZe","LjyDdfeNvo3Ej7xCVchcEEOEnwe","ABwTdssRgo6pI5xQZNhc5RaEnqg","Ow4LdIsRyoTBojxwgJhcdpSZnLf","USqqdMtj2o9cWUxldVycCssGnzc","DCXOd8qxDo4SPFxiFpMcXwoSnzb","N98ndsupYowzUXxTooacbcwgnYd","M8TAdsdBOo8aU2xnA0ncGY8onsf","YArtdPu0yoVqEHxInyXcUMxfnQb","TLdUdxwAJoqw94xrJRRcZNtgnEe","Wat0d1h3UoX3wkx7OfochVu5n2d","CuxmdxG53o9gc4xAlf9cPpzUnkh","KpuFdVVmeoE2SWxkh6zca3kdnie","Mx2udbKoOoJpk3x4MhacHowpnEe","CAFZd1ulcoAZqGxpCkJcH2ldnpc","QayjdFjYcoA1Fzx9YCBc8ITDnjb","Kn2edmu9moNzxlxyi1McboK2nob","Ouy6dUfikouy53xdm7JcBm1InMg","FpIIdFzLGoYe9Tx101GcahZqnbf","UmIZdue72oYt5xxz3kFcKVWZnwb","Nz4Xd9ZCHoODRNxIKmucaiNfnFf","Joo9dO5FDoOGmaxOiRcc1SXLnob","LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50","ViZudgkuDoBCIsxx8mrckuP7n3b","KwT0dSpnwo5npexvkEZcD2iXnLb","VDM3duxTPo5EBBxEWVAcnzGXnxd","HYjQdfSXpoLh8ZxzAPXczH6inGe","XG2SdE7sqo5iyAxJ8Ryc6OGGnof","RJAWdcI8MogTgDxczd9ceWSHnud","ZSHTd8goGoui1qxk4w0csxsKnfd","LmvxdWGsdo9KgNx80EZciEV0nre","MA1adBNOFoVuWgx4nFvcT6cenef","PsZkdQ6j0otjyIxGlclcztEenBh","Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd","KDJudzs3joriUlxi7TFcjJzAnae","VqftdjOWToCr4bxngPwcnlUgnhc","KISNd7eZqoDilYxnFjUcgaOsnVh","CzhPdgsoQofm8hxJ6Akckfctn4e","EZhxdAK2toAfOOxAy0uclImunOh","Iq5ydWe5xoRSVsxVjHVcykzcnZc","LizadPNYloSMBwxCKHqcI2Qznyg","ZXmXdPCeKo0OYHxJVRecVkeYnkf","DvosdDAsdo4glKxpskPclLsUn2c","ECWdd3WzzoJAWSxC8onceqtEnVh","I0Gud48rioj7suxXW6uclNinnMd","RXKxdSPP5orlIrxPewucrNoXnSf","P1YwdZnPloUmiJxk5fXc06eKn6Q","BwyMdpdZyonHALxCWffcOMlDn3c","DMJpdunfGoL22vxb4g9cBiihnth","D1v1d3HdUot3PUxvpXrcZRLGnUg","ZcEPdC501o8kvQxXvr8c4jFLnog","Zh8Jd3sH6odSMDxqDYocEyCPn2c","K3lQdJbx6o7vBvxQOV4cki9ZnNh","TWLediEoqoneynxGNX0cPvoGn4f","RNirdrgRooPv2ex9OcDcNwOdnCc","CIrQdoawpon11DxDfyVcrwVfnQh","NeWsd97RcoZ2Wwx2BbDcM0yznYb","CMC8dittNoOQ2yxW2jpcIZKanKg","TX9qdmyGkoHUtOxzaBccdmipnEd","HIfSdkagOoaAp2xTVFocIqHonSJ","PU6bdY3Koo7AfdxpmASc4SoCnKc","ZiVAdFShEoaTprxh85Fc7Ro5nnd","Ou4YdSo8Yo8UiIxxwt2cYt1RnOb","RRAvdbOYLoVJA3xLO9PcGltZnmd","VTnMde8WYoAnp1xNqTmc6aKbncf","KC5mdqWSgoNHVVxIF8DczyB2nkh","UqyudArJBodu8ZxCrNmczfkfnMg","B8J5dOUg8oyrypxvQhOcNV4Pnqh","VasHdkCnGohyNQx99sucoXJjnKd","SJCrdAKfUoqk3YxXEkQcaBQ9nVd","WsPPdG3xSolHcbxhzjBcdoobnRg","IwY5dBmqlonoQExwHTnc9VlInmh","CRVcdgIAJojwM9xwI9eczPIPn3g","EHU0dEykNoFuinxJL6NcXvoRnNe","PN3edbQABodx4ixyGUYcxtMZnTe","NSvwdYbXWoOIzuxwCLhc6ny1n2b","POXgdXbI1o69vMxZwUfcObrZnCc"],"doc_info":{"editors":["7472320058063110148","7489058965368913939"],"options":["editors","edit_time"],"deleted_editors":[],"option_modified":null},"author":"7472320058063110148","text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+b"},"text":{"0":"词条作业--注意力机制"}}},"align":"","type":"page","parent_id":"","comments":null,"revisions":null}}},"payloadMap":{"Nbj2dHyQRouQiaxLJBScuFcDnph":{"level":1},"SFGbdp0C7ooLUIxfVRHcI9WWnic":{"level":1},"NicZdBnNGoIX7FxecnAc66nunac":{"level":1},"Q7kxdIZhMopEDXxZ6CEcqztEn3b":{"level":1},"H5zmdoYSDonC2vxNVhicCXBMnFd":{"level":1}},"extra":{"channel":"saas","pasteRandomId":"94e291cb-ab77-4893-aee3-ec4aabe88184","mention_page_title":{},"external_mention_url":{},"isEqualBlockSelection":true},"isKeepQuoteContainer":false,"selection":[{"id":44,"type":"text","selection":{"start":0,"end":10},"recordId":"Nbj2dHyQRouQiaxLJBScuFcDnph"},{"id":45,"type":"text","selection":{"start":0,"end":69},"recordId":"SFGbdp0C7ooLUIxfVRHcI9WWnic"},{"id":46,"type":"text","selection":{"start":0,"end":7},"recordId":"NicZdBnNGoIX7FxecnAc66nunac"},{"id":47,"type":"text","selection":{"start":0,"end":35},"recordId":"Q7kxdIZhMopEDXxZ6CEcqztEn3b"},{"id":48,"type":"text","selection":{"start":0,"end":55},"recordId":"H5zmdoYSDonC2vxNVhicCXBMnFd"}],"pasteFlag":"763a5488-2fa9-4107-aa84-183dd617ebd3"}"></span>多头注意力是利用多个查询，来平行地计算从输入信息中选取多个信息。每个注意力关注输入信息的不同部分。
多头注意力机制将输入序列的表示拆分成多个子空间（头），然后在每个子空间内独立地计算注意力权重，最后将各个子空间的结果拼接起来。多头注意力机制使得模型可以在不同的表示子空间中捕获到不同的上下文信息，增强对复杂数据内在结构的建模能力(3-5)。
工作流程:
（1）子空间Q, K, V 向量的生成
给定嵌入向量和头 (Head) 的数目h，在每个头（子空间）中都会通过一组不同的权重矩阵
对输入序列进行线性投影变换得到
, 其维度为
（2）多头并行计算
对每个头独立计算缩放点乘注意力：
（3）多头合并输出
将所有头的输出拼接后通过线性层融合作为多头注意力机制的输出：
[[文件:多头注意力机制.png|缩略图|多头注意力机制]]
=== 注意力机制的应用 ===
在认知科学中，由于信息处理的瓶颈，人类会选择性地关注所有信息的一部分，同时忽略其他可见的信息。通过借鉴人类聚焦关键信息的观察与思考模式，在人工智能领域，注意力机制（Attention Mechanism）是深度学习领域的一项重要技术创新。它允许模型在处理信息时动态地关注最相关的部分，而不是平等对待所有输入信息。注意力机制可以增强可解释性：通过可视化注意力权重，帮助理解模型的决策过程。
=== 注意力机制原理 ===
注意力机制的本质是，从关注全部到关注重点。
比如，当我们阅读文章或观看图像时，视觉系统会自动将注意力集中在关键信息上，而非均匀处理所有细节。
注意力机制的计算过程包含三个核心组件：
查询（Query, Q）：表示当前需要关注什么；键（Key, K）：表示可以被关注的信息；值（Value, V）：表示实际的信息内容
注意力机制计算一个权重分布，该分布表示了查询与每个键之间的相关性。然后，将这些权重与对应的值进行加权求和，得到最终的注意力输出。这个输出可以被视为模型在处理当前查询时，从输入序列中“提取”出的最相关信息。
=== 注意力机制的类型 ===
根据不同的分类标准，注意力机制可以分为多种类型。
按注意力范围分类：
自注意力（Self-Attention）：序列内部元素之间的注意力，如Transformer中的自注意力。
[[文件:Self-attention.png|居中|缩略图|self-attention]]
交叉注意力（Cross-Attention）：不同序列之间的注意力，如编码器-解码器结构中的注意力。
<span class="lark-record-clipboard" data-lark-record-format="docx/record" data-lark-record-data="{"isCut":false,"rootId":"BGqidHR1WoXkk9xC7bqcFw2anEg","parentId":"BGqidHR1WoXkk9xC7bqcFw2anEg","blockIds":[30,31,32,33],"recordIds":["LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50"],"recordMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"id":"LU0zdBh0GoHHQHxMP0HcIK5JnUc","snapshot":{"hidden":false,"text":{"initialAttributedTexts":{"text":{"0":"不同的分类标准，注意力机制可以分为多种类型。"},"attribs":{"0":"*0+m"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}},"folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"locked":false,"revisions":[],"author":"7472320058063110148","children":[],"align":""}},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"id":"Pckgd0iWNo2OoixOqMlcp6cmnBb","snapshot":{"author":"7472320058063110148","type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"revisions":[],"locked":false,"hidden":false,"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"text":{"0":"按注意力范围分类："},"attribs":{"0":"*0+9"}}},"align":"left","folded":false}},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"id":"GHjLdxVaJoyjyOxgRsvcCtk0n0f","snapshot":{"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+1g"},"text":{"0":"自注意力（Self-Attention）：序列内部元素之间的注意力，如Transformer中的自注意力"}}},"folded":false,"type":"text","hidden":false,"revisions":[],"locked":false,"author":"7472320058063110148","align":"left","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[]}},"FStJdh0X2oliUFxY5dzc5Imhn50":{"id":"FStJdh0X2oliUFxY5dzc5Imhn50","snapshot":{"revisions":[],"locked":false,"children":[],"align":"left","folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"hidden":false,"author":"7472320058063110148","text":{"initialAttributedTexts":{"text":{"0":"交叉注意力（Cross-Attention）：不同序列之间的注意力，如编码器-解码器结构中的注意力"},"attribs":{"0":"*0+1d"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}}}},"BGqidHR1WoXkk9xC7bqcFw2anEg":{"id":"BGqidHR1WoXkk9xC7bqcFw2anEg","snapshot":{"locked":false,"hidden":false,"children":["M30AdNYWJo7l4PxDzw7cb0PSn6d","AoxddNG8foQ2sHxMCbDcd9eYnI6","L3prdeCKlouYiPx1XUicfWhxn8d","Thypd4vwTorBglx3jyzcffPgnXc","MmUqdfpEIoYnq1xhP7EcOt06nne","S7lcdvJPFoCbs1xKeMxcipdwndh","EYxJdlFLvoFtLgxHfC4cG4cRnZe","LjyDdfeNvo3Ej7xCVchcEEOEnwe","ABwTdssRgo6pI5xQZNhc5RaEnqg","Ow4LdIsRyoTBojxwgJhcdpSZnLf","USqqdMtj2o9cWUxldVycCssGnzc","DCXOd8qxDo4SPFxiFpMcXwoSnzb","N98ndsupYowzUXxTooacbcwgnYd","M8TAdsdBOo8aU2xnA0ncGY8onsf","YArtdPu0yoVqEHxInyXcUMxfnQb","TLdUdxwAJoqw94xrJRRcZNtgnEe","Wat0d1h3UoX3wkx7OfochVu5n2d","CuxmdxG53o9gc4xAlf9cPpzUnkh","KpuFdVVmeoE2SWxkh6zca3kdnie","Mx2udbKoOoJpk3x4MhacHowpnEe","CAFZd1ulcoAZqGxpCkJcH2ldnpc","QayjdFjYcoA1Fzx9YCBc8ITDnjb","Kn2edmu9moNzxlxyi1McboK2nob","Ouy6dUfikouy53xdm7JcBm1InMg","FpIIdFzLGoYe9Tx101GcahZqnbf","UmIZdue72oYt5xxz3kFcKVWZnwb","Nz4Xd9ZCHoODRNxIKmucaiNfnFf","Joo9dO5FDoOGmaxOiRcc1SXLnob","LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50","ViZudgkuDoBCIsxx8mrckuP7n3b","KwT0dSpnwo5npexvkEZcD2iXnLb","VDM3duxTPo5EBBxEWVAcnzGXnxd","HYjQdfSXpoLh8ZxzAPXczH6inGe","XG2SdE7sqo5iyAxJ8Ryc6OGGnof","RJAWdcI8MogTgDxczd9ceWSHnud","ZSHTd8goGoui1qxk4w0csxsKnfd","LmvxdWGsdo9KgNx80EZciEV0nre","MA1adBNOFoVuWgx4nFvcT6cenef","PsZkdQ6j0otjyIxGlclcztEenBh","Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd","KDJudzs3joriUlxi7TFcjJzAnae","VqftdjOWToCr4bxngPwcnlUgnhc","KISNd7eZqoDilYxnFjUcgaOsnVh","CzhPdgsoQofm8hxJ6Akckfctn4e","EZhxdAK2toAfOOxAy0uclImunOh","Iq5ydWe5xoRSVsxVjHVcykzcnZc","LizadPNYloSMBwxCKHqcI2Qznyg","ZXmXdPCeKo0OYHxJVRecVkeYnkf","DvosdDAsdo4glKxpskPclLsUn2c","ECWdd3WzzoJAWSxC8onceqtEnVh","I0Gud48rioj7suxXW6uclNinnMd","RXKxdSPP5orlIrxPewucrNoXnSf","P1YwdZnPloUmiJxk5fXc06eKn6Q","BwyMdpdZyonHALxCWffcOMlDn3c","DMJpdunfGoL22vxb4g9cBiihnth","D1v1d3HdUot3PUxvpXrcZRLGnUg","ZcEPdC501o8kvQxXvr8c4jFLnog","Zh8Jd3sH6odSMDxqDYocEyCPn2c","K3lQdJbx6o7vBvxQOV4cki9ZnNh","TWLediEoqoneynxGNX0cPvoGn4f","RNirdrgRooPv2ex9OcDcNwOdnCc","CIrQdoawpon11DxDfyVcrwVfnQh","NeWsd97RcoZ2Wwx2BbDcM0yznYb","CMC8dittNoOQ2yxW2jpcIZKanKg","TX9qdmyGkoHUtOxzaBccdmipnEd","HIfSdkagOoaAp2xTVFocIqHonSJ","PU6bdY3Koo7AfdxpmASc4SoCnKc","ZiVAdFShEoaTprxh85Fc7Ro5nnd","Ou4YdSo8Yo8UiIxxwt2cYt1RnOb","RRAvdbOYLoVJA3xLO9PcGltZnmd","VTnMde8WYoAnp1xNqTmc6aKbncf","KC5mdqWSgoNHVVxIF8DczyB2nkh","UqyudArJBodu8ZxCrNmczfkfnMg","B8J5dOUg8oyrypxvQhOcNV4Pnqh","VasHdkCnGohyNQx99sucoXJjnKd","SJCrdAKfUoqk3YxXEkQcaBQ9nVd","WsPPdG3xSolHcbxhzjBcdoobnRg","IwY5dBmqlonoQExwHTnc9VlInmh","CRVcdgIAJojwM9xwI9eczPIPn3g","EHU0dEykNoFuinxJL6NcXvoRnNe","PN3edbQABodx4ixyGUYcxtMZnTe","NSvwdYbXWoOIzuxwCLhc6ny1n2b","POXgdXbI1o69vMxZwUfcObrZnCc"],"doc_info":{"editors":["7472320058063110148","7489058965368913939"],"options":["editors","edit_time"],"deleted_editors":[],"option_modified":null},"author":"7472320058063110148","text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+b"},"text":{"0":"词条作业--注意力机制"}}},"align":"","type":"page","parent_id":"","comments":null,"revisions":null}}},"payloadMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"level":1},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"level":1},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"level":1},"FStJdh0X2oliUFxY5dzc5Imhn50":{"level":1}},"extra":{"channel":"saas","pasteRandomId":"8e7c43b3-d19d-41b0-848f-af317a313ef4","mention_page_title":{},"external_mention_url":{},"isEqualBlockSelection":false},"isKeepQuoteContainer":false,"selection":[{"id":30,"type":"text","selection":{"start":2,"end":24},"recordId":"LU0zdBh0GoHHQHxMP0HcIK5JnUc"},{"id":31,"type":"text","selection":{"start":0,"end":9},"recordId":"Pckgd0iWNo2OoixOqMlcp6cmnBb"},{"id":32,"type":"text","selection":{"start":0,"end":52},"recordId":"GHjLdxVaJoyjyOxgRsvcCtk0n0f"},{"id":33,"type":"text","selection":{"start":0,"end":49},"recordId":"FStJdh0X2oliUFxY5dzc5Imhn50"}],"pasteFlag":"763a5488-2fa9-4107-aa84-183dd617ebd3"}"></span>自注意力机制通过动态权重分配，允许序列中每个元素与其他元素进行交互，从而动态聚合全局上下文信息，精确建模元素间的依赖关系。Transformer模型在诸多自然语言处理任务中展现出的卓越性能源于其全局自注意力机制(1)。
<span class="lark-record-clipboard" data-lark-record-format="docx/record" data-lark-record-data="{"isCut":false,"rootId":"BGqidHR1WoXkk9xC7bqcFw2anEg","parentId":"BGqidHR1WoXkk9xC7bqcFw2anEg","blockIds":[30,31,32,33],"recordIds":["LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50"],"recordMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"id":"LU0zdBh0GoHHQHxMP0HcIK5JnUc","snapshot":{"hidden":false,"text":{"initialAttributedTexts":{"text":{"0":"不同的分类标准，注意力机制可以分为多种类型。"},"attribs":{"0":"*0+m"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}},"folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"locked":false,"revisions":[],"author":"7472320058063110148","children":[],"align":""}},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"id":"Pckgd0iWNo2OoixOqMlcp6cmnBb","snapshot":{"author":"7472320058063110148","type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"revisions":[],"locked":false,"hidden":false,"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"text":{"0":"按注意力范围分类："},"attribs":{"0":"*0+9"}}},"align":"left","folded":false}},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"id":"GHjLdxVaJoyjyOxgRsvcCtk0n0f","snapshot":{"children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+1g"},"text":{"0":"自注意力（Self-Attention）：序列内部元素之间的注意力，如Transformer中的自注意力"}}},"folded":false,"type":"text","hidden":false,"revisions":[],"locked":false,"author":"7472320058063110148","align":"left","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[]}},"FStJdh0X2oliUFxY5dzc5Imhn50":{"id":"FStJdh0X2oliUFxY5dzc5Imhn50","snapshot":{"revisions":[],"locked":false,"children":[],"align":"left","folded":false,"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"hidden":false,"author":"7472320058063110148","text":{"initialAttributedTexts":{"text":{"0":"交叉注意力（Cross-Attention）：不同序列之间的注意力，如编码器-解码器结构中的注意力"},"attribs":{"0":"*0+1d"}},"apool":{"numToAttrib":{"0":["author","7472320058063110148"]},"nextNum":1}}}},"BGqidHR1WoXkk9xC7bqcFw2anEg":{"id":"BGqidHR1WoXkk9xC7bqcFw2anEg","snapshot":{"locked":false,"hidden":false,"children":["M30AdNYWJo7l4PxDzw7cb0PSn6d","AoxddNG8foQ2sHxMCbDcd9eYnI6","L3prdeCKlouYiPx1XUicfWhxn8d","Thypd4vwTorBglx3jyzcffPgnXc","MmUqdfpEIoYnq1xhP7EcOt06nne","S7lcdvJPFoCbs1xKeMxcipdwndh","EYxJdlFLvoFtLgxHfC4cG4cRnZe","LjyDdfeNvo3Ej7xCVchcEEOEnwe","ABwTdssRgo6pI5xQZNhc5RaEnqg","Ow4LdIsRyoTBojxwgJhcdpSZnLf","USqqdMtj2o9cWUxldVycCssGnzc","DCXOd8qxDo4SPFxiFpMcXwoSnzb","N98ndsupYowzUXxTooacbcwgnYd","M8TAdsdBOo8aU2xnA0ncGY8onsf","YArtdPu0yoVqEHxInyXcUMxfnQb","TLdUdxwAJoqw94xrJRRcZNtgnEe","Wat0d1h3UoX3wkx7OfochVu5n2d","CuxmdxG53o9gc4xAlf9cPpzUnkh","KpuFdVVmeoE2SWxkh6zca3kdnie","Mx2udbKoOoJpk3x4MhacHowpnEe","CAFZd1ulcoAZqGxpCkJcH2ldnpc","QayjdFjYcoA1Fzx9YCBc8ITDnjb","Kn2edmu9moNzxlxyi1McboK2nob","Ouy6dUfikouy53xdm7JcBm1InMg","FpIIdFzLGoYe9Tx101GcahZqnbf","UmIZdue72oYt5xxz3kFcKVWZnwb","Nz4Xd9ZCHoODRNxIKmucaiNfnFf","Joo9dO5FDoOGmaxOiRcc1SXLnob","LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50","ViZudgkuDoBCIsxx8mrckuP7n3b","KwT0dSpnwo5npexvkEZcD2iXnLb","VDM3duxTPo5EBBxEWVAcnzGXnxd","HYjQdfSXpoLh8ZxzAPXczH6inGe","XG2SdE7sqo5iyAxJ8Ryc6OGGnof","RJAWdcI8MogTgDxczd9ceWSHnud","ZSHTd8goGoui1qxk4w0csxsKnfd","LmvxdWGsdo9KgNx80EZciEV0nre","MA1adBNOFoVuWgx4nFvcT6cenef","PsZkdQ6j0otjyIxGlclcztEenBh","Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd","KDJudzs3joriUlxi7TFcjJzAnae","VqftdjOWToCr4bxngPwcnlUgnhc","KISNd7eZqoDilYxnFjUcgaOsnVh","CzhPdgsoQofm8hxJ6Akckfctn4e","EZhxdAK2toAfOOxAy0uclImunOh","Iq5ydWe5xoRSVsxVjHVcykzcnZc","LizadPNYloSMBwxCKHqcI2Qznyg","ZXmXdPCeKo0OYHxJVRecVkeYnkf","DvosdDAsdo4glKxpskPclLsUn2c","ECWdd3WzzoJAWSxC8onceqtEnVh","I0Gud48rioj7suxXW6uclNinnMd","RXKxdSPP5orlIrxPewucrNoXnSf","P1YwdZnPloUmiJxk5fXc06eKn6Q","BwyMdpdZyonHALxCWffcOMlDn3c","DMJpdunfGoL22vxb4g9cBiihnth","D1v1d3HdUot3PUxvpXrcZRLGnUg","ZcEPdC501o8kvQxXvr8c4jFLnog","Zh8Jd3sH6odSMDxqDYocEyCPn2c","K3lQdJbx6o7vBvxQOV4cki9ZnNh","TWLediEoqoneynxGNX0cPvoGn4f","RNirdrgRooPv2ex9OcDcNwOdnCc","CIrQdoawpon11DxDfyVcrwVfnQh","NeWsd97RcoZ2Wwx2BbDcM0yznYb","CMC8dittNoOQ2yxW2jpcIZKanKg","TX9qdmyGkoHUtOxzaBccdmipnEd","HIfSdkagOoaAp2xTVFocIqHonSJ","PU6bdY3Koo7AfdxpmASc4SoCnKc","ZiVAdFShEoaTprxh85Fc7Ro5nnd","Ou4YdSo8Yo8UiIxxwt2cYt1RnOb","RRAvdbOYLoVJA3xLO9PcGltZnmd","VTnMde8WYoAnp1xNqTmc6aKbncf","KC5mdqWSgoNHVVxIF8DczyB2nkh","UqyudArJBodu8ZxCrNmczfkfnMg","B8J5dOUg8oyrypxvQhOcNV4Pnqh","VasHdkCnGohyNQx99sucoXJjnKd","SJCrdAKfUoqk3YxXEkQcaBQ9nVd","WsPPdG3xSolHcbxhzjBcdoobnRg","IwY5dBmqlonoQExwHTnc9VlInmh","CRVcdgIAJojwM9xwI9eczPIPn3g","EHU0dEykNoFuinxJL6NcXvoRnNe","PN3edbQABodx4ixyGUYcxtMZnTe","NSvwdYbXWoOIzuxwCLhc6ny1n2b","POXgdXbI1o69vMxZwUfcObrZnCc"],"doc_info":{"editors":["7472320058063110148","7489058965368913939"],"options":["editors","edit_time"],"deleted_editors":[],"option_modified":null},"author":"7472320058063110148","text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+b"},"text":{"0":"词条作业--注意力机制"}}},"align":"","type":"page","parent_id":"","comments":null,"revisions":null}}},"payloadMap":{"LU0zdBh0GoHHQHxMP0HcIK5JnUc":{"level":1},"Pckgd0iWNo2OoixOqMlcp6cmnBb":{"level":1},"GHjLdxVaJoyjyOxgRsvcCtk0n0f":{"level":1},"FStJdh0X2oliUFxY5dzc5Imhn50":{"level":1}},"extra":{"channel":"saas","pasteRandomId":"8e7c43b3-d19d-41b0-848f-af317a313ef4","mention_page_title":{},"external_mention_url":{},"isEqualBlockSelection":false},"isKeepQuoteContainer":false,"selection":[{"id":30,"type":"text","selection":{"start":2,"end":24},"recordId":"LU0zdBh0GoHHQHxMP0HcIK5JnUc"},{"id":31,"type":"text","selection":{"start":0,"end":9},"recordId":"Pckgd0iWNo2OoixOqMlcp6cmnBb"},{"id":32,"type":"text","selection":{"start":0,"end":52},"recordId":"GHjLdxVaJoyjyOxgRsvcCtk0n0f"},{"id":33,"type":"text","selection":{"start":0,"end":49},"recordId":"FStJdh0X2oliUFxY5dzc5Imhn50"}],"pasteFlag":"763a5488-2fa9-4107-aa84-183dd617ebd3"}"></span>
工作流程(2,3)：
（1）查询 (Query), 键(Key), 值 (Value) 的生成
将输入序列转化形成的嵌入向量（可以是词向量，也可以是词向量加上位置编码向量）分别与3个不同的可学习权重矩阵W(Q), W(K), W(V)相乘生成Q, K, V向量。其中Q, K向量的维度为d(k)，V向量的维度为d(v)。在Transformer中，
（2）注意力分数矩阵
将Q向量与所有K向量点乘（matrix multiplication, MatMul）得到注意力分数矩阵。注意实际应用中，如模型在生成第i个词时，只能依赖目标序列的前 i-1 个词，而不能看到目标词。通过添加 mask，可以将目标序列中当前位置之后的词隐藏起来，防止模型在生成过程中提前获取到未来的信息，从而保证模型在训练过程中的自回归特性，使其能够更好地模拟实际的序列生成过程。
（3）归一化点乘矩阵
由于点乘操作，S值可能过大或过小，在归一化之前通常需要除以一个缩放因子来稳定梯度。之后应用Softmax 函数对注意力分数矩阵进行归一化。
（4）加权输出
使用归一化后的注意力分数对V向量进行加权求和，作为自注意力机制的输出。
整体流程可以概括为缩放点乘注意力 (Scaled dot-product attention)，用公式表示为：<syntaxhighlight lang="python3" line="1">
import torch
from torch import nn
class ScaledDotProductAttention(nn.Module):
def __init__(self, d_k):
super().init()
self.sqrt_d_k = torch.sqrt(torch.tensor(d_k, dtype=torch.float32))
def forward(self, Q, K, V, attn_mask=None):
scores = torch.matmul(Q, K.transpose(-1, -2)) / self.sqrt_d_kif attn_mask is not None:
scores = scores.masked_fill(attn_mask == 0, -1e9)
attn = torch.softmax(scores, dim=-1)
context = torch.matmul(attn, V)
return context, attn
</syntaxhighlight><span class="lark-record-clipboard" data-lark-record-format="docx/record" data-lark-record-data="{"isCut":false,"rootId":"BGqidHR1WoXkk9xC7bqcFw2anEg","parentId":"BGqidHR1WoXkk9xC7bqcFw2anEg","blockIds":[44,45,46,47,48],"recordIds":["Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd"],"recordMap":{"Nbj2dHyQRouQiaxLJBScuFcDnph":{"id":"Nbj2dHyQRouQiaxLJBScuFcDnph","snapshot":{"author":"7489058965368913939","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","hidden":false,"revisions":[],"locked":false,"children":[],"text":{"initialAttributedTexts":{"text":{"0":"（3）归一化点乘矩阵"},"attribs":{"0":"*0+a"}},"apool":{"numToAttrib":{"0":["author","7489058965368913939"]},"nextNum":1}},"align":"","folded":false,"type":"text","comments":[]}},"SFGbdp0C7ooLUIxfVRHcI9WWnic":{"id":"SFGbdp0C7ooLUIxfVRHcI9WWnic","snapshot":{"children":[],"text":{"apool":{"numToAttrib":{"0":["author","7489058965368913939"]},"nextNum":1},"initialAttributedTexts":{"attribs":{"0":"*0+1x"},"text":{"0":"由于点乘操作，S值可能过大或过小，在归一化之前通常需要除以一个缩放因子来稳定梯度。之后应用Softmax 函数对注意力分数矩阵进行归一化。"}}},"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"author":"7489058965368913939","folded":false,"revisions":[],"locked":false,"hidden":false,"align":""}},"NicZdBnNGoIX7FxecnAc66nunac":{"id":"NicZdBnNGoIX7FxecnAc66nunac","snapshot":{"type":"text","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","locked":false,"hidden":false,"folded":false,"comments":[],"revisions":[],"author":"7489058965368913939","children":[],"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7489058965368913939"]}},"initialAttributedTexts":{"attribs":{"0":"*0+7"},"text":{"0":"（4）加权输出"}}},"align":""}},"Q7kxdIZhMopEDXxZ6CEcqztEn3b":{"id":"Q7kxdIZhMopEDXxZ6CEcqztEn3b","snapshot":{"text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7489058965368913939"]}},"initialAttributedTexts":{"attribs":{"0":"*0+z"},"text":{"0":"使用归一化后的注意力分数对V向量进行加权求和，作为自注意力机制的输出。"}}},"align":"","revisions":[],"parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","comments":[],"locked":false,"hidden":false,"author":"7489058965368913939","children":[],"folded":false,"type":"text"}},"H5zmdoYSDonC2vxNVhicCXBMnFd":{"id":"H5zmdoYSDonC2vxNVhicCXBMnFd","snapshot":{"children":[],"text":{"initialAttributedTexts":{"text":{"0":"整体流程可以概括为缩放点乘注意力 (Scaled dot-product attention)，用公式表示为："},"attribs":{"0":"*0+1j"}},"apool":{"numToAttrib":{"0":["author","7489058965368913939"]},"nextNum":1}},"align":"","parent_id":"BGqidHR1WoXkk9xC7bqcFw2anEg","author":"7489058965368913939","revisions":[],"locked":false,"hidden":false,"folded":false,"type":"text","comments":[]}},"BGqidHR1WoXkk9xC7bqcFw2anEg":{"id":"BGqidHR1WoXkk9xC7bqcFw2anEg","snapshot":{"locked":false,"hidden":false,"children":["M30AdNYWJo7l4PxDzw7cb0PSn6d","AoxddNG8foQ2sHxMCbDcd9eYnI6","L3prdeCKlouYiPx1XUicfWhxn8d","Thypd4vwTorBglx3jyzcffPgnXc","MmUqdfpEIoYnq1xhP7EcOt06nne","S7lcdvJPFoCbs1xKeMxcipdwndh","EYxJdlFLvoFtLgxHfC4cG4cRnZe","LjyDdfeNvo3Ej7xCVchcEEOEnwe","ABwTdssRgo6pI5xQZNhc5RaEnqg","Ow4LdIsRyoTBojxwgJhcdpSZnLf","USqqdMtj2o9cWUxldVycCssGnzc","DCXOd8qxDo4SPFxiFpMcXwoSnzb","N98ndsupYowzUXxTooacbcwgnYd","M8TAdsdBOo8aU2xnA0ncGY8onsf","YArtdPu0yoVqEHxInyXcUMxfnQb","TLdUdxwAJoqw94xrJRRcZNtgnEe","Wat0d1h3UoX3wkx7OfochVu5n2d","CuxmdxG53o9gc4xAlf9cPpzUnkh","KpuFdVVmeoE2SWxkh6zca3kdnie","Mx2udbKoOoJpk3x4MhacHowpnEe","CAFZd1ulcoAZqGxpCkJcH2ldnpc","QayjdFjYcoA1Fzx9YCBc8ITDnjb","Kn2edmu9moNzxlxyi1McboK2nob","Ouy6dUfikouy53xdm7JcBm1InMg","FpIIdFzLGoYe9Tx101GcahZqnbf","UmIZdue72oYt5xxz3kFcKVWZnwb","Nz4Xd9ZCHoODRNxIKmucaiNfnFf","Joo9dO5FDoOGmaxOiRcc1SXLnob","LU0zdBh0GoHHQHxMP0HcIK5JnUc","Pckgd0iWNo2OoixOqMlcp6cmnBb","GHjLdxVaJoyjyOxgRsvcCtk0n0f","FStJdh0X2oliUFxY5dzc5Imhn50","ViZudgkuDoBCIsxx8mrckuP7n3b","KwT0dSpnwo5npexvkEZcD2iXnLb","VDM3duxTPo5EBBxEWVAcnzGXnxd","HYjQdfSXpoLh8ZxzAPXczH6inGe","XG2SdE7sqo5iyAxJ8Ryc6OGGnof","RJAWdcI8MogTgDxczd9ceWSHnud","ZSHTd8goGoui1qxk4w0csxsKnfd","LmvxdWGsdo9KgNx80EZciEV0nre","MA1adBNOFoVuWgx4nFvcT6cenef","PsZkdQ6j0otjyIxGlclcztEenBh","Nbj2dHyQRouQiaxLJBScuFcDnph","SFGbdp0C7ooLUIxfVRHcI9WWnic","NicZdBnNGoIX7FxecnAc66nunac","Q7kxdIZhMopEDXxZ6CEcqztEn3b","H5zmdoYSDonC2vxNVhicCXBMnFd","KDJudzs3joriUlxi7TFcjJzAnae","VqftdjOWToCr4bxngPwcnlUgnhc","KISNd7eZqoDilYxnFjUcgaOsnVh","CzhPdgsoQofm8hxJ6Akckfctn4e","EZhxdAK2toAfOOxAy0uclImunOh","Iq5ydWe5xoRSVsxVjHVcykzcnZc","LizadPNYloSMBwxCKHqcI2Qznyg","ZXmXdPCeKo0OYHxJVRecVkeYnkf","DvosdDAsdo4glKxpskPclLsUn2c","ECWdd3WzzoJAWSxC8onceqtEnVh","I0Gud48rioj7suxXW6uclNinnMd","RXKxdSPP5orlIrxPewucrNoXnSf","P1YwdZnPloUmiJxk5fXc06eKn6Q","BwyMdpdZyonHALxCWffcOMlDn3c","DMJpdunfGoL22vxb4g9cBiihnth","D1v1d3HdUot3PUxvpXrcZRLGnUg","ZcEPdC501o8kvQxXvr8c4jFLnog","Zh8Jd3sH6odSMDxqDYocEyCPn2c","K3lQdJbx6o7vBvxQOV4cki9ZnNh","TWLediEoqoneynxGNX0cPvoGn4f","RNirdrgRooPv2ex9OcDcNwOdnCc","CIrQdoawpon11DxDfyVcrwVfnQh","NeWsd97RcoZ2Wwx2BbDcM0yznYb","CMC8dittNoOQ2yxW2jpcIZKanKg","TX9qdmyGkoHUtOxzaBccdmipnEd","HIfSdkagOoaAp2xTVFocIqHonSJ","PU6bdY3Koo7AfdxpmASc4SoCnKc","ZiVAdFShEoaTprxh85Fc7Ro5nnd","Ou4YdSo8Yo8UiIxxwt2cYt1RnOb","RRAvdbOYLoVJA3xLO9PcGltZnmd","VTnMde8WYoAnp1xNqTmc6aKbncf","KC5mdqWSgoNHVVxIF8DczyB2nkh","UqyudArJBodu8ZxCrNmczfkfnMg","B8J5dOUg8oyrypxvQhOcNV4Pnqh","VasHdkCnGohyNQx99sucoXJjnKd","SJCrdAKfUoqk3YxXEkQcaBQ9nVd","WsPPdG3xSolHcbxhzjBcdoobnRg","IwY5dBmqlonoQExwHTnc9VlInmh","CRVcdgIAJojwM9xwI9eczPIPn3g","EHU0dEykNoFuinxJL6NcXvoRnNe","PN3edbQABodx4ixyGUYcxtMZnTe","NSvwdYbXWoOIzuxwCLhc6ny1n2b","POXgdXbI1o69vMxZwUfcObrZnCc"],"doc_info":{"editors":["7472320058063110148","7489058965368913939"],"options":["editors","edit_time"],"deleted_editors":[],"option_modified":null},"author":"7472320058063110148","text":{"apool":{"nextNum":1,"numToAttrib":{"0":["author","7472320058063110148"]}},"initialAttributedTexts":{"attribs":{"0":"*0+b"},"text":{"0":"词条作业--注意力机制"}}},"align":"","type":"page","parent_id":"","comments":null,"revisions":null}}},"payloadMap":{"Nbj2dHyQRouQiaxLJBScuFcDnph":{"level":1},"SFGbdp0C7ooLUIxfVRHcI9WWnic":{"level":1},"NicZdBnNGoIX7FxecnAc66nunac":{"level":1},"Q7kxdIZhMopEDXxZ6CEcqztEn3b":{"level":1},"H5zmdoYSDonC2vxNVhicCXBMnFd":{"level":1}},"extra":{"channel":"saas","pasteRandomId":"94e291cb-ab77-4893-aee3-ec4aabe88184","mention_page_title":{},"external_mention_url":{},"isEqualBlockSelection":true},"isKeepQuoteContainer":false,"selection":[{"id":44,"type":"text","selection":{"start":0,"end":10},"recordId":"Nbj2dHyQRouQiaxLJBScuFcDnph"},{"id":45,"type":"text","selection":{"start":0,"end":69},"recordId":"SFGbdp0C7ooLUIxfVRHcI9WWnic"},{"id":46,"type":"text","selection":{"start":0,"end":7},"recordId":"NicZdBnNGoIX7FxecnAc66nunac"},{"id":47,"type":"text","selection":{"start":0,"end":35},"recordId":"Q7kxdIZhMopEDXxZ6CEcqztEn3b"},{"id":48,"type":"text","selection":{"start":0,"end":55},"recordId":"H5zmdoYSDonC2vxNVhicCXBMnFd"}],"pasteFlag":"763a5488-2fa9-4107-aa84-183dd617ebd3"}"></span>多头注意力是利用多个查询，来平行地计算从输入信息中选取多个信息。每个注意力关注输入信息的不同部分。
多头注意力机制将输入序列的表示拆分成多个子空间（头），然后在每个子空间内独立地计算注意力权重，最后将各个子空间的结果拼接起来。多头注意力机制使得模型可以在不同的表示子空间中捕获到不同的上下文信息，增强对复杂数据内在结构的建模能力(3-5)。
工作流程:
（1）子空间Q, K, V 向量的生成
给定嵌入向量和头 (Head) 的数目h，在每个头（子空间）中都会通过一组不同的权重矩阵
对输入序列进行线性投影变换得到
, 其维度为
（2）多头并行计算
对每个头独立计算缩放点乘注意力：
（3）多头合并输出
将所有头的输出拼接后通过线性层融合作为多头注意力机制的输出：
[[文件:多头注意力机制.png|缩略图|多头注意力机制]]
=== 注意力机制的应用 ===