| 第92行： |
第92行： |
| | | | |
| | | | |
| − | | + | <math></math> |
| − | | |
| − | \documentclass{article}
| |
| − | \usepackage{amsmath}
| |
| − | \usepackage{amssymb}
| |
| − | \usepackage{graphicx}
| |
| − | \usepackage{caption}
| |
| − | \usepackage{hyperref}
| |
| − | | |
| − | \begin{document}
| |
| − | | |
| − | \title{Nonnegative Decomposition of Multivariate Information}
| |
| − | \author{Paul L. Williams$^1$, Randall D. Beer$^{1,2}$ \\
| |
| − | $^1$Cognitive Science Program and $^2$School of Informatics and Computing \\
| |
| − | Indiana University, Bloomington, Indiana 47406 USA \\
| |
| − | (Dated: November 26, 2024)}
| |
| − | \date{}
| |
| − | \maketitle
| |
| − | | |
| − | \begin{abstract}
| |
| − | Of the various attempts to generalize information theory to multiple variables, the most widely utilized, interaction information, suffers from the problem that it is sometimes negative. Here we reconsider from first principles the general structure of the information that a set of sources provides about a given variable. We begin with a new definition of redundancy as the minimum information that any source provides about each possible outcome of the variable, averaged over all possible outcomes. We then show how this measure of redundancy induces a lattice over sets of sources that clarifies the general structure of multivariate information. Finally, we use this redundancy lattice to propose a definition of partial information atoms that exhaustively decompose the Shannon information in a multivariate system in terms of the redundancy between synergies of subsets of the sources. Unlike interaction information, the atoms of our partial information decomposition are never negative and always support a clear interpretation as informational quantities. Our analysis also demonstrates how the negativity of interaction information can be explained by its confounding of redundancy and synergy.
| |
| − | \end{abstract}
| |
| − | | |
| − | \section{Introduction}
| |
| − | From its roots in Shannon's seminal work on reliability and coding in communication systems, information theory has grown into a ubiquitous general tool for the analysis of complex systems, with application in neuroscience, genetics, physics, machine learning, and many other areas. Somewhat surprisingly, the vast majority of work in information theory concerns only the simplest possible case: the information that a single variable provides about another. This is quantified by Shannon's mutual information, which is by far the most widely used concept from information theory \cite{shannon1949mathematical}. The second most popular concept, conditional mutual information, considers interactions between multiple variables in only the most rudimentary sense: it seeks to eliminate the influence of other variables in order to isolate the dependency between two variables of interest. In contrast, many of the most interesting and challenging scientific questions, such as many-body problems in physics \cite{pines1997many}, n-person games in game theory \cite{luce1989games}, and population coding in neuroscience \cite{dayan2001theoretical}, involve understanding the structure of interactions between three or more variables.
| |
| − | | |
| − | \section{The Structure of Multivariate Information}
| |
| − | Suppose we are given a random variable \( S \) and a random vector \( \mathbf{R} = \{R_1, R_2, \ldots, R_{n-1}\} \). Then our goal is to decompose the information that \( \mathbf{R} \) provides about \( S \) in terms of the partial information contributed either individually or jointly by various subsets of \( \mathbf{R} \). For example, in a neuroscience context, \( S \) may correspond to a stimulus that takes on different values and \( \mathbf{R} \) to the evoked responses of different neurons. In this case, we would like to quantify the information that the joint neural response provides about the stimulus, and to distinguish between information due to responses of individual neurons versus combinations of them \cite{rieke1999spikes}.
| |
| − | | |
| − | \section{Measuring Redundancy}
| |
| − | Let \( \mathbf{A}_1, \mathbf{A}_2, \ldots, \mathbf{A}_k \) be nonempty and potentially overlapping subsets of \( \mathbf{R} \), which we call sources. How can we quantify the redundant information that all sources provide about \( S \)?
| |
| − | | |
| − | The information supplied by each \( \mathbf{A}_i \) is given simply by \( I(S; \mathbf{A}_i) \), the mutual information between \( S \) and \( \mathbf{A}_i \). However, it is crucial to note that mutual information is actually a measure of average or expected information, where the expected value is taken over outcomes of the random variables. Thus, for instance, two sources might provide the same average amount of information, while also providing information about different outcomes of \( S \). Stated formally, the information provided by a source \( \mathbf{A} \) can be written as
| |
| − | \begin{equation}
| |
| − | I(S; \mathbf{A}) = \sum_{s} p(s) I(S = s; \mathbf{A})
| |
| − | \end{equation}
| |
| − | where the specific information \( I(S = s; \mathbf{A}) \) quantifies the information associated with a particular outcome \( s \) of \( S \). Various definitions of specific information have been proposed to quantify different relationships between \( S \) and \( \mathbf{A} \) (see Appendix A), but for our purposes the most useful is
| |
| − | \begin{equation}
| |
| − | I(S = s; \mathbf{A}) = \sum_{\mathbf{a}} p(\mathbf{a} | s) \left[ \log \frac{1}{p(s)} - \log \frac{1}{p(s | \mathbf{a})} \right].
| |
| − | \end{equation}
| |
| − | The term \( \frac{1}{p(s)} \) is called the surprise of \( s \), so \( I(S = s; \mathbf{A}) \) is the average reduction in surprise of \( s \) given knowledge of \( \mathbf{A} \). In other words, \( I(S = s; \mathbf{A}) \) quantifies the information that \( \mathbf{A} \) provides about each particular outcome \( s \in S \), while \( I(S; \mathbf{A}) \) is the expected value of this quantity over all outcomes of \( S \).
| |
| − | | |
| − | \section{Partial Information Decomposition}
| |
| − | The redundant information associated with each node of the redundancy lattice includes, but is not limited to, the redundant information provided by all nodes lower in the lattice. Thus, moving from node to node up the lattice, \( I_{\min} \) can be thought of as a kind of "cumulative information function," effectively integrating the information provided by increasingly inclusive collections of sources. Next, we derive an inverse of \( I_{\min} \) called the partial information function (PI-function). Whereas \( I_{\min} \) quantifies cumulative information, the PI-function measures the partial information contributed uniquely by each particular collection of sources. This partial information will form the atoms into which we decompose the total information that \( \mathbf{R} \) provides about \( S \).
| |
| − | | |
| − | For a collection of sources \( \alpha \in \mathcal{A}(\mathbf{R}) \), the PI-function, denoted \( \Pi_{\mathbf{R}} \), is defined implicitly by
| |
| − | \begin{equation}
| |
| − | I_{\min}(S; \alpha) = \sum_{\beta \prec \alpha} \Pi_{\mathbf{R}}(S; \beta).
| |
| − | \end{equation}
| |
| − | Formally, \( \Pi_{\mathbf{R}} \) corresponds to the Möbius inverse of \( I_{\min} \) \cite{rota1964foundations, stanley1997enumerative}. From this relationship, it is clear that \( \Pi_{\mathbf{R}} \) can be calculated recursively as
| |
| − | \begin{equation}
| |
| − | \Pi_{\mathbf{R}}(S; \alpha) = I_{\min}(S; \alpha) - \sum_{\beta \prec \alpha} \Pi_{\mathbf{R}}(S; \beta).
| |
| − | \end{equation}
| |
| − | Put into words, \( \Pi_{\mathbf{R}}(S; \alpha) \) quantifies the information provided redundantly by the sources of \( \alpha \) that is not provided by any simpler collection of sources (i.e., any \( \beta \) lower than \( \alpha \) on the redundancy lattice). In Appendix D, it is shown that \( \Pi_{\mathbf{R}} \) can be written in closed form as
| |
| − | \begin{equation}
| |
| − | \Pi_{\mathbf{R}}(S; \alpha) = I_{\min}(S; \alpha) - \sum_{s} p(s) \max_{\beta \in \alpha^{-}} \min_{\mathbf{B} \in \beta} I(S = s; \mathbf{B})
| |
| − | \end{equation}
| |
| − | where \( \alpha^{-} \) represents the nodes immediately below \( \alpha \) in the redundancy lattice. From this formulation, it is readily shown that \( \Pi_{\mathbf{
| |
| | | | |
| | == 延伸与应用 == | | == 延伸与应用 == |