更改

Causal Emergence (查看源代码)

2025年3月8日 (六) 16:28的版本

删除60字节、 2025年3月8日 (六) 16:28

→‎Identification of Causal Emergence

第317行：第317行：

|}

−

==Identification of ~~Causal Emergence~~==

+

==Identification of causal emergence==

Some works on quantifying emergence through measures of causal effects and other information-theoretic indicators have been introduced previously. However, in practical applications, we can often only collect observational data and cannot obtain the true dynamics of the system. Therefore, identifying whether causal emergence has occurred in a system from observable data is a more important problem. The following describes two identification methods of causal emergence. The first is the approximate method based on [[Rosas's causal emergence theory]] (the method based on mutual information approximation and the method based on [[machine learning]]). The second is the [[neural information squeezer]] (NIS, NIS+) method proposed by Chinese scholars.

−

+

====Approximate method based on Rosas's causal emergence theory====

−

====Approximate ~~Method Based~~ on Rosas's ~~Causal Emergence Theory~~====

[[Rosas's causal emergence theory]] emplys quantification method based on both [[synergistic information]] and [[unique information]]. The second method can bypass the combinatorial explosion problem of multivariate variables, but it depends on the coarse-graining method and the selection of macroscopic state variable <math>V</math>. To solve this problem, the author gives two solutions. One is to specify a macroscopic state <math>V</math> by the researcher, and the other is a machine learning-based method that allows the system to automatically learn the macroscopic state variable <math>V</math> by maximizing <math>\mathrm{\Psi}</math>. The following sections introduce these two methods in detail:

−

+

=====Method based on mutual information approximation=====

−

=====Method ~~Based~~ on ~~Mutual Information Approximation~~=====

Although [[Rosas's causal emergence theory]] has given a strict definition of causal emergence. However it involves the combinatorial explosion problem of many variables in the calculation, making it difficult to apply this method to actual systems. To solve this problem, Rosas et al. bypassed the exact calculation of unique information and synergistic information <ref name=":5" /> and proposed an approximate formula that only needs to calculate [[mutual information]]. They also derived a sufficient condition for determining the occurrence of causal emergence.

−

The authors proposed three new indicators based on [[mutual information]], <math>\mathrm{\Psi}</math>, <math>\mathrm{\Delta}</math> and <math>\mathrm{\Gamma}</math>, which can be used to identify causal emergence, [[causal decoupling]] and [[downward causation]] in the system respectively. The specific calculation formulas of the three indicators are as follows:

−

* Indicator for judging causal emergence:

第336行：第332行：

<math>\Psi_{t, t + 1}(V):=I\left(V_t ; V_{t + 1}\right)-\sum_j I\left(X_t^j ; V_{t + 1}\right)</math>

|{{EquationRef|1}}}}

−

Here <math>X_t^j</math> represents the microscopic variable at time t in the j-th dimension, and <math>V_t ; V_{t + 1}</math> respectively represent macroscopic state variables at two consecutive times. Rosas et al. defined that when <math>\mathrm{\Psi}>0</math>, emergence occurs in the system; but when <math>\mathrm{\Psi}<0</math>, we cannot determine whether the system has emergence, because this condition is only a sufficient condition for the occurrence of causal emergence.

−

* Indicator for judging [[downward causation]]:

−

<math>\Delta_{t, t + 1}(V):=\max _j\left(I\left(V_t ; X_{t + 1}^j\right)-\sum_i I\left(X_t^i ; X_{t + 1}^j\right)\right)</math>

−

When <math>\mathrm{\Delta}>0</math>, there is downward causation from macroscopic variable <math>V</math> to microscopic variable <math>X</math>.

−

* Indicator for judging [[causal decoupling]]:

−

<math>\Gamma_{t, t + 1}(V):=\max _j I\left(V_t ; X_{t + 1}^j\right)</math>

−

When <math>\mathrm{\Delta}>0</math> and <math>\mathrm{\Gamma}=0</math>, causal emergence occurs in the system and there is causal decoupling.

−

The reason why we can use <math>\mathrm{\Psi}</math> to identify the occurrence of causal emergence is that <math>\mathrm{\Psi}</math> is also the lower bound of unique information. We have the following relationship:

−

<math>Un(V_t;X_{t + 1}|X_t)\geq I\left(V_t ; V_{t + 1}\right)-\sum_j I\left(X_t^j ; V_{t + 1}\right)+Red(V_t, V_{t + 1};X_t)</math>

−

Since <math>Red(V_t, V_{t + 1};X_t)</math> is non-negative, we can thus propose a sufficient but not necessary condition: when <math>\Psi_{t, t + 1}(V)>0</math>.

−

In summary, this method is relatively convenient to calculate because it is based on mutual information, and there is no assumption or requirement of Markov property for the dynamics of the system. However, this theory also has many shortcomings: 1) The indicators <math>\mathrm{\Psi}</math>, <math>\mathrm{\Delta}</math> and <math>\mathrm{\Gamma}</math> based only on [[mutual information]] and do not account for causality; 2) The method provides only a sufficient condition for causal emergence; 3) Results depend heavily on the selection of macroscopic variables, with different choices leading to significantly different outcomes; 4) For systems with large amounts of redundant information or many variables, the [[computational complexity]] becomes very high. At the same time, since <math>\Psi</math> is an approximate calculation, there will be a very large error in high-dimensional systems, and it is also very easy to obtain negative values, making it difficult to reliably determine the presence of causal emergence.

−

To verify that the information related to macaque movement is an emergent feature of its cortical activity, Rosas et al. conducted an experiment using the electrocorticogram (ECoG) of macaques as the observational data of microscopic dynamics. To obtain the macroscopic state variable <math>V</math>, the authors chose the time series data of the limb movement trajectory of macaques obtained by motion capture (MoCap). The ECoG consisted of 64 channels, while the MoCap data consisted of 3 channels. Since the original MoCap data does not satisfy the conditional independence assumption of the supervenience feature, they used [[partial least squares]] and [[support vector machine]] algorithms to infer the part of neural activity encoded in the ECoG signal related to predicting macaque behavior. They then speculated that this information is an emergent feature of potential neural activity. Finally, based on the microscopic state and the calculated macroscopic features, the authors verified the existence of causal emergence.

−

=====Machine ~~Learning~~-based ~~Method~~=====

+

=====Machine learning-based method=====

Kaplanis et al. <ref name=":2" /> use a [[representation learning]] to learn the macroscopic state variable <math>V</math> by maximizing <math>\mathrm{\Psi}</math> (i.e., Equation {{EquationNote|1}}). Specifically, the authors use a neural network <math>f_{\theta}</math> to learn the representation function that coarsens the microscopic input <math>X_t</math> into the macroscopic output <math>V_t</math>. At the same time, they use neural networks <math>g_{\phi}</math> and <math>h_{\xi}</math> to learn the calculation of mutual information, such as <math>I(V_t;V_{t + 1})</math> and <math>\sum_i(I(V_{t + 1};X_{t}^i))</math>, respectively. Finally, this method optimizes the neural network by maximizing the difference between the two (i.e., <math>\mathrm{\Psi}</math>). The architecture diagram of this neural network system is shown in Figure a below.

−

[[文件:Architectures for learning causal emergent representations1.png|居左|替代=|700x600像素]]

−

Figure b shows a toy model example. The microscopic input <math>X_t(X_t^1,...,X_t^6) \in \{0,1\}^6</math> has six dimensions, with each dimension being binary (0 or 1). <math>X_{t + 1}</math> is the output of <math>X_{t}</math> at the next moment. The macroscopic variable is <math>V_{t}=\oplus_{i = 1}^{5}X_t^i</math>, where <math>\oplus_{i = 1}^{5}X_t^i</math> represents the result of adding the first five dimensions of the microscopic input <math>X_t</math> and taking the modulo 2. There exists a probability <math>\gamma</math> that the macroscopic states at two consecutive moments are equal (<math>p(\oplus_{j = 1..5}X_{t + 1}^j=\oplus_{j = 1..5}X_t^j)= \gamma</math>). The sixth dimension of the microscopic input at two consecutive moments is equal with <math>\gamma_{extra}</math> (<math>p(X_{t + 1}^6=X_t^6)= \gamma_{extra}</math>).

−

The results show that in the simple example shown in Figure (b), by maximizing <math>\mathrm{\Psi}</math> through the model constructed in Figure a, the experiment finds that the learned <math>\mathrm{\Psi}</math> is approximately equal to the true groundtruth <math>\mathrm{\Psi}</math>. This outcome verifies the effectiveness of model learning. This system is effective in identifying the occurrence of causal emergence. However, this method also has the problem of being difficult to deal with complex multivariate situations. This is because the number of neural networks on the right side of the figure is proportional to the number of macroscopic and microscopic variable pairs. Therefore, the more the number of microscopic variables (dimensions), the more the number of neural networks will increase proportionally, which will lead to an increase in computational complexity. In addition, this method is only tested on very few cases, so it cannot be scaled up yet. Moreover, since the network calculates an approximate index of causal emergence, it inherits the limitations of the earlier approximate algorithms.

−

====Neural ~~Information Squeezer~~====

+

====Neural information squeezer====

In recent years, emerging [[artificial intelligence]] technologies have overcome a series of major problems. Machine learning methods incorporate carefully designed [[neural network]] structures and [[automatic differentiation]] technologies. These technologies enable theapproximation of any function in a vast function space. Therefore, [[Zhang Jiang]] et al. proposed a data-driven method using neural networks to identify causal emergence from time series data <ref name="NIS">Zhang J, Liu K. Neural information squeezer for causal emergence[J]. Entropy, 2022, 25(1): 26.</ref><ref name=":6" />. This method can automatically extract effective coarse-graining strategies and macroscopic dynamics, overcoming various deficiencies of the Rosas method <ref name=":5" />.

−

In this work, the input consists of time series data <math>(X_1,X_2,...,X_T )</math>, where <math>X_t\equiv (X_t^1,X_t^2,…,X_t^p )</math> and <math>p</math> denotes the dimensionality of the input data. The author assumes that this set of data is generated by a general [[stochastic dynamical system]]:

−

−

[math]X(t)[/math] represents the microscopic state variable, [math]f[/math] denotes the microscopic dynamics, and <math>\xi</math> represents the noise in the system. However, <math>f</math> is unknown.

−

The causal emergence identification problem is defined as the following functional optimization problem:

−

{{NumBlk|:|

第406行：第382行：

</math>

|{{EquationRef|2}}}}

−

[math]\mathcal{J}[/math] represents the dimension-averaged <math>EI</math> (see the entry [[effective information]]), and <math>\mathrm{\phi}</math> is the coarse-graining strategy function. <math>f_{q}</math> denotes the macroscopic dynamics, and <math>q</math> represents the dimension of the coarsened macroscopic state. [math]\hat{X}_{t + 1}[/math] is the prediction of the microscopic state at time <math>t + 1</math> by the entire framework. This prediction is obtained by applying the inverse coarse-graining function ([math]\phi^{\dagger}[/math]) to the macroscopic state prediction of the macroscopic state <math>\hat{Y}_{t + 1}</math> at time <math>t + 1</math>. Here [math]\hat{Y}_{t + 1}\equiv f_q(Y_t)[/math] represents the prediction of the macroscopic state at time <math>t + 1</math> by the dynamics learner, based on the macroscopic state [math]Y_t[/math] at time <math>t</math>. The macroscopic state [math]Y_t\equiv \phi(X_t)[/math] at time <math>t</math> is obtained by coarse-graining the microscopic state [math]X_t[/math] using the function [math]\phi[/math]. Finally, the difference between [math]\hat{X}_{t + 1}[/math] and the real microscopic state data [math]X_{t + 1}[/math] is compared to obtain the microscopic prediction error.

−

The entire optimization framework is shown below:

−

[[文件:NIS_Optimization.png|替代=NIS优化框架|居左|400x400像素|NIS优化框架]]

−

The objective function, <math>EI</math>, is a functional of [math]\phi,\hat{f}_q,\phi^{\dagger}[/math], making it difficult to optimize as [math]q[/math] is a hyperparameter. We need to use machine learning methods to solve it.

−

=====NIS=====

To identify causal emergence in the system, the author proposes a [[neural information squeezer]] (NIS), a neural network architecture designed for this purpose <ref name="NIS" />. The architecture is based on an encoder-dynamics learner-decoder framework, consisting of three components: coarse-graining the raw data into macrostates, fitting the macrodynamics, and performing a reverse coarse-graining operation (decoding the macrostates, combined with random noise, back into microstates). Among them, the authors use [[invertible neural network]] (INN) to construct the encoder and decoder, which approximately correspond to the coarse-graining function ϕ and the inverse coarse-graining function ϕ† respectively. [[Invertible neural networks]] are used because they can be inverted to approximate theinverse coarse-graining function (i.e., [math]\phi^{\dagger}\approx \phi^{-1}[/math]). This model framework can be regarded as a neural information squeezer. The model compresses noisy microscopic state data into a macroscopic state through a narrow information channel, discarding irrelevant information. This strengthens the causality of macroscopic dynamics and allows the model to get the predictions of the microscopic state via decoding. The model framework of the NIS method is shown in the following figure:

−

[[文件:The_framework_diagram_of_the_NIS_model1.png|替代=|无框|600x600像素]]

−

Specifically, the encoder function [math]\phi[/math] consists of two parts:

−

<math>

\phi\equiv \chi\circ\psi

</math>

−

Here [math]\psi[/math] is an invertible function implemented via an [[invertible neural network]], and [math]\chi[/math] is a [[projection function]] that removes the last <math>p - q</math> dimensional components from the <math>p</math>-dimensional vector. Here, <math>p,q</math> represent the dimensions of the microscopic and macroscopic variables, respectively. <math>\circ</math> is the composition operation of functions.

−

The decoder is the function [math]\phi^{\dagger}[/math], which is defined as:

−

<math>

\phi^{\dagger}(y)\equiv \psi^{-1}(y\bigoplus z)

</math>

−

Here [math]z\sim\mathcal{Ν}\left (0,I_{p - q}\right )[/math] is a [math]p-q[/math]-dimensional random vector that obeys the standard normal distribution.

−

However, if we directly optimize the dimension-averaged EI, there will be certain difficulties. The article <ref name="NIS" /> does not directly optimize Equation {{EquationNote|1}}, but adopts a clever method. To solve this problem, the author divides the optimization process into two stages. The first stage is to minimize the microscopic state prediction error under the condition of a given macroscopic scale <math>q</math>, that is, <math>\min _{\phi, f_q, \phi^{\dagger}}\left\|\phi^{\dagger}(Y(t + 1)) - X_{t + 1}\right\|<\epsilon</math> and obtain the optimal macroscopic state dynamics [math]f_q^\ast[/math]; the second stage is to search for the hyperparameter <math>q</math> to maximize the EI [math]\mathcal{J}[/math], that is, <math>\max_{q}\mathcal{J}(f_{q}^\ast)</math>. Practice has proved that this method can effectively find macroscopic dynamics and coarse-graining functions, but it cannot truly maximize EI in advance.

−

In addition to being able to automatically identify causal emergence based on time series data, this framework also has good theoretical properties. There are two important theorems:

−

'''Theorem 1''': The [[information bottleneck]] of the neural information squeezer. That is, for any bijection <math>\mathrm{\psi}</math>, projection <math>\chi</math>, macroscopic dynamics <math>f</math> and Gaussian noise <math>z_{p - q}\sim\mathcal{Ν}\left (0,I_{p - q}\right )</math>,

−

<math>

I\left(Y_t;Y_{t + 1}\right)=I\left(X_t;{\hat{X}}_{t + 1}\right)

</math>

−

always holds. This means that all the information discarded by the encoder is actually noise information unrelated to prediction.

−

'''Theorem 2''': For a trained model, <math>I\left(X_t;{\hat{X}}_{t + 1}\right)\approx I\left(X_t;X_{t + 1}\right)</math>.

Therefore, combining Theorem 1 and Theorem 2, we can obtain for a trained model:

−

<math>

第475行：第432行：

</math>

−

+

======Comparison with classical theories======

−

======Comparison with ~~Classical Theories~~======

The [[NIS]] framework has many similarities with the [[computational mechanics]] framework mentioned in the previous sections. NIS can be regarded as an <math>\epsilon</math>-machine. The set of all historical processes <math>\overleftarrow{S}</math> in [[computational mechanics]] can be regarded as a set of microscopic states. All <math>R \in \mathcal{R}</math> represent macroscopic states. The function <math>\eta</math> acts as a coarse-graining function, while <math>\epsilon</math> represents an effective coarse-graining strategy. <math>T</math> corresponds to a effective macroscopic dynamic. The characteristic of minimum randomness index characterizes the determinism of macroscopic dynamics and can be replaced by EI in causal emergence. At this point, the encoded macroscopic state converges to the effective state, which can be regarded as the [[causal state]] in computational mechanics.

−

At the same time, the [[NIS]] framework also has similarities with the G-emergence theory mentioned earlier. For example, [[NIS]] also adopts the idea of [[Granger causality]]: optimizing the effective macroscopic state by predicting the microscopic state at the next time step. However, there are several obvious differences between these two frameworks: a) In the G-emergence theory, the macroscopic state needs to be manually selected, while in [[NIS]], the macroscopic state is obtained by automatically optimizing the coarse-graining strategy; b) NIS uses neural networks to predict future states, whereas G-emergence relies on autoregressive techniques to fit the data.

−

======Computational ~~Examples~~======

+

======Computational examples======

The author of NIS conducted experiments in the [[spring oscillator model]], and the results are shown in the following figure. Figure a demostrates that the results for the next moment align linearly with the iterative macroscopic dynamics, validating the model's effectiveness. Figure b illustrates that the learned and real dynamics align, further validating the model. Figure c illustrates the model's accuracy in multi-step predictions, as the predicted and real curves are closely aligned. Figure d shows the magnitude of causal emergence at different scales. Causal emergence is most significant at scale is 2, consistent with the real spring oscillator model, where only two states (position and velocity) are needed to describe the system.

−

[[文件:弹簧振子模型1.png|居左|600x600像素|替代=弹簧振子模型1|弹簧振子模型]]

第491行：第445行：

Although NIS took the lead in proposing a scheme to optimize EI to identify causal emergence in data, this method has some shortcomings: the author divides the optimization process into two stages, but does not truly maximize the EI, that is, Equation {{EquationNote|1}}. Therefore, [[Yang Mingzhe]] et al. <ref name=":6" /> further improved this method and proposed the [[NIS+]] scheme. By introducing reverse dynamics and [[reweighting technique]], the original maximization of EI is transformed into maximizing its variational lower bound. This is achieved using [[variational inequality]], enabling the direct optimization of the objective function.

−

+

======Mathematical principles======

−

======Mathematical ~~Principles~~======

Specifically, according to variational inequality and [[inverse probability weighting]] method, the constrained optimization problem given by Equation {{EquationNote|2}} can be transformed into the following unconstrained minimization problem:

−

<math>\min_{\omega,\theta,\theta'} \sum_{i = 0}^{T - 1}w(\boldsymbol{x}_t)||\boldsymbol{y}_t-g_{\theta'}(\boldsymbol{y}_{t + 1})||+\lambda||\hat{\boldsymbol{x}}_{t + 1}-\boldsymbol{x}_{t + 1}||</math>

−

Here <math>g</math> represents the reverse dynamics, which can be approximated by a neural network and trained with the data of [math]y_{t + 1},y_{t}[/math] through the macroscopic state. <math>w(x_t)</math> is the inverse probability weight, and the specific calculation method is as follows:

−

<math>

w(\boldsymbol{x}_t)=\frac{\tilde{p}(\boldsymbol{y}_t)}{p(\boldsymbol{y}_t)}=\frac{\tilde{p}(\phi(\boldsymbol{x}_t))}{p(\phi(\boldsymbol{x}_t))}

</math>

−

Here <math>\tilde{p}(\boldsymbol{y}_{t})</math> is the target distribution and <math>p(\boldsymbol{y}_{t})</math> is the original distribution of the data.

−

+

======Workflow and model architecture======

−

======Workflow and ~~Model Architecture~~======

The following figure shows the entire model framework of NIS+. Figure a is the input of the model: time series data, such as trajectory sequence, continuous image sequence, and EEG time series data; Figure c is the output of the model, which includes causal emergence measures, macroscopic dynamics, emergent patterns, and coarse-graining strategies; Figure b is the specific model architecture. Different from the NIS method, two parts of reverse dynamics and reweighting technology are added.

−

[[文件:NIS+.png|居左|600x600像素|替代=NIS模型框架图|NIS+模型框架图]]

−

+

======Case analysis======

−

======Case ~~Analysis~~======

The article conducts experiments on datasets including [[SIR dynamics]], the [[Boids model]], [[Game of Life]], and fMRI signals from the [[brain nervous system]] of human subjects. This section focuses on the bird flock and brain signal experiments for detailed analysis.

−

The following figure shows the experimental results of NIS+ learning the flocking behavior of the Boids model. (a) and (e) give the actual and predicted trajectories of bird flocks under different conditions. Specifically, the author divides the bird flock into two groups and compares the multi-step prediction results under different noise levels (<math>\alpha</math> is 0.001 and 0.4 respectively). The prediction is good when the noise is relatively small, and the prediction curve will diverge when the noise is relatively large. (b) shows that the mean absolute error (MAE) of multi-step prediction gradually increases as the radius r increases. (c) shows the change of [[causal emergence measure]] <math>\Delta J</math> and prediction error (MAE) under different dimensions (q) with the change of training epoch. The author finds that causal emergence is most significant when the macroscopic state dimension q = 8. (d) is the attribution analysis of macroscopic variables to microscopic variables, and the obtained significance map intuitively describes the learned coarse-graining function. Here, each macroscopic dimension can correspond to the spatial coordinate (microscopic dimension) of each bird. The darker the color, the higher the correlation. Here, the microscopic coordinates corresponding to the maximum correlation of each macroscopic state dimension are highlighted with orange dots. These attribution significance values are obtained by using the [[Integrated Gradient]] (referred to as IG) method. The horizontal axis represents the x and y coordinates of 16 birds in the microscopic state, and the vertical axis represents 8 macroscopic dimensions. The light blue dotted line distinguishes the coordinates of different individual Boids, and the blue solid line separates the two bird flocks. (f) and (g) represent the changing trends of causal emergence measure ΔJ and normalized error MAE under different noise levels. (f) represents the influence of changes in external noise (that is, adding observation noise to microscopic data) on causal emergence. (g) represents the influence of internal noise (represented by α, added by modifying the dynamics of the Boids model) on causal emergence. In (f) and (g), the horizontal line represents the threshold that violates the error constraint in Equation {{EquationNote|1}}. When the normalized MAE is greater than the threshold of 0.3, the constraint is violated and the result is unreliable.

−

This set of experiments shows that [[NIS+]] can learn macroscopic states and coarse-graining strategies by maximizing EI. This maximization improves the model's generalization ability to situations outside the training data range. The learned macroscopic state effectively identifies the average [[group behavior]] and can be attributed to individual positions using the gradient integration method. The degree of causal emergence increases with external noise but decreases with increase of internal noise. This result indicates that the model reduces external noise through coarse-graining but cannot mitigate internal noise.

−

[[文件:NIS+ boids.png|居左|700x700像素|鸟群中的因果涌现]]

−

The brain experiment uses fMRI data collected from two experiments involving 830 human subjects. In the first group, subjects performed a visual task by watching a short movie clip, during which recordings were made. In the second group of experiments, they were asked to be in a resting state and the recording was completed. To manage the high original dimension, the authors reduced the original 14000-dimensional data to 100 dimensions using the [[Schaefer atlas]] method, with each dimension corresponding to a brain region. After that, the authors learned these data through NIS+ and extracted the dynamics at six different macroscopic scales. Figure a shows the multi-step prediction error results at different scales. Figure b shows the comparison of EI of NIS and NIS+ methods on different macroscopic dimensions in the resting state and the visual task of watching movies. The authors found that in the visual task, causal emergence is most significant when the macroscopic state dimension is q = 1. Through attribution analysis, it is found that the visual area plays the largest role (Figure c), which is consistent with the real scene. Figure d shows different perspective views of brain region attribution. In the resting state, one macroscopic dimension is not enough to predict the microscopic time series data. The dimension with the largest causal emergence is between 3 and 7 dimensions.

−

[[文件:NIS+ 脑数据.png|居左|700x700像素|脑神经系统中的因果涌现]]

−

These experiments demonstrates that [[NIS+]] identifies causal emergence, discovers macroscopic dynamics, and develops coarse-graining strategies. Additionally, other experiments show that EI maximization enhances the model's e out-of-distribution generalization ability.

Complexivist Ran

453

个编辑