# 贝叶斯推断

## 贝叶斯规则介绍

A geometric visualisation of Bayes' theorem. In the table, the values 2, 3, 6 and 9 give the relative weights of each corresponding condition and case. The figures denote the cells of the table involved in each metric, the probability being the fraction of each figure that is shaded. This shows that P(A|B) P(B) = P(B|A) P(A) i.e. P(A|B) = 模板:Sfrac . Similar reasoning can be used to show that P(¬A|B) = 模板:Sfrac etc.

### 正式定义

Contingency table

hypothesis
H
Violates
hypothesis
¬H

Total
Has evidence
E
P(H|E)·P(E)
= P(E|H)·P(H)
P(¬H|E)·P(E)
= P(E|¬H)·P(¬H)
P(E)
No evidence
¬E
P(H|¬E)·P(¬E)
= P(¬E|H)·P(H)
P(¬H|¬E)·P(¬E)
= P(¬E|¬H)·P(¬H)
P(¬E) =
1−P(E)
Total    P(H) P(¬H)&hairsp;=&hairsp;1−P(H) 1

Bayesian inference derives the posterior probability as a consequence of two antecedents: a prior probability and a "likelihood function" derived from a statistical model for the observed data. Bayesian inference computes the posterior probability according to Bayes' theorem:

$\displaystyle{ P(H\mid E) = \frac{P(E\mid H) \cdot P(H)}{P(E)} }$

where:

• $\displaystyle{ \textstyle H }$ stands for any hypothesis whose probability may be affected by data (called evidence below). Often there are competing hypotheses, and the task is to determine which is the most probable.
• $\displaystyle{ \textstyle P(H) }$, the prior probability, is the estimate of the probability of the hypothesis $\displaystyle{ \textstyle H }$ before the data $\displaystyle{ \textstyle E }$, the current evidence, is observed.
• $\displaystyle{ \textstyle E }$, the evidence, corresponds to new data that were not used in computing the prior probability.
• $\displaystyle{ \textstyle P(H\mid E) }$, the posterior probability, is the probability of $\displaystyle{ \textstyle H }$ given $\displaystyle{ \textstyle E }$, i.e., after $\displaystyle{ \textstyle E }$ is observed. This is what we want to know: the probability of a hypothesis given the observed evidence.
• $\displaystyle{ \textstyle P(E\mid H) }$ is the probability of observing $\displaystyle{ \textstyle E }$ given $\displaystyle{ \textstyle H }$, and is called the likelihood. As a function of $\displaystyle{ \textstyle E }$ with $\displaystyle{ \textstyle H }$ fixed, it indicates the compatibility of the evidence with the given hypothesis. The likelihood function is a function of the evidence, $\displaystyle{ \textstyle E }$, while the posterior probability is a function of the hypothesis, $\displaystyle{ \textstyle H }$.
• $\displaystyle{ \textstyle P(E) }$ is sometimes termed the marginal likelihood or "model evidence". This factor is the same for all possible hypotheses being considered (as is evident from the fact that the hypothesis $\displaystyle{ \textstyle H }$ does not appear anywhere in the symbol, unlike for all the other factors), so this factor does not enter into determining the relative probabilities of different hypotheses.

• $\displaystyle{ \textstyle H }$ 代表任何可能受数据(也称为证据)影响的假设。通常我们有好几个备选的假设，我们的目标就是确定哪一个假设是最有可能的。
• $\displaystyle{ \textstyle H }$ ，先验概率，是假设 $\displaystyle{ \textstyle H }$ 在当前的证据 $\displaystyle{ \textstyle E }$被观察到之前的概率。
• $\displaystyle{ \textstyle E }$，证据，则是我们刚刚观测到的还没有用于计算先验概率的新数据。
• $\displaystyle{ \textstyle P(H\mid E) }$，后验概率，是给定了证据 $\displaystyle{ \textstyle E }$后得到 $\displaystyle{ \textstyle H }$ 的概率。这就是我们想知道的: 当我们观察到某些证据后假设的概率。
• $\displaystyle{ \textstyle P(E\mid H) }$ 是指在给定假设$\displaystyle{ \textstyle H }$ 的情况下，观察到$\displaystyle{ \textstyle P(E) }$的概率，我们把它称为似然函数。它表明了证据与给定假设的兼容性。似然函数是关于证据的一个函数，$\displaystyle{ \textstyle E }$，而后验概率是关于假设的一个函数，$\displaystyle{ \textstyle H }$
• $\displaystyle{ \textstyle E }$有时被称为边际似然或“模型证据”。这个值对于所有被考虑的可能假设是相同的，它不影响不同假设的相对概率。

For different values of $\displaystyle{ \textstyle H }$, only the factors $\displaystyle{ \textstyle P(H) }$ and $\displaystyle{ \textstyle P(E\mid H) }$, both in the numerator, affect the value of $\displaystyle{ \textstyle P(H\mid E) }$ – the posterior probability of a hypothesis is proportional to its prior probability (its inherent likeliness) and the newly acquired likelihood (its compatibility with the new observed evidence).

Bayes' rule can also be written as follows:

\displaystyle{ \begin{align}P(H\mid E) &= \frac{P(E\mid H) P(H)}{P(E)} \\ \\ &= \frac{P(E\mid H) P(H)}{P(E\mid H) P(H) + P(E\mid \neg H) P(\neg H)} \\ \\ &= \frac{1}{1 + \left(\frac{1}{P(H)}-1 \right) \frac{P(E\mid \neg H)}{P(E\mid H)} } \\ \end{align} }

because

$\displaystyle{ P(E) = P(E\mid H) P(H) + P(E\mid \neg H) P(\neg H) }$

and

$\displaystyle{ P(H) + P(\neg H) = 1 }$

where $\displaystyle{ \neg H }$ is "not $\displaystyle{ \textstyle H }$", the logical negation of $\displaystyle{ \textstyle H }$.

\displaystyle{ \begin{align}P(H\mid E) &= \frac{P(E\mid H) P(H)}{P(E)} \\ \\ &= \frac{P(E\mid H) P(H)}{P(E\mid H) P(H) + P(E\mid \neg H) P(\neg H)} \\ \\ &= \frac{1}{1 + \left(\frac{1}{P(H)}-1 \right) \frac{P(E\mid \neg H)}{P(E\mid H)} } \\ \end{align} }

$\displaystyle{ P(E) = P(E\mid H) P(H) + P(E\mid \neg H) P(\neg H) }$

$\displaystyle{ P(H) + P(\neg H) = 1 }$

One quick and easy way to remember the equation would be to use Rule of Multiplication:

$\displaystyle{ P(E\cap H) = P(E\mid H) P(H) = P(H\mid E) P(E) }$

### 贝叶斯更新的另一种方案

Bayesian updating is widely used and computationally convenient. However, it is not the only updating rule that might be considered rational.

Ian Hacking noted that traditional "Dutch book" arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. Hacking wrote "And neither the Dutch book argument nor any other in the personalist arsenal of proofs of the probability axioms entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. "

Ian Hacking 指出，传统的“荷兰赌”论点没有指定贝叶斯更新: 它们保留了非贝叶斯更新规则可以避免荷兰书籍的可能性。Hacking写道：“ 无论是荷兰赌的论证，还是个人主义者（个人主义认为概率是对个人信念的度量）对于概率公理的其他证据，都不需要动态假设。没有一个人是贝叶斯主义者。 当个人主义加入了动态假设后则变成了贝叶斯主义。”

Indeed, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics") following the publication of Richard C. Jeffrey's rule, which applies Bayes' rule to the case where the evidence itself is assigned a probability. The additional hypotheses needed to uniquely require Bayesian updating have been deemed to be substantial, complicated, and unsatisfactary.

# 贝叶斯推断的正式描述

## 定义

• $\displaystyle{ x }$, a data point in general. This may in fact be a vector of values.
• $\displaystyle{ \theta }$, the parameter of the data point's distribution, i.e., $\displaystyle{ x \sim p(x \mid \theta) }$ . This may be a vector of parameters.
• $\displaystyle{ \alpha }$, the hyperparameter of the parameter distribution, i.e., $\displaystyle{ \theta \sim p(\theta \mid \alpha) }$ . This may be a vector of hyperparameters.
• $\displaystyle{ \mathbf{X} }$ is the sample, a set of $\displaystyle{ n }$ observed data points, i.e., $\displaystyle{ x_1,\ldots,x_n }$.
• $\displaystyle{ \tilde{x} }$, a new data point whose distribution is to be predicted.
• $\displaystyle{ x }$，一个数据点。可能是一个由多个值组成的矢量。
• $\displaystyle{ \theta }$，是描述数据点分布的参数，即 $\displaystyle{ x \sim p(x \mid \theta) }$。这可能是一个多个参数组成的矢量。
• $\displaystyle{ \alpha }$，是描述参数分布的超参数，即$\displaystyle{ \theta \sim p(\theta \mid \alpha) }$。这可能是一个多个超参数组成的向量。
• $\displaystyle{ \mathbf{X} }$是一组 n 个观测点组成的数据样本，即$\displaystyle{ x_1,\ldots,x_n }$
• $\displaystyle{ \tilde{x} }$ ，一个新的数据点，其分布还尚待预测。

## 贝叶斯推断

• The prior distribution is the distribution of the parameter(s) before any data is observed, i.e. $\displaystyle{ p(\theta \mid \alpha) }$ . The prior distribution might not be easily determined; in such a case, one possibility may be to use the Jeffreys prior to obtain a prior distribution before updating it with newer observations.
• 先验分布是在观测到任何数据之前参数的分布，即$\displaystyle{ p(\theta \mid \alpha) }$。先验分布可能不容易确定; 在这种没有获得任何新观测数据的情况下，我们可以使用 Jeffreys prior
• The sampling distribution is the distribution of the observed data conditional on its parameters, i.e. $\displaystyle{ p(\mathbf{X} \mid \theta) }$ . This is also termed the likelihood, especially when viewed as a function of the parameter(s), sometimes written $\displaystyle{ \operatorname{L}(\theta \mid \mathbf{X}) = p(\mathbf{X} \mid \theta) }$ .
• 抽样分布是观测数据按其参数分布的分布。$\displaystyle{ p(\mathbf{X} \mid \theta) }$.这也被称为似然函数，特别是当它被看作为关于参数的函数时，有时写为$\displaystyle{ \operatorname{L}(\theta \mid \mathbf{X}) = p(\mathbf{X} \mid \theta) }$
• The marginal likelihood (sometimes also termed the evidence) is the distribution of the observed data marginalized over the parameter(s), i.e. $\displaystyle{ p(\mathbf{X} \mid \alpha) = \int p(\mathbf{X} \mid \theta) p(\theta \mid \alpha) \operatorname{d}\!\theta }$ .
• 边际概率(有时也称为证据)是被边缘化的观测数据在参数上的分布，即$\displaystyle{ p(\mathbf{X} \mid \alpha) = \int p(\mathbf{X} \mid \theta) p(\theta \mid \alpha) \operatorname{d}\!\theta }$
• The posterior distribution is the distribution of the parameter(s) after taking into account the observed data. This is determined by Bayes' rule, which forms the heart of Bayesian inference:
$\displaystyle{ p(\theta \mid \mathbf{X},\alpha) = \frac{p(\theta,\mathbf{X},\alpha)}{p(\mathbf{X},\alpha)} = \frac{p(\mathbf{X}\mid\theta,\alpha)p(\theta,\alpha)}{p(\mathbf{X}\mid\alpha)p(\alpha)} = \frac{p(\mathbf{X} \mid \theta,\alpha) p(\theta \mid \alpha)}{p(\mathbf{X} \mid \alpha)} \propto p(\mathbf{X} \mid \theta,\alpha) p(\theta \mid \alpha) }$.
• 后验概率是考虑到观测数据后参数的分布情况。这是根据贝叶斯定律得到的，也是贝叶斯推断的核心：

$\displaystyle{ p(\theta \mid \mathbf{X},\alpha) = \frac{p(\theta,\mathbf{X},\alpha)}{p(\mathbf{X},\alpha)} = \frac{p(\mathbf{X}\mid\theta,\alpha)p(\theta,\alpha)}{p(\mathbf{X}\mid\alpha)p(\alpha)} = \frac{p(\mathbf{X} \mid \theta,\alpha) p(\theta \mid \alpha)}{p(\mathbf{X} \mid \alpha)} \propto p(\mathbf{X} \mid \theta,\alpha) p(\theta \mid \alpha) }$.

• In practice, for almost all complex Bayesian models used in machine learning, the posterior distribution $\displaystyle{ p(\theta \mid \mathbf{X},\alpha) }$ is not obtained in a closed form distribution, mainly because the parameter space for $\displaystyle{ \theta }$ can be very high, or the Bayesian model retains certain hierarchical structure formulated from the observations $\displaystyle{ \mathbf{X} }$ and parameter $\displaystyle{ \theta }$. In such situations, we need to resort to approximation techniques.
• 在实践中，对于几乎所有用于机器学习的复杂贝叶斯模型，后验概率$\displaystyle{ p(\theta \mid \mathbf{X},\alpha) }$都不是从有解析表达式的分布中获得的。这主要是因为 $\displaystyle{ \theta }$的参数空间维度可以非常高，又或者贝叶斯模型保留了某些由观测$\displaystyle{ \mathbf{X} }$和参数$\displaystyle{ \theta }$构成的结构。在这种情况下，我们需要求助于近似求解法。

### 贝叶斯预测

$\displaystyle{ p(\tilde{x} \mid \mathbf{X},\alpha) = \int p(\tilde{x} \mid \theta) p(\theta \mid \mathbf{X},\alpha) \operatorname{d}\!\theta }$
$\displaystyle{ p(\tilde{x} \mid \alpha) = \int p(\tilde{x} \mid \theta) p(\theta \mid \alpha) \operatorname{d}\!\theta }$
• 后验预测分布是一个新的数据点在后验分布上边缘化了以后的结果:

$\displaystyle{ p(\tilde{x} \mid \mathbf{X},\alpha) = \int p(\tilde{x} \mid \theta) p(\theta \mid \mathbf{X},\alpha) \operatorname{d}\!\theta }$

• 先验预测分布是一个新数据点在先验分布上边缘化了以后的结果:

$\displaystyle{ p(\tilde{x} \mid \alpha) = \int p(\tilde{x} \mid \theta) p(\theta \mid \alpha) \operatorname{d}\!\theta }$

Bayesian theory calls for the use of the posterior predictive distribution to do predictive inference, i.e., to predict the distribution of a new, unobserved data point. That is, instead of a fixed point as a prediction, a distribution over possible points is returned. By comparison, prediction in frequentist statistics often involves finding an optimum point estimate of the parameter(s)—e.g., by maximum likelihood or maximum a posteriori estimation (MAP)—and then plugging this estimate into the formula for the distribution of a data point. This has the disadvantage that it does not account for any uncertainty in the value of the parameter, and hence will underestimate the variance of the predictive distribution.

(In some instances, frequentist statistics can work around this problem. For example, confidence intervals and prediction intervals in frequentist statistics when constructed from a normal distribution with unknown mean and variance are constructed using a Student's t-distribution. This correctly estimates the variance, due to the facts that (1) the average of normally distributed random variables is also normally distributed, and (2) the predictive distribution of a normally distributed data point with unknown mean and variance, using conjugate or uninformative priors, has a Student's t-distribution. In Bayesian statistics, however, the posterior predictive distribution can always be determined exactly—or at least to an arbitrary level of precision when numerical methods are used.

(在某些情况下，频率统计可以绕过这个问题。例如，当频率统计中的置信区间和预测区间由一个均值和方差未知的正态分布构造时，我们可以使用一个 学生 t 分布来构造它。这种方法可以正确地估计方差，因为(1)正态分布随机变量的平均值也是正态分布的，(2)一个具有未知均值和方差的正态分布数据点的预测分布，使用共轭或无信息的先验函数，具有一个学生 t 分布。在贝叶斯统计，当使用数值方法时，后验预测分布总是可以精确地确定---- 或者至少可以达到任意精度。

Both types of predictive distributions have the form of a compound probability distribution (as does the marginal likelihood). In fact, if the prior distribution is a conjugate prior, such that the prior and posterior distributions come from the same family, it can be seen that both prior and posterior predictive distributions also come from the same family of compound distributions. The only difference is that the posterior predictive distribution uses the updated values of the hyperparameters (applying the Bayesian update rules given in the conjugate prior article), while the prior predictive distribution uses the values of the hyperparameters that appear in the prior distribution.

## 互斥的概率推断

If evidence is simultaneously used to update belief over a set of exclusive and exhaustive propositions, Bayesian inference may be thought of as acting on this belief distribution as a whole.

### 一般公式

Diagram illustrating event space $\displaystyle{ \Omega }$ in general formulation of Bayesian inference. Although this diagram shows discrete models and events, the continuous case may be visualized similarly using probability densities.

Suppose a process is generating independent and identically distributed events $\displaystyle{ E_n, \,\, n=1,2,3,\ldots }$, but the probability distribution is unknown. Let the event space $\displaystyle{ \Omega }$ represent the current state of belief for this process. Each model is represented by event $\displaystyle{ M_m }$. The conditional probabilities $\displaystyle{ P(E_n \mid M_m) }$ are specified to define the models. $\displaystyle{ P(M_m) }$ is the degree of belief in $\displaystyle{ M_m }$. Before the first inference step, $\displaystyle{ \{P(M_m)\} }$ is a set of initial prior probabilities. These must sum to 1, but are otherwise arbitrary.

Suppose that the process is observed to generate $\displaystyle{ \textstyle E \in \{E_n\} }$. For each $\displaystyle{ M \in \{M_m\} }$, the prior $\displaystyle{ P(M) }$ is updated to the posterior $\displaystyle{ P(M \mid E) }$. From Bayes' theorem:

$\displaystyle{ P(M \mid E) = \frac{P(E \mid M)}{\sum_m {P(E \mid M_m) P(M_m)}} \cdot P(M) }$

Upon observation of further evidence, this procedure may be repeated.

Venn diagram for the fundamental sets frequently used in Bayesian inference and computations 

### 多重观测

For a sequence of independent and identically distributed observations $\displaystyle{ \mathbf{E} = (e_1, \dots, e_n) }$, it can be shown by induction that repeated application of the above is equivalent to

$\displaystyle{ P(M \mid \mathbf{E}) = \frac{P(\mathbf{E} \mid M)}{\sum_m {P(\mathbf{E} \mid M_m) P(M_m)}} \cdot P(M) }$

$\displaystyle{ P(M \mid \mathbf{E}) = \frac{P(\mathbf{E} \mid M)}{\sum_m {P(\mathbf{E} \mid M_m) P(M_m)}} \cdot P(M) }$

Where

$\displaystyle{ P(\mathbf{E} \mid M) = \prod_k{P(e_k \mid M)}. }$

### 参数公式

By parameterizing the space of models, the belief in all models may be updated in a single step. The distribution of belief over the model space may then be thought of as a distribution of belief over the parameter space. The distributions in this section are expressed as continuous, represented by probability densities, as this is the usual situation. The technique is however equally applicable to discrete distributions.

Let the vector $\displaystyle{ \mathbf{\theta} }$ span the parameter space. Let the initial prior distribution over $\displaystyle{ \mathbf{\theta} }$ be $\displaystyle{ p(\mathbf{\theta} \mid \mathbf{\alpha}) }$, where $\displaystyle{ \mathbf{\alpha} }$ is a set of parameters to the prior itself, or hyperparameters. Let $\displaystyle{ \mathbf{E} = (e_1, \dots, e_n) }$ be a sequence of independent and identically distributed event observations, where all $\displaystyle{ e_i }$ are distributed as $\displaystyle{ p(e \mid \mathbf{\theta}) }$ for some $\displaystyle{ \mathbf{\theta} }$. Bayes' theorem is applied to find the posterior distribution over $\displaystyle{ \mathbf{\theta} }$:

\displaystyle{ \begin{align} p(\mathbf{\theta} \mid \mathbf{E},\mathbf{\alpha}) &= \frac{p(\mathbf{E} \mid \mathbf{\theta},\mathbf{\alpha})}{p(\mathbf{E} \mid \mathbf{\alpha})} \cdot p(\mathbf{\theta}\mid\mathbf{\alpha}) \\ &= \frac{p(\mathbf{E} \mid \mathbf{\theta},\mathbf{\alpha})}{\int p(\mathbf{E}|\mathbf{\theta},\mathbf{\alpha}) p(\mathbf{\theta} \mid \mathbf{\alpha}) \, d\mathbf{\theta}} \cdot p(\mathbf{\theta} \mid \mathbf{\alpha}) \end{align} }

Where

$\displaystyle{ p(\mathbf{E} \mid \mathbf{\theta},\mathbf{\alpha}) = \prod_k p(e_k \mid \mathbf{\theta}) }$

## 数学性质

### 解释因子

$\displaystyle{ \textstyle \frac{P(E \mid M)}{P(E)} \gt 1 \Rightarrow \textstyle P(E \mid M) \gt P(E) }$. That is, if the model were true, the evidence would be more likely than is predicted by the current state of belief. The reverse applies for a decrease in belief. If the belief does not change, $\displaystyle{ \textstyle \frac{P(E \mid M)}{P(E)} = 1 \Rightarrow \textstyle P(E \mid M) = P(E) }$. That is, the evidence is independent of the model. If the model were true, the evidence would be exactly as likely as predicted by the current state of belief.

$\displaystyle{ \textstyle \frac{P(E \mid M)}{P(E)} \gt 1 \Rightarrow \textstyle P(E \mid M) \gt P(E) }$ 。也就是说，如果这个模型是正确的，那么证据发生的概率将比目前的信仰状态所预测的更有可能。相反的情况适用于信念的减少。如果信念没有改变，$\displaystyle{ \textstyle \frac{P(E \mid M)}{P(E)} = 1 \Rightarrow \textstyle P(E \mid M) = P(E) }$。也就是说，证据是独立于模型的。如果这个模型是正确的，那么证据发生的可能性就和当前的信仰状态所预测的一样。

### Cromwell法则

If $\displaystyle{ P(M)=0 }$ then $\displaystyle{ P(M \mid E)=0 }$. If $\displaystyle{ P(M)=1 }$, then $\displaystyle{ P(M|E)=1 }$. This can be interpreted to mean that hard convictions are insensitive to counter-evidence.

The former follows directly from Bayes' theorem. The latter can be derived by applying the first rule to the event "not $\displaystyle{ M }$" in place of "$\displaystyle{ M }$", yielding "if $\displaystyle{ 1 - P(M)=0 }$, then $\displaystyle{ 1 - P(M \mid E)=0 }$", from which the result immediately follows.

### 后验的渐近性

Consider the behaviour of a belief distribution as it is updated a large number of times with independent and identically distributed trials. For sufficiently nice prior probabilities, the Bernstein-von Mises theorem gives that in the limit of infinite trials, the posterior converges to a Gaussian distribution independent of the initial prior under some conditions firstly outlined and rigorously proven by Joseph L. Doob in 1948, namely if the random variable in consideration has a finite probability space. The more general results were obtained later by the statistician David A. Freedman who published in two seminal research papers in 1963  and 1965  when and under what circumstances the asymptotic behaviour of posterior is guaranteed. His 1963 paper treats, like Doob (1949), the finite case and comes to a satisfactory conclusion. However, if the random variable has an infinite but countable probability space (i.e., corresponding to a die with infinite many faces) the 1965 paper demonstrates that for a dense subset of priors the Bernstein-von Mises theorem is not applicable. In this case there is almost surely no asymptotic convergence. Later in the 1980s and 1990s Freedman and Persi Diaconis continued to work on the case of infinite countable probability spaces. To summarise, there may be insufficient trials to suppress the effects of the initial choice, and especially for large (but finite) systems the convergence might be very slow.

### 共轭先验

In parameterized form, the prior distribution is often assumed to come from a family of distributions called conjugate priors. The usefulness of a conjugate prior is that the corresponding posterior distribution will be in the same family, and the calculation may be expressed in closed form.

### 估计参数和预测

It is often desired to use a posterior distribution to estimate a parameter or variable. Several methods of Bayesian estimation select measurements of central tendency from the posterior distribution.

For one-dimensional problems, a unique median exists for practical continuous problems. The posterior median is attractive as a robust estimator.

If there exists a finite mean for the posterior distribution, then the posterior mean is a method of estimation.

$\displaystyle{ \tilde \theta = \operatorname{E}[\theta] = \int \theta \, p(\theta \mid \mathbf{X},\alpha) \, d\theta }$

$\displaystyle{ \tilde \theta = \operatorname{E}[\theta] = \int \theta \, p(\theta \mid \mathbf{X},\alpha) \, d\theta }$

Taking a value with the greatest probability defines maximum a posteriori (MAP) estimates:

$\displaystyle{ \{ \theta_{\text{MAP}}\} \subset \arg \max_\theta p(\theta \mid \mathbf{X},\alpha) . }$

There are examples where no maximum is attained, in which case the set of MAP estimates is empty.

There are other methods of estimation that minimize the posterior risk (expected-posterior loss) with respect to a loss function, and these are of interest to statistical decision theory using the sampling distribution ("frequentist statistics").

The posterior predictive distribution of a new observation $\displaystyle{ \tilde{x} }$ (that is independent of previous observations) is determined by

$\displaystyle{ p(\tilde{x}|\mathbf{X},\alpha) = \int p(\tilde{x},\theta \mid \mathbf{X},\alpha) \, d\theta = \int p(\tilde{x} \mid \theta) p(\theta \mid \mathbf{X},\alpha) \, d\theta . }$

## 实例

### 一个假设的概率

Contingency table

H1
#2
H2

Total
Plain, E 30 20 50
Choc, ¬E 10 20 30
Total 40 40 80
P (H1|E) = 30 / 50 = 0.6

Suppose there are two full bowls of cookies. Bowl #1 has 10 chocolate chip and 30 plain cookies, while bowl #2 has 20 of each. Our friend Fred picks a bowl at random, and then picks a cookie at random. We may assume there is no reason to believe Fred treats one bowl differently from another, likewise for the cookies. The cookie turns out to be a plain one. How probable is it that Fred picked it out of bowl #1?

Contingency table
#1H1 #2H2 Total
Plain, E 30 20 50
Choc, ¬E 10 20 30
Total 40 40 80
P (H1|E) = 30 / 50 = 0.6

Intuitively, it seems clear that the answer should be more than a half, since there are more plain cookies in bowl #1. The precise answer is given by Bayes' theorem. Let $\displaystyle{ H_1 }$ correspond to bowl #1, and $\displaystyle{ H_2 }$ to bowl #2. It is given that the bowls are identical from Fred's point of view, thus $\displaystyle{ P(H_1)=P(H_2) }$, and the two must add up to 1, so both are equal to 0.5. The event $\displaystyle{ E }$ is the observation of a plain cookie. From the contents of the bowls, we know that $\displaystyle{ P(E \mid H_1) = 30/40 = 0.75 }$ and $\displaystyle{ P(E \mid H_2) = 20/40 = 0.5. }$ Bayes' formula then yields

\displaystyle{ \begin{align} P(H_1 \mid E) &= \frac{P(E \mid H_1)\,P(H_1)}{P(E \mid H_1)\,P(H_1)\;+\;P(E \mid H_2)\,P(H_2)} \\ \\ \ & = \frac{0.75 \times 0.5}{0.75 \times 0.5 + 0.5 \times 0.5} \\ \\ \ & = 0.6 \end{align} }

\displaystyle{ \begin{align} P(H_1 \mid E) &= \frac{P(E \mid H_1)\,P(H_1)}{P(E \mid H_1)\,P(H_1)\;+\;P(E \mid H_2)\,P(H_2)} \\ \\ \ & = \frac{0.75 \times 0.5}{0.75 \times 0.5 + 0.5 \times 0.5} \\ \\ \ & = 0.6 \end{align} }

Before we observed the cookie, the probability we assigned for Fred having chosen bowl #1 was the prior probability, $\displaystyle{ P(H_1) }$, which was 0.5. After observing the cookie, we must revise the probability to $\displaystyle{ P(H_1 \mid E) }$, which is 0.6.

### 做出预测

An archaeologist is working at a site thought to be from the medieval period, between the 11th century to the 16th century. However, it is uncertain exactly when in this period the site was inhabited. Fragments of pottery are found, some of which are glazed and some of which are decorated. It is expected that if the site were inhabited during the early medieval period, then 1% of the pottery would be glazed and 50% of its area decorated, whereas if it had been inhabited in the late medieval period then 81% would be glazed and 5% of its area decorated. How confident can the archaeologist be in the date of inhabitation as fragments are unearthed?

The degree of belief in the continuous variable $\displaystyle{ C }$ (century) is to be calculated, with the discrete set of events $\displaystyle{ \{GD,G \bar D, \bar G D, \bar G \bar D\} }$ as evidence. Assuming linear variation of glaze and decoration with time, and that these variables are independent,

$\displaystyle{ P(E=GD \mid C=c) = (0.01 + \frac{0.81-0.01}{16-11}(c-11))(0.5 - \frac{0.5-0.05}{16-11}(c-11)) }$
$\displaystyle{ P(E=G \bar D \mid C=c) = (0.01 + \frac{0.81-0.01}{16-11}(c-11))(0.5 + \frac{0.5-0.05}{16-11}(c-11)) }$
$\displaystyle{ P(E=\bar G D \mid C=c) = ((1-0.01) - \frac{0.81-0.01}{16-11}(c-11))(0.5 - \frac{0.5-0.05}{16-11}(c-11)) }$
$\displaystyle{ P(E=\bar G \bar D \mid C=c) = ((1-0.01) - \frac{0.81-0.01}{16-11}(c-11))(0.5 + \frac{0.5-0.05}{16-11}(c-11)) }$

Assume a uniform prior of $\displaystyle{ \textstyle f_C(c) = 0.2 }$, and that trials are independent and identically distributed. When a new fragment of type $\displaystyle{ e }$ is discovered, Bayes' theorem is applied to update the degree of belief for each $\displaystyle{ c }$:

$\displaystyle{ f_C(c \mid E=e) = \frac{P(E=e \mid C=c)}{P(E=e)}f_C(c) = \frac{P(E=e \mid C=c)}{\int_{11}^{16}{P(E=e \mid C=c)f_C(c)dc}}f_C(c) }$

A computer simulation of the changing belief as 50 fragments are unearthed is shown on the graph. In the simulation, the site was inhabited around 1420, or $\displaystyle{ c=15.2 }$. By calculating the area under the relevant portion of the graph for 50 trials, the archaeologist can say that there is practically no chance the site was inhabited in the 11th and 12th centuries, about 1% chance that it was inhabited during the 13th century, 63% chance during the 14th century and 36% during the 15th century. The Bernstein-von Mises theorem asserts here the asymptotic convergence to the "true" distribution because the probability space corresponding to the discrete set of events $\displaystyle{ \{GD,G \bar D, \bar G D, \bar G \bar D\} }$ is finite (see above section on asymptotic behaviour of the posterior).

## 在频率统计学和决策理论中

A decision-theoretic justification of the use of Bayesian inference was given by Abraham Wald, who proved that every unique Bayesian procedure is admissible. Conversely, every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures.

Abraham Wald,给出了一个决策论的理由来证明使用贝叶斯推断的合理性，他证明了每一个独特的贝叶斯程序都是可以采纳的。相反，每个可采纳的统计过程要么是贝叶斯过程，要么是贝叶斯过程的限制。

Wald characterized admissible procedures as Bayesian procedures (and limits of Bayesian procedures), making the Bayesian formalism a central technique in such areas of frequentist inference as parameter estimation, hypothesis testing, and computing confidence intervals. For example:

• "Under some conditions, all admissible procedures are either Bayes procedures or limits of Bayes procedures (in various senses). These remarkable results, at least in their original form, are due essentially to Wald. They are useful because the property of being Bayes is easier to analyze than admissibility."
• "In decision theory, a quite general method for proving admissibility consists in exhibiting a procedure as a unique Bayes solution."
• "In the first chapters of this work, prior distributions with finite support and the corresponding Bayes procedures were used to establish some of the main theorems relating to the comparison of experiments. Bayes procedures with respect to more general prior distributions have played a very important role in the development of statistics, including its asymptotic theory." "There are many problems where a glance at posterior distributions, for suitable priors, yields immediately interesting information. Also, this technique can hardly be avoided in sequential analysis."
• "A useful fact is that any Bayes decision rule obtained by taking a proper prior over the whole parameter space must be admissible"
• "An important area of investigation in the development of admissibility ideas has been that of conventional sampling-theory procedures, and many interesting results have been obtained."

Wald 将可容许的过程描述为贝叶斯 过程(以及贝叶斯过程的限制) ，使 贝叶斯形式化成为频率论推断领域的核心技术，如参数估计、假设检验和计算置信区间。例如:

• “在某些条件下，所有可接受的过程要么是贝叶斯过程，要么是贝叶斯过程的限制(在各种意义上)。这些显著的成果，至少在其原始形式，主要是Wald的功劳 。它们之所以有用，是因为在决策论中贝叶斯的性质比可容许性更容易分析。
• 证明可采性的一个相当普遍的方法是展示其为一个唯一的贝叶斯解
• “在这项工作的第一章中，有限支持的先验分布和相应的贝叶斯过程被用来建立一些与实验相关的主要定理。与更普遍的先验分布相关的贝叶斯过程在统计学的发展中发挥了非常重要的作用，包括在渐近理论分布中。毕竟只要看一眼后验分布，找到合适的先验，就能立即得到有趣的信息。此外，这种技术在顺序分析中也难以避免被经常使用。”
• “一个有用的事实是，通过对整个参数空间取得适当先验而获得的任何贝叶斯决定规则都必须是可接受的”
• “在可接受性思想的发展过程中，一个重要的领域是传统的抽样理论，其中已获得许多有趣的结果”

### 模型的选择

Bayesian methodology also plays a role in model selection where the aim is to select one model from a set of competing models that represents most closely the underlying process that generated the observed data. In Bayesian model comparison, the model with the highest posterior probability given the data is selected. The posterior probability of a model depends on the evidence, or marginal likelihood, which reflects the probability that the data is generated by the model, and on the prior belief of the model. When two competing models are a priori considered to be equiprobable, the ratio of their posterior probabilities corresponds to the Bayes factor. Since Bayesian model comparison is aimed on selecting the model with the highest posterior probability, this methodology is also referred to as the maximum a posteriori (MAP) selection rule  or the MAP probability rule.

## 概率编程

While conceptually simple, Bayesian methods can be mathematically and numerically challenging. Probabilistic programming languages (PPLs) implement functions to easily build Bayesian models together with efficient automatic inference methods. This helps separate the model building from the inference, allowing practitioners to focus on their specific problems and leaving PPLs to handle the computational details for them.

## 应用程序

### 计算机应用

Bayesian inference has applications in artificial intelligence and expert systems. Bayesian inference techniques have been a fundamental part of computerized pattern recognition techniques since the late 1950s. There is also an ever-growing connection between Bayesian methods and simulation-based Monte Carlo techniques since complex models cannot be processed in closed form by a Bayesian analysis, while a graphical model structure may allow for efficient simulation algorithms like the Gibbs sampling and other Metropolis–Hastings algorithm schemes. Recently模板:When Bayesian inference has gained popularity among the phylogenetics community for these reasons; a number of applications allow many demographic and evolutionary parameters to be estimated simultaneously.

As applied to statistical classification, Bayesian inference has been used to develop algorithms for identifying e-mail spam. Applications which make use of Bayesian inference for spam filtering include CRM114, DSPAM, Bogofilter, SpamAssassin, SpamBayes, Mozilla, XEAMS, and others. Spam classification is treated in more detail in the article on the naïve Bayes classifier.

Solomonoff's Inductive inference is the theory of prediction based on observations; for example, predicting the next symbol based upon a given series of symbols. The only assumption is that the environment follows some unknown but computable probability distribution. It is a formal inductive framework that combines two well-studied principles of inductive inference: Bayesian statistics and Occam’s Razor.模板:Rs inline Solomonoff's universal prior probability of any prefix p of a computable sequence x is the sum of the probabilities of all programs (for a universal computer) that compute something starting with p. Given some p and any computable but unknown probability distribution from which x is sampled, the universal prior and Bayes' theorem can be used to predict the yet unseen parts of x in optimal fashion.

Solomonoff的归纳推理是基于观察的预测理论; 例如，基于给定的一系列符号预测下一个符号。唯一的假设是，环境遵循一些未知但可计算的概率分布。这是一个正式的归纳框架，它结合了两个经过科学家深入研究的归纳推理原则: 贝叶斯统计和 Occam 的剃刀（越简单的模型越好）。Solomonoff's 的可计算序列 x 的任意前缀 p 的通用先验概率是所有以 p 开始计算的程序的概率之和。给定一些 p 和任何可计算但未知的概率分布，从中取样 x，通用先验和贝叶斯定理可以用最优方式预测 x 中尚未看见的部分。

### 生物信息学和医疗保健应用

Bayesian inference has been applied in different Bioinformatics applications, including differential gene expression analysis. Bayesian inference is also used in a general cancer risk model, called CIRI (Continuous Individualized Risk Index), where serial measurements are incorporated to update a Bayesian model which is primarily built from prior knowledge.

### 在法庭上

Bayesian inference can be used by jurors to coherently accumulate the evidence for and against a defendant, and to see whether, in totality, it meets their personal threshold for 'beyond a reasonable doubt'. Bayes' theorem is applied successively to all evidence presented, with the posterior from one stage becoming the prior for the next. The benefit of a Bayesian approach is that it gives the juror an unbiased, rational mechanism for combining evidence. It may be appropriate to explain Bayes' theorem to jurors in odds form, as betting odds are more widely understood than probabilities. Alternatively, a logarithmic approach, replacing multiplication with addition, might be easier for a jury to handle.

If the existence of the crime is not in doubt, only the identity of the culprit, it has been suggested that the prior should be uniform over the qualifying population. For example, if 1,000 people could have committed the crime, the prior probability of guilt would be 1/1000.

The use of Bayes' theorem by jurors is controversial. In the United Kingdom, a defence expert witness explained Bayes' theorem to the jury in R v Adams. The jury convicted, but the case went to appeal on the basis that no means of accumulating evidence had been provided for jurors who did not wish to use Bayes' theorem. The Court of Appeal upheld the conviction, but it also gave the opinion that "To introduce Bayes' Theorem, or any similar method, into a criminal trial plunges the jury into inappropriate and unnecessary realms of theory and complexity, deflecting them from their proper task."

Gardner-Medwin argues that the criterion on which a verdict in a criminal trial should be based is not the probability of guilt, but rather the probability of the evidence, given that the defendant is innocent (akin to a frequentist p-value). He argues that if the posterior probability of guilt is to be computed by Bayes' theorem, the prior probability of guilt must be known. This will depend on the incidence of the crime, which is an unusual piece of evidence to consider in a criminal trial. Consider the following three propositions:

Gardner-Medwin 认为，在刑事审判中作出判决所依据的标准不是有罪的可能性，而是证据的可能性，因为被告是无辜的(类似于频率 p 值)。他认为，如果要用贝叶斯定理计算罪行的后验概率，那么罪行的先验概率必须是已知的。这将取决于犯罪的发生率，在刑事审判中，这是一个不寻常的证据。考虑以下三个主张:

A The known facts and testimony could have arisen if the defendant is guilty
B The known facts and testimony could have arisen if the defendant is innocent
C The defendant is guilty.

A 如果被告有罪，就可能产生的已知的事实和证词;

B如果被告无罪，就可能产生的已知的事实和证词;

C被告有罪。

Gardner-Medwin argues that the jury should believe both A and not-B in order to convict. A and not-B implies the truth of C, but the reverse is not true. It is possible that B and C are both true, but in this case he argues that a jury should acquit, even though they know that they will be letting some guilty people go free. See also Lindley's paradox.

Gardner-Medwin argues that the jury should believe both A and not-B in order to convict. A and not-B implies the truth of C, but the reverse is not true. It is possible that B and C are both true, but in this case he argues that a jury should acquit, even though they know that they will be letting some guilty people go free. See also Lindley's paradox.

Gardner-Medwin认为，陪审团为了定罪，应该同时相信 A 和 B。A 和 not-B 意味着 cC的真理，但反之则不然。有可能 B 和 cC都是正确的，但是在这个案例中，他认为陪审团应该宣判无罪，即使他们知道他们将会释放一些有罪的人。参见Lindley悖论。

### 贝叶斯认识论

Bayesian epistemology is a movement that advocates for Bayesian inference as a means of justifying the rules of inductive logic.

Karl Popper and David Miller have rejected the idea of Bayesian rationalism, i.e. using Bayes rule to make epistemological inferences: It is prone to the same vicious circle as any other justificationist epistemology, because it presupposes what it attempts to justify. According to this view, a rational interpretation of Bayesian inference would see it merely as a probabilistic version of falsification, rejecting the belief, commonly held by Bayesians, that high likelihood achieved by a series of Bayesian updates would prove the hypothesis beyond any reasonable doubt, or even with likelihood greater than 0.

Karl PopperDavid Miller 拒绝了贝叶斯理性主义的想法，即运用贝叶斯规则进行认识论推论: 它与其他任何正义主义认识论一样，容易陷入同样的恶性循环，因为它预先假定了它试图证明的东西。根据这种观点，对贝叶斯推断的理性解释将仅仅把它看作是一种曲解的概率版本，否定了贝叶斯学派普遍持有的观点，即通过一系列贝叶斯更新实现的高可能性将证明假设超越了任何合理怀疑，甚至可能性大于0。

### 其他

• 贝叶斯搜索理论是用来搜寻遗失物品的。
• 贝叶斯贝叶斯推断在系统发育学中的应用
• 贝叶斯工具用于甲基化分析
• 贝叶斯方法用于大脑功能研究，将大脑作为一种贝叶斯机制。
• 生态学研究贝叶斯推断
• 贝叶斯推断用于估计随机化学动力学模型的参数
• 货币或股票市场预测经济物理学贝叶斯推断
• 市场推广贝叶斯推断
• 动力学习贝叶斯推断

## 历史

The term Bayesian refers to Thomas Bayes (1702–1761), who proved that probabilistic limits could be placed on an unknown event. However, it was Pierre-Simon Laplace (1749–1827) who introduced (as Principle VI) what is now called Bayes' theorem and used it to address problems in celestial mechanics, medical statistics, reliability, and jurisprudence. Early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes). After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called frequentist statistics.

In the 20th century, the ideas of Laplace were further developed in two different directions, giving rise to objective and subjective currents in Bayesian practice. In the objective or "non-informative" current, the statistical analysis depends on only the model assumed, the data analyzed, and the method assigning the prior, which differs from one objective Bayesian practitioner to another. In the subjective or "informative" current, the specification of the prior depends on the belief (that is, propositions on which the analysis is prepared to act), which can summarize information from experts, previous studies, etc.

20世纪，Laplace的思想进一步向两个不同的方向发展，在贝叶斯实践中产生了客观和主观的潮流。在客观或“非信息”流派中，统计分析仅依赖于假设的模型、分析的数据和赋予先验的方法，这些对于一个客观贝叶斯实践者来说和对另一个客观贝叶斯实践者来说是不同的。在主观的或“信息性的”流派中，先验的说明取决于信念(即分析准备采取行动的命题) ，它可以来自专家的信息，以前的研究等。

In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods, which removed many of the computational problems, and an increasing interest in nonstandard, complex applications. Despite growth of Bayesian research, most undergraduate teaching is still based on frequentist statistics. Nonetheless, Bayesian methods are widely accepted and used, such as for example in the field of machine learning.

## 参考

### 引用

1. Hacking, Ian (December 1967). "Slightly More Realistic Personal Probability". Philosophy of Science. 34 (4): 316. doi:10.1086/288169. S2CID 14344339.
2. Hacking (1988, p. 124)模板:Full citation needed
3. "Bayes' Theorem (Stanford Encyclopedia of Philosophy)". Plato.stanford.edu. Retrieved 2014-01-05.
4. van Fraassen, B. (1989) Laws and Symmetry, Oxford University Press.
5. Lee, Se Yoon (2021). "Gibbs sampler and coordinate ascent variational inference: A set-theoretical review". Communications in Statistics - Theory and Methods: 1–21. arXiv:2008.01006. doi:10.1080/03610926.2021.1921214. S2CID 220935477.
6. Gelman, Andrew; Carlin, John B.; Stern, Hal S.; Dunson, David B.;Vehtari, Aki; Rubin, Donald B. (2013). Bayesian Data Analysis, Third Edition. Chapman and Hall/CRC. .
7. Lee, Se Yoon (2021). "Gibbs sampler and coordinate ascent variational inference: A set-theoretical review". Communications in Statistics - Theory and Methods: 1–21. arXiv:2008.01006. doi:10.1080/03610926.2021.1921214. S2CID 220935477.
8. Freedman, DA (1963). "On the asymptotic behavior of Bayes' estimates in the discrete case". The Annals of Mathematical Statistics. 34 (4): 1386–1403. doi:10.1214/aoms/1177703871. JSTOR 2238346.
9. Freedman, DA (1965). "On the asymptotic behavior of Bayes estimates in the discrete case II". The Annals of Mathematical Statistics. 36 (2): 454–456. doi:10.1214/aoms/1177700155. JSTOR 2238150.
10. Robins, James; Wasserman, Larry (2000). "Conditioning, likelihood, and coherence: A review of some foundational concepts". JASA. 95 (452): 1340–1346. doi:10.1080/01621459.2000.10474344. S2CID 120767108.
11. Sen, Pranab K.; Keating, J. P.; Mason, R. L. (1993). Pitman's measure of closeness: A comparison of statistical estimators. Philadelphia: SIAM.
12. Choudhuri, Nidhan; Ghosal, Subhashis; Roy, Anindya (2005-01-01). Bayesian Methods for Function Estimation. Bayesian Thinking. 25. pp. 373–414. doi:10.1016/s0169-7161(05)25013-7. ISBN 9780444515391.
13. "Maximum A Posteriori (MAP) Estimation". www.probabilitycourse.com (in English). Retrieved 2017-06-02.
14. Yu, Angela. "Introduction to Bayesian Decision Theory" (PDF). cogsci.ucsd.edu/. Archived from the original (PDF) on 2013-02-28.
15. Hitchcock, David. "Posterior Predictive Distribution Stat Slide" (PDF). stat.sc.edu.
16. Bickel & Doksum (2001, p. 32)
17. Kiefer, J.; Schwartz R. (1965). "Admissible Bayes Character of T2-, R2-, and Other Fully Invariant Tests for Multivariate Normal Problems". Annals of Mathematical Statistics. 36 (3): 747–770. doi:10.1214/aoms/1177700051.
18. Schwartz, R. (1969). "Invariant Proper Bayes Tests for Exponential Families". Annals of Mathematical Statistics. 40: 270–283. doi:10.1214/aoms/1177697822.
19. Hwang, J. T. & Casella, George (1982). "Minimax Confidence Sets for the Mean of a Multivariate Normal Distribution" (PDF). Annals of Statistics. 10 (3): 868–881. doi:10.1214/aos/1176345877.
20. Lehmann, Erich (1986). Testing Statistical Hypotheses (Second ed.).  (see p. 309 of Chapter 6.7 "Admissibility", and pp. 17–18 of Chapter 1.8 "Complete Classes"
21. Le Cam, Lucien (1986). Asymptotic Methods in Statistical Decision Theory. Springer-Verlag. ISBN 978-0-387-96307-5.  (From "Chapter 12 Posterior Distributions and Bayes Solutions", p. 324)
22. Cox, D. R.; Hinkley, D.V. (1974). Theoretical Statistics. Chapman and Hall. p. 432. ISBN 978-0-04-121537-3.
23. Cox, D. R.; Hinkley, D.V. (1974). Theoretical Statistics. Chapman and Hall. p. 433. ISBN 978-0-04-121537-3. )
24. Stoica, P.; Selen, Y. (2004). "A review of information criterion rules". IEEE Signal Processing Magazine. 21 (4): 36–47. doi:10.1109/MSP.2004.1311138. S2CID 17338979.
25. Fatermans, J.; Van Aert, S.; den Dekker, A.J. (2019). "The maximum a posteriori probability rule for atom column detection from HAADF STEM images". Ultramicroscopy. 201: 81–91. arXiv:1902.05809. doi:10.1016/j.ultramic.2019.02.003. PMID 30991277. S2CID 104419861.
26. Bessiere, P., Mazer, E., Ahuactzin, J. M., & Mekhnacha, K. (2013). Bayesian Programming (1 edition) Chapman and Hall/CRC.
27. Daniel Roy (2015). "Probabilistic Programming". probabilistic-programming.org. Archived from the original on 2016-01-10. Retrieved 2020-01-02.
28. Ghahramani, Z (2015). "Probabilistic machine learning and artificial intelligence". Nature. 521 (7553): 452–459. Bibcode:2015Natur.521..452G. doi:10.1038/nature14541. PMID 26017444. S2CID 216356.
29. Fienberg, Stephen E. (2006-03-01). "When did Bayesian inference become "Bayesian"?". Bayesian Analysis. 1 (1). doi:10.1214/06-BA101.
30. Jim Albert (2009). Bayesian Computation with R, Second edition. New York, Dordrecht, etc.: Springer. ISBN 978-0-387-92297-3.
31. Rathmanner, Samuel; Hutter, Marcus; Ormerod, Thomas C (2011). "A Philosophical Treatise of Universal Induction". Entropy. 13 (6): 1076–1136. arXiv:1105.5721. Bibcode:2011Entrp..13.1076R. doi:10.3390/e13061076. S2CID 2499910.
32. Hutter, Marcus; He, Yang-Hui; Ormerod, Thomas C (2007). "On Universal Prediction and Bayesian Confirmation". Theoretical Computer Science. 384 (2007): 33–48. arXiv:0709.1516. Bibcode:2007arXiv0709.1516H. doi:10.1016/j.tcs.2007.05.016. S2CID 1500830.
33. Gács, Peter; Vitányi, Paul M. B. (2 December 2010). "Raymond J. Solomonoff 1926-2009". CiteSeerX. CiteSeerX 10.1.1.186.8268. {{cite journal}}: Cite journal requires |journal= (help)
34. Robinson, Mark D & McCarthy, Davis J & Smyth, Gordon K edgeR: a Bioconductor package for differential expression analysis of digital gene expression data, Bioinformatics.
35. "CIRI". ciri.stanford.edu. Retrieved 2019-08-11.
36. Kurtz, David M.; Esfahani, Mohammad S.; Scherer, Florian; Soo, Joanne; Jin, Michael C.; Liu, Chih Long; Newman, Aaron M.; Dührsen, Ulrich; Hüttmann, Andreas (2019-07-25). "Dynamic Risk Profiling Using Serial Tumor Biomarkers for Personalized Outcome Prediction". Cell. 178 (3): 699–713.e19. doi:10.1016/j.cell.2019.06.011. ISSN 1097-4172. PMC 7380118. PMID 31280963.
37. Dawid, A. P. and Mortera, J. (1996) "Coherent Analysis of Forensic Identification Evidence". Journal of the Royal Statistical Society, Series B, 58, 425–443.
38. Foreman, L. A.; Smith, A. F. M., and Evett, I. W. (1997). "Bayesian analysis of deoxyribonucleic acid profiling data in forensic identification applications (with discussion)". Journal of the Royal Statistical Society, Series A, 160, 429–469.
39. Robertson, B. and Vignaux, G. A. (1995) Interpreting Evidence: Evaluating Forensic Science in the Courtroom. John Wiley and Sons. Chichester.
40. Dawid, A. P. (2001) Bayes' Theorem and Weighing Evidence by Juries -{zh-cn:互联网档案馆; zh-tw:網際網路檔案館; zh-hk:互聯網檔案館;}-存檔，存档日期2015-07-01.
41. Gardner-Medwin, A. (2005) "What Probability Should the Jury Address?". Significance, 2 (1), March 2005
42. Miller, David (1994). Critical Rationalism. Chicago: Open Court. ISBN 978-0-8126-9197-9.
43. Howson & Urbach (2005), Jaynes (2003)
44. Cai, X.Q.; Wu, X.Y.; Zhou, X. (2009). "Stochastic scheduling subject to breakdown-repeat breakdowns with incomplete information". Operations Research. 57 (5): 1236–1249. doi:10.1287/opre.1080.0660.
45. Ogle, Kiona; Tucker, Colin; Cable, Jessica M. (2014-01-01). "Beyond simple linear mixing models: process-based isotope partitioning of ecological processes". Ecological Applications (in English). 24 (1): 181–195. doi:10.1890/1051-0761-24.1.181. ISSN 1939-5582. PMID 24640543.
46. Evaristo, Jaivime; McDonnell, Jeffrey J.; Scholl, Martha A.; Bruijnzeel, L. Adrian; Chun, Kwok P. (2016-01-01). "Insights into plant water uptake from xylem-water isotope measurements in two tropical catchments with contrasting moisture conditions". Hydrological Processes (in English). 30 (18): 3210–3227. Bibcode:2016HyPr...30.3210E. doi:10.1002/hyp.10841. ISSN 1099-1085.
47. Gupta, Ankur; Rawlings, James B. (April 2014). "Comparison of Parameter Estimation Methods in Stochastic Chemical Kinetic Models: Examples in Systems Biology". AIChE Journal. 60 (4): 1253–1268. doi:10.1002/aic.14409. ISSN 0001-1541. PMC 4946376. PMID 27429455.
48. Fornalski, K.W. (2016). "The Tadpole Bayesian Model for Detecting Trend Changes in Financial Quotations" (PDF). R&R Journal of Statistics and Mathematical Sciences. 2 (1): 117–122.
49. Schütz, N.; Holschneider, M. (2011). "Detection of trend changes in time series using Bayesian inference". Physical Review E. 84 (2): 021120. arXiv:1104.3448. Bibcode:2011PhRvE..84b1120S. doi:10.1103/PhysRevE.84.021120. PMID 21928962. S2CID 11460968.
50. Reference Needed
51. Stigler, Stephen M. (1986). "Chapter 3". The History of Statistics. Harvard University Press. ISBN 9780674403406.
52. Fienberg, Stephen E. (2006). "When did Bayesian Inference Become 'Bayesian'?". Bayesian Analysis. 1 (1): 1–40 [p. 5]. doi:10.1214/06-ba101.
53. Bernardo, José-Miguel (2005). "Reference analysis". Handbook of statistics. 25. pp. 17–90.
54. Wolpert, R. L. (2004). "A Conversation with James O. Berger". Statistical Science. 19 (1): 205–218. CiteSeerX 10.1.1.71.6112. doi:10.1214/088342304000000053. MR 2082155.
55. Bernardo, José M. (2006). "A Bayesian mathematical statistics primer" (PDF). Icots-7.
56. Bishop, C. M. (2007). Pattern Recognition and Machine Learning. New York: Springer. ISBN 978-0387310732.

### 资源

• Aster, Richard; Borchers, Brian, and Thurber, Clifford (2012). Parameter Estimation and Inverse Problems, Second Edition, Elsevier. ,
• Box, G. E. P. and Tiao, G. C. (1973) Bayesian Inference in Statistical Analysis, Wiley,
• Jaynes E. T. (2003) Probability Theory: The Logic of Science, CUP. (Link to Fragmentary Edition of March 1996).

• Aster，Richard; Borchers，Brian，and Thurber，Clifford (2012).参数估计和反问题，第二版，爱思唯尔。，
• Box，G.E.p. and Tiao，G.c. (1973)贝叶斯推断统计分析，Wiley，
• Jaynes E.t. (2003)概率论: 科学的逻辑，CUP。(链接到1996年3月的零碎版)。

# 进一步阅读

• For a full report on the history of Bayesian statistics and the debates with frequentists approaches, read Vallverdu, Jordi (2016). Bayesians Versus Frequentists A Philosophical Debate on Statistical Reasoning. New York: Springer. ISBN 978-3-662-48638-2.
• For a full report on the history of Bayesian statistics and the debates with frequentists approaches, read

### 初级

The following books are listed in ascending order of probabilistic sophistication: