Notes: Elements of Causal Inference (Peters, Janzing, Schölkopf)

---

Page 35

p. 35

This seems a conceptually novel way of looking at equivalence of partial pooling in causal models. Might be exact equivalence though.

Though it is worth noting that the independence seems to be a bigger term than in statistics. Independence of cause and mechanism - mechanism being

$$p(t|a)$$

and cause being

$$p(a).$$

If we start changing the cause distribution then the t → a causal model does not keep the mechanism intact.

The generative process or mechanism generating the effect from the cause is indpeendent of the generative process or mechanism that is generating the cause.

Three differnet visions of independence.

1. Mechanistic - the light, the position of a chair and angle you are watching the chair are indpendent and there exists the assumption of generic viewpoint. You can change the light, position of the chair or perspective without being surprised.

2. Information theoretic independence - learning about the structure of the conditional does not in any way inform you about the structure of the prior.

3. Independence of noises - where is noise generated from with regards to the generative process. Not sure how this generalises last - it was mentioned that this can model or homogeneous processes as well. To break independence we can look at mechanisms or subprocesses $f_k$ such that the process is selected randomly when rv $X_t=k$. When we for example know that $X_{t+1}=X_t+1$ then processes $f_k$ at $\tau =t$ or $\tau =t+1$ are not independent.

Page 56

p. 56

Problem 3.5.

p36

Problem 3.6. Recall Eq (3.4):

$$P_C^{\mathcal{C};do(E:=2)}=\mathcal{N}(0,1)\ne P_{C|E=2}^{\mathcal{C}},$$

where $C:=N_C,E:=4C+N_E$ and $N_C,N_E\sim \mathcal{N}(0,1)$.

Show that $P_{C|E=2}^{\mathcal{C}}$ is Gaussian:

$$C|E=2\sim \mathcal{N}(8/17,{\sigma }^2=1/17)$$

Per Bayes’ theorem $p(C|E=2)=p(E=2|C)p(C)/p(E=2)$, and our goal is to massage the right-hand side into the standard Gaussian form

$$p(C|E=2)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(C-\mu {)}^2}{2{\sigma }^2}\right),\mu =8/17,{\sigma }^2=1/17.$$

(0) First let us remind ourselves how the change of variables work.

When we have a continuous measure $\mu \ll \mathrm{Leb}$ then we can write $\int d\mu =\int p(x)dx$ where $p(x)$ is the density function with respect to the Lebesgue measure. We can write $F(x):=P(X<x)$. Now let us look at some transformation $g:X\mapsto g(X)$ and we care about $G(x):=P(g(X)<x)$.

Let $(S,\Sigma )$ be measurable and $h:S\to \mathbb{R}$ function and $\mathcal{B}:=\mathcal{B}(\mathbb{R})$. We say that $h$ is $\Sigma -$measurable or $\Sigma \setminus \mathcal{B}$-measurable if every Borel set has it’s original in $\Sigma$.

Now let us look at measure spaces $(S,\Sigma ,\mu )$ and $(S^{\prime },{\Sigma }^{\prime })$. Let us define a generalisation for $\mu h^{-1}$.

Let us have a function $T:S\to S^{\prime }$ such that $T^{-1}\in \Sigma$ for every $B\in {\Sigma }^{\prime }$. Now define

$$\mu T^{-1}:{\Sigma }^{\prime }\to [0,\infty ],\mu T^{-1}(A^{\prime })=\mu (T^{-1}{A}^{\prime }).$$

If $(S^{\prime },{\Sigma }^{\prime })=(\mathbb{R},\mathcal{B})$ then $\mu T^{-1}=\mu h^{-1}$. If $f\in m{\Sigma }^{\prime +}$, then for every $A^{\prime }\in {\Sigma }^{\prime }$ we have

$${\int }_{T^{-1}{A}^{\prime }}f(Ts)\mu (ds)={\int }_{A^{\prime }}f(s^{\prime })\mu T^{-1}(d{s}^{\prime })\text{ (*) and}$$ $$f\in {\mathcal{L}}_1(S^{\prime },{\Sigma }^{\prime },\mu T^{-1})\text{ iff }f\circ T\in {\mathcal{L}}_1(S,\Sigma ,\mu )\text{ and equation (*) holds.}$$

Now specialise this machinery to densities — which is all the “Jacobian” ever was. Let $X$ be a random variable on $(\Omega ,\mathcal{F},\mathbb{P})$ with law $P_X:=\mathbb{P}{X}^{-1}$ on $(\mathbb{R},\mathcal{B})$, and suppose $P_X$ has a density $p_X$ with respect to Lebesgue measure $\lambda$, i.e. $d{P}_X=p_{X}d\lambda$. Take $T:\mathbb{R}\to \mathbb{R}$ a $C^1$ diffeomorphism and set $Y:=T(X)$. Pushforwards compose, so the law of $Y$ is

$$P_Y=\mathbb{P}{Y}^{-1}=\mathbb{P}(T\circ X{)}^{-1}=(\mathbb{P}{X}^{-1})T^{-1}=P_X{T}^{-1}.$$

For any $A^{\prime }\in \mathcal{B}$, the image-measure identity (*) with $f=1_{A^{\prime }}$ gives

$$P_Y(A^{\prime })=P_X(T^{-1}{A}^{\prime })={\int }_{\mathbb{R}}{1}_{A^{\prime }}(Tx)p_X(x)\lambda (dx).$$

So far there is no Jacobian: the abstract change of variables (*) is exact for any measurable $T$. The factor enters only when we re-express this integral against $\lambda$ itself, via the substitution $y=Tx$. That substitution is not imported from calculus as a black box — it is justified, within this same framework, by the following lemma.

Lemma (density of the pushforward of Lebesgue measure). For a $C^1$ diffeomorphism $T:\mathbb{R}\to \mathbb{R}$, the pushforward $\lambda T^{-1}$ is absolutely continuous with respect to $\lambda$, with

$$\frac{d(\lambda T^{-1})}{d\lambda }(y)=|(T^{-1}{)}^{\prime }(y)|,\text{i.e.}(\lambda T^{-1})(dy)=|(T^{-1}{)}^{\prime }(y)|\lambda (dy).$$

Proof. Take $T$ increasing (the decreasing case is identical up to a sign). For an interval $A^{\prime }=(c,d]$ we have $T^{-1}{A}^{\prime }=(T^{-1}c,T^{-1}d]$, so by the fundamental theorem of calculus (valid since $T^{-1}\in C^1$),

$$(\lambda T^{-1})(A^{\prime })=\lambda (T^{-1}{A}^{\prime })=T^{-1}(d)-T^{-1}(c)={\int }_c^d(T^{-1}{)}^{\prime }(y)dy={\int }_{A^{\prime }}|(T^{-1}{)}^{\prime }|d\lambda .$$

For $T$ decreasing the endpoints swap and $(T^{-1}{)}^{\prime }<0$, which is exactly what produces the absolute value. Both sides are measures in $A^{\prime }$ agreeing on the $\pi$-system of intervals, hence agree on all of $\mathcal{B}$ by uniqueness of measures. $\square$

Now combine the two. Apply (*) with $f=g\circ T^{-1}$, so that $f(Tx)=g(T^{-1}Tx)=g(x)$:

$${\int }_{T^{-1}{A}^{\prime }}g(x)\lambda (dx)={\int }_{A^{\prime }}g(T^{-1}y)(\lambda T^{-1})(dy)={\int }_{A^{\prime }}g(T^{-1}y)|(T^{-1}{)}^{\prime }(y)|\lambda (dy),$$

Trying to derive the first equation in prev eq.

$${\int }_{T^{-1}{A}^{\prime }}g(x)\lambda (dx)={\int }_{A^{\prime }}=g(T^{-1}y)\lambda T^{-1}(dy)$$

the last equality by the lemma. This is the substitution rule $\lambda (dx)=|(T^{-1}{)}^{\prime }(y)|\lambda (dy)$ in differential shorthand; the only analytic input was the fundamental theorem of calculus applied to $\lambda T^{-1}$ on intervals. Taking $g=p_X$ turns our expression for the law of $Y$ into

$$P_Y(A^{\prime })={\int }_{A^{\prime }}{p}_X(T^{-1}y)|(T^{-1}{)}^{\prime }(y)|\lambda (dy).$$

Since this holds for every $A^{\prime }\in \mathcal{B}$, uniqueness of the Radon–Nikodym derivative identifies the density of $Y$:

$$p_Y(y)=\frac{d{P}_Y}{d\lambda }(y)=p_X(T^{-1}y)|(T^{-1}{)}^{\prime }(y)|.$$

The factor $|(T^{-1}{)}^{\prime }(y)|$ is precisely the Radon–Nikodym derivative $\frac{d(\lambda T^{-1})}{d\lambda }$ — the density with which $T$ distorts Lebesgue measure. That is the whole content of the word “Jacobian”: not a separate rule for densities, but the price of measuring against a $\lambda$ that $T$ does not preserve.

For the affine map $T(x)=ax+b$ (with $a\ne 0$) we have $T^{-1}(y)=(y-b)/a$ and $|(T^{-1}{)}^{\prime }(y)|=1/|a|$, so

$$p_{aX+b}(y)=\frac{1}{|a|}{p}_X\left(\frac{y-b}{a}\right).$$

With $a=4$, $b=0$ this is the $p_{4X}(x)=\frac{1}{4}{p}_X(x/4)$ used next; for $X\sim \mathcal{N}(0,1)$ it returns $\mathcal{N}(b,a^2)$.

(1) Recall the convolution operation. For standard normal rvs X,Y we have

$$p(X+Y=x)=\int dap(X=a)p(Y=x-a)=(\frac{1}{\sqrt{2\pi }{\sigma }_X{\sigma }_Y})\exp -\frac{(a-{\mu }_X{)}^2}{2{\sigma }_X^2}\exp -\frac{(x-a-{\mu }_Y{)}^2}{2{\sigma }_Y^2}$$

X+Y should have $\mu ={\mu }_X+{\mu }_Y$ and ${\sigma }^2={\sigma }_X^2+{\sigma }_Y^2$

Let it be standard normal so we simplify to

$$p(X+Y=x)=\int dap(X=a)p(Y=x-a)=\int da\frac{1}{2\pi }\exp (-\frac{a^2}{2})\exp (-\frac{(x-a{)}^2}{2})$$

Complete the square in $a$ to separate the $x$ part from the integration variable:

$$-\frac{1}{2}\left[a^2+(x-a{)}^2\right]=-\left[a^2-ax+\frac{x^2}{2}\right]=-(a-\frac{x}{2}{)}^2-\frac{x^2}{4}$$

so

$$p(X+Y=x)=\frac{1}{2\pi }\exp (-\frac{x^2}{4})\int da\exp (-(a-\frac{x}{2}{)}^2)$$

The remaining integral is a plain Gaussian, $\int da\exp (-(a-\frac{x}{2}{)}^2)=\sqrt{\pi }$, independent of $x$. Hence

$$p(X+Y=x)=\frac{1}{2\pi }\sqrt{\pi }\exp (-\frac{x^2}{4})=\frac{1}{2\sqrt{\pi }}\exp (-\frac{x^2}{4})=\frac{1}{\sqrt{2\pi \cdot 2}}\exp (-\frac{x^2}{2\cdot 2})$$

which is exactly $X+Y\sim \mathcal{N}(0,2)$, confirming $\mu ={\mu }_X+{\mu }_Y=0$ and ${\sigma }^2={\sigma }_X^2+{\sigma }_Y^2=2$.

(2) Apply the affine rule from (0) with $a=4$, $b=0$:

$$p(4X=x)=\frac{1}{4}p(X=x/4)=\frac{1}{4}\cdot \frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x/4-\mu {)}^2}{2{\sigma }^2}\right)=\frac{1}{\sqrt{2\pi }(4\sigma )}\exp \left(-\frac{(x-4\mu {)}^2}{2(4\sigma {)}^2}\right)$$

which is $\mathcal{N}(4\mu ,(4\sigma {)}^2)=\mathcal{N}(4\mu ,16{\sigma }^2)$: the $\frac{1}{4}=|(T^{-1}{)}^{\prime }|$ from (0) turns the prefactor into $\frac{1}{\sqrt{2\pi }(4\sigma )}$, exactly the normalization for standard deviation $4\sigma$.

Now assemble the three pieces. Since $E=4C+N_E$ with $N_E\sim \mathcal{N}(0,1)$, conditioning on $C$ shifts the mean by $4C$, so $E\mid C\sim \mathcal{N}(4C,1)$. The prior is $C\sim \mathcal{N}(0,1)$, and the evidence $E=4C+N_E\sim \mathcal{N}(0,16+1)$ is a plain number (here the $\frac{1}{\sqrt{17}}$ is the $a=\sqrt{17}$ factor from the affine rule in (0)):

$$p(E=2\mid C)=\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{1}{2}(2-4C{)}^2\right),p(C)=\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{1}{2}{C}^2\right),$$ $$p(E=2)=\frac{1}{\sqrt{2\pi \cdot 17}}\exp \left(-\frac{2^2}{2\cdot 17}\right)=\frac{1}{\sqrt{34\pi }}\exp \left(-\frac{2}{17}\right).$$

Multiply likelihood and prior and complete the square in $C$:

$$p(E=2\mid C)p(C)=\frac{1}{2\pi }\exp \left(-\frac{1}{2}\left(17C^2-16C+4\right)\right),$$

and since $17C^2-16C+4=17{\left(C-\frac{8}{17}\right)}^2+\frac{4}{17}$ (using $17\cdot \frac{64}{289}=\frac{64}{17}$ and $4-\frac{64}{17}=\frac{4}{17}$),

$$p(E=2\mid C)p(C)=\frac{1}{2\pi }\exp \left(-\frac{17}{2}{\left(C-\frac{8}{17}\right)}^2\right)\exp \left(-\frac{2}{17}\right).$$

Now divide by $p(E=2)$ — the algebra closes on its own, no appeal to “it must be a normalizer.” The $\exp (-2/17)$ factors cancel exactly, and the prefactor collapses:

$$\frac{1/(2\pi )}{1/\sqrt{34\pi }}=\frac{\sqrt{34\pi }}{2\pi }=\frac{\sqrt{34}}{2\sqrt{\pi }}=\frac{\sqrt{17}}{\sqrt{2\pi }},$$

so

$$p(C\mid E=2)=\frac{\sqrt{17}}{\sqrt{2\pi }}\exp \left(-\frac{17}{2}{\left(C-\frac{8}{17}\right)}^2\right)=\frac{1}{\sqrt{2\pi \cdot \frac{1}{17}}}\exp \left(-\frac{{\left(C-\frac{8}{17}\right)}^2}{2\cdot \frac{1}{17}}\right).$$

This is exactly the density of $\mathcal{N}\left(\frac{8}{17},\frac{1}{17}\right)$, correctly normalized by direct computation — the dividing $p(E=2)$ supplies precisely the $\sqrt{17}/\sqrt{2\pi }$ that a ${\sigma }^2=1/17$ Gaussian requires.

Link to original

Page 57

p. 57

Problem 3.8 Test ABC4

Page 86

p. 86

p415

Problem 4.15 (ANMs)

1. Consider the SCM

$$\begin{array}{rl}X& :=N_X\\ Y& :=2X+N_Y\end{array}$$

with $N_X$ uniformly distributed between $1$ and $3$ and $N_Y$ uniformly distributed between $-0.5$ and $0.5$ and independent of $N_X$. The distribution $P_{X,Y}$ admits an ANM from $X$ to $Y$. Draw the support of the joint distribution of $X,Y$ and convince yourself that $P_{X,Y}$ does not admit an ANM from $Y$ to $X$, that is, there is no function $g$ and independent noise variables $M_X$ and $M_Y$ such that

$$\begin{array}{rl}X& =g(Y)+M_X\\ Y& =M_Y\end{array}$$

with $M_X$ independent of $M_Y$.

Solution Clearly $P\{X<1\vee X>3\}=0.$ Also clearly $\mathrm{supp}Y=[1.5,6.5]$

$$p(x)=\{\begin{array}{l}1/2,\text{ if }x\in [1,3]\\ 0,\text{ if }x\notin [1,3]\end{array}$$ $$p(n)=\{\begin{array}{l}1,\text{ if }x\in [-1/2,1/2]\\ 0,\text{ if }x\notin [-1/2,1/2]\end{array}$$

Recall convolution $p(y)=p_n(n)+{\int }_{-\infty }^{\infty }{p}_x(\frac{y-n}{2})dn$ or $p(y)=\int p_x(x)p_n(y-2x)dx$

So we get

$$p(y\mid x)=p_n(y-2x)=\{\begin{array}{ll}1,& \text{if }y-2x\in [-1/2,1/2]\\ 0,& \text{if }y-2x\notin [-1/2,1/2]\end{array}$$

Multiplying the marginal by the conditional gives the joint density

$$p(x,y)=p(x)p(y\mid x)=\{\begin{array}{ll}1/2,& \text{if }x\in [1,3]\text{ and }|y-2x|\le 1/2\\ 0,& \text{otherwise}\end{array}$$

so the support is the slanted band

$$\mathrm{supp}{P}_{X,Y}=\left\{(x,y):1\le x\le 3,2x-\frac{1}{2}\le y\le 2x+\frac{1}{2}\right\},$$

a parallelogram with vertices $(1,\frac{3}{2}),(1,\frac{5}{2}),(3,\frac{11}{2}),(3,\frac{13}{2})$.

Now read the band the other way. Conditioning on $Y=y$,

$$X\mid Y=y\sim \mathcal{U}\left[\max \left(1,\frac{y}{2}-\frac{1}{4}\right),\min \left(3,\frac{y}{2}+\frac{1}{4}\right)\right].$$

For $y\in [\frac{5}{2},\frac{11}{2}]$ neither bound binds and the interval is $\left[\frac{y}{2}-\frac{1}{4},\frac{y}{2}+\frac{1}{4}\right]$: width $\frac{1}{2}$, a pure translate of a fixed set, which is exactly what an ANM needs. But for $y\in [\frac{3}{2},\frac{5}{2})$ the constraint $x\ge 1$ truncates it to $[1,\frac{y}{2}+\frac{1}{4}]$, of width $\frac{y}{2}-\frac{3}{4}<\frac{1}{2}$, and symmetrically near $y=\frac{13}{2}$.

Why this rules out an ANM (Definition 4.4, p. 67). An ANM from $Y$ to $X$ means $X=g(Y)+M_X$ with $M_X\perp \perp Y$. Independence gives, for bounded measurable $h$ and Borel $B$, by Fubini,

$$\mathbb{E}[h(X)1_B(Y)]={\int }_B\mathbb{E}[h(g(y)+M_X)]P_Y(dy),$$

so for $P_Y$-a.e. $y$ the conditional law $P_{X\mid Y=y}$ is the law of $g(y)+M_X$. Hence $\mathrm{Var}(X\mid Y=y)=\mathrm{Var}(M_X)$ is a.e. constant in $y$. But from the conditional above, $\mathrm{Var}(X\mid Y=y)=\frac{1}{48}$ for $y\in (\frac{5}{2},\frac{11}{2})$ and $=\frac{1}{12}(\frac{y}{2}-\frac{3}{4}{)}^2<\frac{1}{48}$ for $y\in (\frac{3}{2},\frac{5}{2})$, and both sets have positive $P_Y$-mass. Contradiction: there is no ANM from $Y$ to $X$. (Theorem 4.5, p. 67–68, does not apply: it requires strictly positive, smooth densities.)

1. Similarly as in part a), consider the SCM

$$\begin{array}{rl}X& :=N_X\\ Y& :=X^2+N_Y\end{array}$$

with $N_X$ uniformly distributed between $1$ and $3$ and $N_Y$ uniformly distributed between $-0.5$ and $0.5$ and independent of $N_X$. Again, draw the support of $P_{X,Y}$ and convince yourself that there is no ANM from $Y$ to $X$.

Link to original

p416

Problem 4.16 (Maximum likelihood) Assume that we are given an i.i.d. data set $(X_1,Y_1),\dots ,(X_n,Y_n)$ from the model

$$Y=f(X)+N_Y,\text{with }X\sim \mathcal{N}({\mu }_X,{\sigma }_X^2),\text{ and }{N}_Y\sim \mathcal{N}({\mu }_{N_Y},{\sigma }_{N_Y}^2)\text{ independent},$$

where the function $f$ is supposed to be known.

1. Prove that $f(x)=\mathbb{E}[Y\mid X=x]$.

2. Write $\mathbf{x}:=(x_1,\dots ,x_n)$, $\mathbf{y}:=(y_1,\dots ,y_n)$ and consider the log-likelihood function

$${\ell }_{\theta }(\mathbf{x},\mathbf{y})={\ell }_{\theta }((x_1,y_1),\dots ,(x_n,y_n))=\sum _{i=1}^n\log p_{\theta }(x_i,y_i),$$

where $p_{\theta }$ is the joint density over $(X,Y)$ and $\theta :=({\mu }_X,{\mu }_{N_Y},{\sigma }_X^2,{\sigma }_{N_Y}^2)$. Prove that for some $c_1,c_2\in \mathbb{R}$ with $c_2>0$

$$\underset{\theta }{\max }{\ell }_{\theta }(\mathbf{x},\mathbf{y})=c_2\cdot (c_1-\log \widehat{\mathrm{var}}[\mathbf{x}]-\log \widehat{\mathrm{var}}[\mathbf{y}-f(\mathbf{x})]),\text{\hspace{1em}(4.20)}$$

where $\widehat{\mathrm{var}}[\mathbf{z}]:=\frac{1}{n}{\sum }_{i=1}^n(z_i-\frac{1}{n}{\sum }_{k=1}^n{z}_k{)}^2$ estimates the variance.

Equation (4.20) motivates the comparison of expressions (4.18) and (4.19):

$$L_{X\to Y}=-\log \widehat{\mathrm{var}}[X]-\log \widehat{\mathrm{var}}[R_Y],L_{X\leftarrow Y}=-\log \widehat{\mathrm{var}}[R_X]-\log \widehat{\mathrm{var}}[Y],$$

with $R_Y:=Y-{\hat{f}}_Y(X)$ and $R_X:=X-{\hat{f}}_X(Y)$ the regression residuals. The main difference is that in this exercise, we have used the conditional mean and not the outcome of the regression method.

Solution a) As stated this needs ${\mu }_{N_Y}=0$: by linearity of conditional expectation,

$$\mathbb{E}[Y\mid X=x]=\mathbb{E}[f(X)\mid X=x]+\mathbb{E}[N_Y\mid X=x]=f(x)+{\mu }_{N_Y}.$$

Why? Let us use Part A of cond_exp.pdf. Set ${\kappa }_x:={\delta }_{f(x)}$. This is a regular conditional distribution of $f(X)$ given $X$: (K1), (K2) are immediate, and (K3) holds since for Borel $A,B$

$${\int }_A{\delta }_{f(x)}(B)P_X(dx)={\int }_A{1}_{f^{-1}(B)}(x)P_X(dx)=P_X(A\cap f^{-1}(B))=P(X\in A,f(X)\in B).$$

Hence we can write

$$E\left[f(X)\mid X=x\right]=\int z{\delta }_{f(x)}(dz)=f(x).$$

And the second equals the unconditional mean $\mathbb{E}[N_Y]$ because $N_Y\perp \perp X$, because of lemma 2 (note that the independence is used in second to last equation in the equation chain). So $f(x)=\mathbb{E}[Y\mid X=x]$ iff ${\mu }_{N_Y}=0$. This is a harmless normalization: replacing $(f,{\mu }_{N_Y})$ by $(\tilde{f},0)$ with $\tilde{f}:=f+{\mu }_{N_Y}$ gives the identical joint distribution, so only the sum $f+{\mu }_{N_Y}$ — that is, the regression function $x\mapsto \mathbb{E}[Y\mid X=x]$ — is identifiable.

Solution b) The joint density factorizes as $p_{\theta }(x,y)=p(x)p(y\mid x)$ with $X\sim \mathcal{N}({\mu }_X,{\sigma }_X^2)$ and $Y\mid X=x\sim \mathcal{N}(f(x)+{\mu }_{N_Y},{\sigma }_{N_Y}^2)$.

Why does the conditional distribution look like that? Left as a gap.

Writing $r_k:=y_k-f(x_k)$,

$${\ell }_{\theta }(\mathbf{x},\mathbf{y})=\sum _{k=1}^n\left[-\frac{1}{2}\log (2\pi {\sigma }_X^2)-\frac{(x_k-{\mu }_X{)}^2}{2{\sigma }_X^2}\right]+\sum _{k=1}^n\left[-\frac{1}{2}\log (2\pi {\sigma }_{N_Y}^2)-\frac{(r_k-{\mu }_{N_Y}{)}^2}{2{\sigma }_{N_Y}^2}\right]=:{\ell }_{\theta }(\mathbf{x})+{\ell }_{\theta }(\mathbf{r}).$$

Note the $-\frac{n}{2}\log (2\pi {\sigma }^2)$ terms are not constant in $\theta$; without them the ${\sigma }^2$-derivatives would be strictly positive and have no critical point. Note also that ${\mu }_{N_Y}$ must stay free: $\theta$ has four components, and maximizing over the slice ${\mu }_{N_Y}=0$ is a different (smaller) number in general.

The two blocks have the same form, so it suffices to maximize one, say ${\ell }_{\theta }(\mathbf{x})$ over $({\mu }_X,{\sigma }_X^2)$. First-order conditions:

$$\frac{\partial \ell }{\partial {\mu }_X}=\sum _k\frac{x_k-{\mu }_X}{{\sigma }_X^2}=0\Rightarrow {\hat{\mu }}_X=\bar{x}:=\frac{1}{n}{\sum }_k{x}_k,$$ $$\frac{\partial \ell }{\partial {\sigma }_X^2}=-\frac{n}{2{\sigma }_X^2}+\sum _k\frac{(x_k-{\mu }_X{)}^2}{2{\sigma }_X^4}=0\Rightarrow {\hat{\sigma }}_X^2=\frac{1}{n}{\sum }_k(x_k-\bar{x}{)}^2=\widehat{\mathrm{var}}[\mathbf{x}].$$

(This critical point is the maximum, because the density funciton is smooth and can be checked by having the Hessian minors’ determinants have signs -1, 1. Then we can check that it is a global min) Substituting back, the quadratic term collapses to a constant:

$$\sum _k\frac{(x_k-\bar{x}{)}^2}{2\widehat{\mathrm{var}}[\mathbf{x}]}=\frac{n\widehat{\mathrm{var}}[\mathbf{x}]}{2\widehat{\mathrm{var}}[\mathbf{x}]}=\frac{n}{2},$$

so

$$\underset{{\mu }_X,{\sigma }_X^2}{\max }{\ell }_{\theta }(\mathbf{x})=-\frac{n}{2}\log \widehat{\mathrm{var}}[\mathbf{x}]-\frac{n}{2}\log (2\pi )-\frac{n}{2}.$$

The residual block is identical with $r_k$ in place of $x_k$: ${\hat{\mu }}_{N_Y}=\bar{r}$, ${\hat{\sigma }}_{N_Y}^2=\widehat{\mathrm{var}}[\mathbf{y}-f(\mathbf{x})]$ — the free ${\mu }_{N_Y}$ is exactly what centers the residuals, producing the centered $\widehat{\mathrm{var}}$ rather than $\frac{1}{n}{\sum }_k{r}_k^2$: the ${\sigma }^2$-condition is ${\hat{\sigma }}_{N_Y}^2=\frac{1}{n}{\sum }_k(r_k-{\mu }_{N_Y}{)}^2$; with ${\mu }_{N_Y}$ free, ${\hat{\mu }}_{N_Y}=\bar{r}$ and this is $\widehat{\mathrm{var}}[\mathbf{r}]$; with ${\mu }_{N_Y}$ fixed at ${\mu }_0$ it would be $\widehat{\mathrm{var}}[\mathbf{r}]+(\bar{r}-{\mu }_0{)}^2$ and (4.20) would fail as stated. Adding and factoring out $n/2$:

$$\underset{\theta }{\max }{\ell }_{\theta }(\mathbf{x},\mathbf{y})=\frac{n}{2}(\underset{c_1}{\underbrace{-2\log (2\pi )-2}}-\log \widehat{\mathrm{var}}[\mathbf{x}]-\log \widehat{\mathrm{var}}[\mathbf{y}-f(\mathbf{x})]),$$

which is (4.20) with $c_2=n/2>0$ and $c_1=-2(\log 2\pi +1)$. The coefficient 1 on each log term pins $c_2=n/2$ uniquely — $(c_1,c_2)$ cannot be rescaled to, say, $(n,-\log 2\pi -1)$ (why?). Since $c_1,c_2$ do not depend on the direction of the model, comparing causal directions by maximum likelihood reduces to comparing $\log \widehat{\mathrm{var}}[\mathbf{x}]+\log \widehat{\mathrm{var}}[\mathbf{y}-f(\mathbf{x})]$ with its flipped counterpart, i.e. the scores (4.18)/(4.19).

Link to original

Page 96

p. 96

p51

Problem 5.1 (Independence of mechanisms) Let $P_X$ be the mixture of $k$ sharp Gaussian peaks at positions $s_1,\dots ,s_k$ as shown in Figure 5.5, left. Let $Y$ be obtained from $X$ by adding some Gaussian noise $N$ with zero mean and a width ${\sigma }_N$ such that the separate peaks remain visible as in Figure 5.5, right.

1. Argue intuitively why $P_{X\mid Y}$ also contains information about the positions $s_1,\dots ,s_k$ of the peaks and thus $P_{X\mid Y}$ and $P_Y$ share this information.

2. The transition between $P_X$ and $P_Y$ can be described by convolution (from $P_X$ to $P_Y$) and deconvolution (from $P_Y$ to $P_X$). If $P_{Y\mid X}$ is considered as the linear map converting the input $P_X$ to the output $P_Y$, then $P_{Y\mid X}$ coincides with the convolution map. Argue why $P_{X\mid Y}$ does not coincide with the deconvolution map (as one may think at first glance).

Setup in formulas. $P_X={\sum }_{i=1}^k{w}_i\mathcal{N}(s_i,{\sigma }_0^2)$ with weights $w_i>0$, ${\sum }_i{w}_i=1$, and ${\sigma }_0$ small; $Y=X+N$ with $N\sim \mathcal{N}(0,{\sigma }_N^2)$, $N\perp \perp X$, and ${\sigma }_0\ll {\sigma }_N\ll {\min }_{i\ne j}|s_i-s_j|$.

Definition (convolution). For two probability measures $\mu ,\nu$ on $\mathbb{R}$, their convolution is the probability measure

$$(\mu \ast \nu )(B):={\int }_{\mathbb{R}}{\int }_{\mathbb{R}}{1}_B(x+z)\mu (dx)\nu (dz),B\in \mathcal{B}(\mathbb{R}).$$

It is the law of $X+Z$ when $X\sim \mu$, $Z\sim \nu$, and $X\perp \perp Z$ (by the product-measure/Fubini argument of Claim 1 in anm_claims.pdf). If $\mu ,\nu$ have Lebesgue densities $p,q$, then $\mu \ast \nu$ has density

$$(p\ast q)(y)={\int }_{\mathbb{R}}p(x)q(y-x)dx.$$

Convolution is commutative, associative and linear in each argument. In the problem, $P_Y=P_X\ast \mathcal{N}(0,{\sigma }_N^2)$, i.e. $p_Y(y)=\int p_X(x){\phi }_{{\sigma }_N}(y-x)dx$ with ${\phi }_{\sigma }$ the $\mathcal{N}(0,{\sigma }^2)$ density; the map $p_X\mapsto p_X\ast {\phi }_{{\sigma }_N}$ is linear and is exactly the kernel $P_{Y\mid X=x}=\mathcal{N}(x,{\sigma }_N^2)$ integrated against $P_X$.

Definition (deconvolution). Deconvolution is the inverse problem: given $q$ (the noise density) and the output $r=p\ast q$, recover the input $p$. Formally it is the inverse of the linear map $p\mapsto p\ast q$, whenever that inverse exists. With Fourier transforms $\hat{p}(t):=\int e^{itx}p(x)dx$ one has $\widehat{p\ast q}=\hat{p}\hat{q}$, so

$$p={\mathcal{F}}^{-1}(\hat{r}/\hat{q})$$

provided $\hat{q}$ has no zeros. For Gaussian $q$, $\hat{q}(t)=e^{-{\sigma }_N^2{t}^2/2}$ never vanishes, so the inverse exists on a suitable class of functions, but it divides by an exponentially small number at high frequencies: deconvolution is linear but ill-posed (small perturbations of $r$ blow up). Note it is a fixed linear map determined by $q$ alone; it does not depend on which $p$ produced $r$.

Solution

a) We have $X\to Y,Y=X+N,N\sim \mathcal{N}(0,{\sigma }_N^2)$.

Solution a) If we choose $y$ midway between two adjacent peaks, say $s_1$ and $s_2$, then $P_{X\mid Y=y}$ puts its mass equally at $s_1$ and $s_2$ with a gap in the middle, so we know that we are not uniquely near any $s_i$. If we choose $y$ near $s_1$, then $P_{X\mid Y=y}$ has much more mass around $s_1$. Reason: $p(x\mid y)\propto p_X(x){\phi }_{{\sigma }_N}(y-x)$, and the factor $p_X(x)$ is $\approx 0$ away from the $s_i$, so for every $y$ the mass of $P_{X\mid Y=y}$ sits at the peak positions; $y$ only sets their relative weights $\propto w_i{\phi }_{{\sigma }_N}(y-s_i)$. Hence $P_{X\mid Y}$ reveals $s_1,\dots ,s_k$, just as $P_Y$ does, while $P_{Y\mid X=x}=\mathcal{N}(x,{\sigma }_N^2)$ does not.

Solution b) Fix the model $P_X={\sum }_k{w}_k\mathcal{N}(s_k,{\sigma }_0^2)$, $Y=X+N$, $N\sim \mathcal{N}(0,{\sigma }_N^2)$, $N\perp \perp X$, and write $S:=\{s_1,\dots ,s_k\}$, $s_{\min }:=\min S$, $s_{\max }:=\max S$,

$$\kappa :=\frac{{\sigma }_0^2}{{\sigma }_0^2+{\sigma }_N^2},v:=\frac{{\sigma }_0^2{\sigma }_N^2}{{\sigma }_0^2+{\sigma }_N^2},m_j(y):=(1-\kappa )s_j+\kappa y.$$

By Bayes (deconv_helpers), the posterior is $P_{X\mid Y=y}={\sum }_j{\pi }_j(y)\mathcal{N}(m_j(y),v)$ with ${\pi }_j(y)\propto w_j{\phi }_{\sqrt{{\sigma }_0^2+{\sigma }_N^2}}(y-s_j)$ (^bde910, ). The two maps on distributions:

$$C(\tilde{Q}):=\text{deconvolution of }\tilde{Q}\text{ by }\mathcal{N}(0,{\sigma }_N^2),K(\tilde{Q})(B):=\int P_{X\mid Y=y}(B)\tilde{Q}(dy).$$

Both send $P_Y\mapsto P_X$ (Recall $P_X(B)=\int P_{X\mid Y=y}(B)P_Y(dy)$ from deconv_helpers and this is simply (actually it is a nightmare) Bayes for Kernel map and cf subtraction for deconv). Two maps differ iff they differ at one input, so we exhibit a $\tilde{Q}$ and a Borel set $B$ with $C(\tilde{Q})(B)\ne K(\tilde{Q})(B)$. (In fact they agree only at $P_Y$; not needed here.)

Representation. A sample $X^{\prime }\sim K(\tilde{Q})$ can be generated in two stages: draw $\tilde{Y}\sim \tilde{Q}$; given $\tilde{Y}=y$, pick the index $J=j$ with probability ${\pi }_j(y)$ and set $X^{\prime }=m_J(\tilde{Y})+Z$, $Z\sim \mathcal{N}(0,v)$ independent (why?). Hence

$$X^{\prime }=(1-\kappa )S^{\prime }+\kappa \tilde{Y}+Z,S^{\prime }:=s_J\in [s_{\min },s_{\max }].$$

All dependence on the intractable weights ${\pi }_j$ is confined to $S^{\prime }$, of which we use only the bound $s_{\min }\le S^{\prime }\le s_{\max }$.

Sandwich. If $X=A+W$ and $X^{\prime \prime }=A^{\prime \prime }+W$ on one probability space with $A\le A^{\prime \prime }$ a.s., then $P(X>x)\le P(X^{\prime \prime }>x)$ for all $x$ (why?). Applying this with $A=(1-\kappa )S^{\prime }$ gives, for every input $\tilde{Q}$,

$$\text{law of }(1-\kappa )s_{\min }+\kappa \tilde{Y}+Z⪯K(\tilde{Q})⪯\text{law of }(1-\kappa )s_{\max }+\kappa \tilde{Y}+Z,$$

where $\mu ⪯\nu$ means $\mu ((x,\infty ))\le \nu ((x,\infty ))$ for all $x$.

Separating input. Take $\tilde{Q}:=P_Y(\cdot -t)={\sum }_j{w}_j\mathcal{N}(s_j+t,{\sigma }_0^2+{\sigma }_N^2)$ for a shift $t>0$ to be chosen. Writing $\tilde{Y}=S^{\prime \prime }+t+W$ with $S^{\prime \prime }\in [s_{\min },s_{\max }]$, $W\sim \mathcal{N}(0,{\sigma }_0^2+{\sigma }_N^2)$ (why?), the upper bound collapses:

$$K(\tilde{Q})⪯\text{law of }{s}_{\max }+\kappa t+\kappa W+Z=\mathcal{N}(s_{\max }+\kappa t,{\sigma }_0^2),$$

using ${\kappa }^2({\sigma }_0^2+{\sigma }_N^2)+v={\sigma }_0^2$, which is the law of total variance $\mathrm{Var}X=\mathrm{Var}\mathbb{E}[X\mid Y]+\mathbb{E}\mathrm{Var}[X\mid Y]$ in the single-peak toy (why?). On the other side, deconvolution commutes with translation (why?), so $C(\tilde{Q})=P_X(\cdot -t)={\sum }_j{w}_j\mathcal{N}(s_j+t,{\sigma }_0^2)$, and the same selection trick gives $C(\tilde{Q})⪰\mathcal{N}(s_{\min }+t,{\sigma }_0^2)$.

Threshold. The bounding Gaussians have equal width ${\sigma }_0$ and centre gap $(s_{\min }+t)-(s_{\max }+\kappa t)=(1-\kappa )t-\mathrm{diam}(S)$. Choose $t>\mathrm{diam}(S)/(1-\kappa )$, so the gap is positive, and let $x^{\ast }$ be the midpoint of the two centres. Then

$$K(\tilde{Q})((x^{\ast },\infty ))<\frac{1}{2}<C(\tilde{Q})((x^{\ast },\infty )),$$

so $K(\tilde{Q})\ne C(\tilde{Q})$ and the maps differ. $\square$

Moral. $C$ is built from the noise alone and follows its input anywhere; $K$ carries the prior inside ($s_j,\kappa ,v,{\pi }_j$) and every output is dragged back to within $\kappa$-reach of the old peaks. The two coincide exactly at the one input $P_Y$ that the old prior generated — Bayes’ inversion of this joint, not an inverse of the blur.

Lemma (uniqueness). $C(\tilde{Q})=K(\tilde{Q})$ holds only at $\tilde{Q}=P_Y$.

Tools. (i) A Markov kernel $T(x,\cdot )$ is a probability measure on $\mathbb{R}$ for each $x$, measurable in $x$; it acts on measures by $(\mu T)(B):=\int T(x,B)\mu (dx)$. $\mu$ is invariant if $\mu T=\mu$. (ii) Hahn decomposition: every finite signed measure $\nu$ splits $\mathbb{R}=P\bigsqcup N$ with $\nu \ge 0$ on subsets of $P$ and $\nu \le 0$ on subsets of $N$; then ${\nu }^{+}(B):=\nu (B\cap P)$, ${\nu }^{-}(B):=-\nu (B\cap N)$ are finite measures with $\nu ={\nu }^{+}-{\nu }^{-}$.

Step 1 (reformulate as a fixed point). Define the Gibbs kernel (go $x\to y$ by the forward noise, then $y\to x^{\prime }$ by the posterior)

$$T(x,B):=\int P_{X\mid Y=y}(B){\phi }_{{\sigma }_N}(y-x)dy.$$

Then $\mu T=K(\mu \ast \mathcal{N}(0,{\sigma }_N^2))$ for every probability $\mu$ (why?). If $C(\tilde{Q})=K(\tilde{Q})=:R$, then $\tilde{Q}=R\ast \mathcal{N}(0,{\sigma }_N^2)$ by definition of $C$, hence $R=K(\tilde{Q})=RT$: $R$ is invariant for $T$. Conversely an invariant $R$ gives the agreement point $\tilde{Q}=R\ast \mathcal{N}(0,{\sigma }_N^2)$. So agreement points $\leftrightarrow$ invariant probabilities of $T$, via $\tilde{Q}=R\ast \mathcal{N}(0,{\sigma }_N^2)$.

Step 2 ($P_X$ is invariant). $P_X\ast \mathcal{N}(0,{\sigma }_N^2)=P_Y$ and $K(P_Y)=P_X$, so $P_{X}T=P_X$. This gives the known agreement point $\tilde{Q}=P_Y$.

Step 3 (no other invariant probability). $T(x,\cdot )$ has a strictly positive Lebesgue density for every $x$, since $P_{X\mid Y=y}$ is a Gaussian mixture with $v>0$ (why?). Let ${\pi }_1,{\pi }_2$ be invariant probabilities and $\nu :={\pi }_1-{\pi }_2$, so $\nu T=\nu$ and $\nu (\mathbb{R})=0$. Take a Hahn decomposition $\mathbb{R}=P\bigsqcup N$ of $\nu$. For every Borel $B$,

$${\nu }^{+}(B)=\nu (B\cap P)=({\nu }^{+}T)(B\cap P)-({\nu }^{-}T)(B\cap P)\le ({\nu }^{+}T)(B\cap P)\le ({\nu }^{+}T)(B),$$

and ${\nu }^{+}(\mathbb{R})=({\nu }^{+}T)(\mathbb{R})$ because $T$ preserves mass. A measure dominated by another of equal total mass equals it (why?), so ${\nu }^{+}={\nu }^{+}T$. Then

$$0={\nu }^{+}(N)=({\nu }^{+}T)(N)=\int T(x,N){\nu }^{+}(dx),$$

and $T(x,N)>0$ for all $x$ whenever $\mathrm{Leb}(N)>0$. Hence ${\nu }^{+}=0$ or $\mathrm{Leb}(N)=0$. Symmetrically ${\nu }^{-}=0$ or $\mathrm{Leb}(P)=0$. If $\nu \ne 0$ then ${\nu }^{+},{\nu }^{-}$ are both nonzero (equal mass, since $\nu (\mathbb{R})=0$), forcing $\mathrm{Leb}(P)=\mathrm{Leb}(N)=0$, i.e. $\mathrm{Leb}(\mathbb{R})=0$. Contradiction, so ${\pi }_1={\pi }_2$. By Steps 1–2 the unique agreement point is $P_Y$. $\square$

Check in the single-peak toy (Fourier). With $k=1$, $s_1=0$: $K(\tilde{Q})$ is the law of $\kappa \tilde{Y}+Z$, so $\widehat{K(\tilde{Q})}(t)=\hat{\tilde{Q}}(\kappa t)e^{-v{t}^2/2}$, while $\widehat{C(\tilde{Q})}(t)=\hat{\tilde{Q}}(t)e^{{\sigma }_N^2{t}^2/2}$ (Helper 3). Agreement means $\hat{\tilde{Q}}(t)=\hat{\tilde{Q}}(\kappa t)e^{-(v+{\sigma }_N^2)t^2/2}$; iterating $n$ times and letting $n\to \infty$ (using $\kappa <1$ and $\hat{\tilde{Q}}$ continuous at $0$ with $\hat{\tilde{Q}}(0)=1$) gives $\hat{\tilde{Q}}(t)=e^{-(v+{\sigma }_N^2)t^2/(2(1-{\kappa }^2))}$, and $(v+{\sigma }_N^2)/(1-{\kappa }^2)={\sigma }_0^2+{\sigma }_N^2$ (why?). So $\tilde{Q}=\mathcal{N}(0,{\sigma }_0^2+{\sigma }_N^2)=P_Y$. This route uses that convolution and $K$ are both diagonal in the Fourier basis (multiply/rescale $\hat{\tilde{Q}}$); for $k\ge 2$ the $y$-dependent weights ${\pi }_j(y)$ break this, which is why the lemma goes through $T$ instead.

Link to original

deconv_helpers

Helpers: convolution and deconvolution.

Helper 1 (convolution as a matrix, finite case). Let $X\in \{1,\dots ,m\}$ with probability vector $p$, and noise $N\in \{0,1\}$ with $P(N=0)=P(N=1)=\frac{1}{2}$, independent of $X$. Then $Y=X+N\in \{1,\dots ,m+1\}$ has

$$r(y)=\frac{1}{2}p(y)+\frac{1}{2}p(y-1),p(0):=p(m+1):=0,$$

i.e. $r=Ap$ for the $(m+1)\times m$ banded matrix $A$ with $\frac{1}{2}$ on the diagonal and subdiagonal. Deconvolution is solving $Ap=r$ for $p$: $p(1)=2r(1)$, $p(2)=2r(2)-p(1)$, …, a back-substitution that exists and is unique. Note (i) $A$ depends only on the noise law; (ii) the recipe does not care what $p$ is; (iii) errors in $r$ double at every step (instability).

Helper 2 (Gaussians convolve by adding variances). If $X\sim \mathcal{N}(\mu ,a^2)$, $N\sim \mathcal{N}(0,b^2)$, independent, then $Y=X+N\sim \mathcal{N}(\mu ,a^2+b^2)$. (Prove once: either $\int {\phi }_a(x-\mu ){\phi }_b(y-x)dx$ by completing the square, or via characteristic functions $e^{i\mu t-a^2{t}^2/2}\cdot e^{-b^2{t}^2/2}$.) Hence within the Gaussian family deconvolution is “subtract the noise variance”: given $Y\sim \mathcal{N}(\mu ,c^2)$ and known $b$, the input was $\mathcal{N}(\mu ,c^2-b^2)$. It exists iff $c^2\ge b^2$: a $P_Y$ narrower than the noise cannot have been produced by any $P_X$. Again the map $c^2\mapsto c^2-b^2$ is fixed by $b$ alone.

Helper 3 (Fourier makes it linear algebra). For densities, $\widehat{p\ast q}=\hat{p}\hat{q}$ with $\hat{p}(t):=\int e^{itx}p(x)dx$; this is the independence identity $\mathbb{E}{e}^{it(X+N)}=\mathbb{E}{e}^{itX}\mathbb{E}{e}^{itN}$ [T6 in cond_exp.pdf]. So convolution is a diagonal operator in the Fourier basis, with “eigenvalue” $\hat{q}(t)$ at frequency $t$, and deconvolution divides by it: $\hat{p}=\hat{r}/\hat{q}$. Helper 2 is this with $\hat{q}(t)=e^{-b^2{t}^2/2}$: dividing subtracts $b^2$ from the variance. Ill-posedness $=$ $\hat{q}(t)\to 0$ as $|t|\to \infty$, so high-frequency error in $r$ is amplified without bound.

Common thread. Deconvolution is a map on distributions, $P_Y\mapsto P_X$, determined by the noise law only, and it is linear. Keep this in mind for Problem 5.1 b).

Exercise (stays inside Helper 2). For $X\sim \mathcal{N}(0,a^2)$, $N\sim \mathcal{N}(0,b^2)$, compute the conditional law $P_{X\mid Y=y}$ (Gaussian; same computation as Problem 3.6 in p36.tex). Then ask what map on distributions $P_{X\mid Y}$ induces via $P_X(B)=\int P_{X\mid Y=y}(B)P_Y(dy)$, and compare its ingredients with the deconvolution map’s ingredients.

Solution Let us fix $y$ and write

$$p(x|y)=\frac{p(x)p(y|x)}{p(y)}=\frac{1/a\varphi (x/a)\cdot 1/b\varphi ((y-x)/b)}{1/\sqrt{a^2+b^2}\varphi (y/\sqrt{a^2+b^2})}=c(y)\exp \left[-(x/a{)}^2/2-(y-x{)}^2/2b^2+y^2/2(a+b)\right]$$

Now note that $y^2/(a+b)\text{ and },y^2/2b^2$ can be subsumed into some new $\tilde{c}(y)$. Now focus on the exponent terms containing $x$

$$\begin{array}{rl}p(x|y)& =c(y)\exp \left[-x^2/2a^2+xy/b^2-x^2/2b^2\right]\\ & =\exp \left[\frac{-b^2{x}^2+2a^{2}xy-a^2{x}^2}{2a^2{b}^2}\right]\\ & =\exp \left[\frac{-(\sqrt{a^2+b^2}x+\dots )}{2a^2{b}^2}\right]\end{array}$$

Results side by side ($X\sim \mathcal{N}(0,a^2)$, $N\sim \mathcal{N}(0,b^2)$, $Y=X+N$, $c^2:=a^2+b^2$):

$$\begin{array}{rlrl}{P}_{X\mid Y=y}& =\mathcal{N}(\frac{a^2}{a^2+b^2}y,\frac{a^2{b}^2}{a^2+b^2})& & \text{(conditional; mean depends on }y\text{, variance }<a^2,<b^2)\\ P_X& =\mathcal{N}(0,c^2-b^2)& & \text{(deconvolution of }{P}_Y=\mathcal{N}(0,c^2)\text{; ingredients: }{P}_Y\text{ and }b\text{ only)}\end{array}$$

Both recover $P_X$ from $P_Y$: the second directly, the first via $P_X(B)=\int P_{X\mid Y=y}(B)P_Y(dy)$. They are different maps on distributions: the first has $a$ (i.e. $P_X$) baked into its coefficients, the second does not.

The two maps. Fix the model $X\sim \mathcal{N}(\mu ,a^2)$, $N\sim \mathcal{N}(0,b^2)$, $N\perp \perp X$, $Y=X+N$, and write $\kappa :=a^2/(a^2+b^2)$, $v:=a^2{b}^2/(a^2+b^2)$, $m(y):=\mu +\kappa (y-\mu )$, so that $P_{X\mid Y=y}=\mathcal{N}(m(y),v)$. Both maps take a probability measure $\tilde{Q}$ on the $y$-axis to one on the $x$-axis:

$$\begin{array}{rlrl}C(\tilde{Q})& :=\mathcal{N}(\tilde{\mu },{\tilde{c}}^2-b^2)\text{for }\tilde{Q}=\mathcal{N}(\tilde{\mu },{\tilde{c}}^2)& & \text{(deconvolution; built from }b\text{ only)}\\ K(\tilde{Q})(B)& :=\int \mathcal{N}(m(y),v)(B)\tilde{Q}(dy)& & \text{(kernel map; built from }\mu ,\kappa ,v\text{, i.e. from }{P}_X\text{)}\end{array}$$

Here $\mu ,a,b$ (hence $\kappa ,v,m$) are frozen into the maps; $\tilde{\mu },{\tilde{c}}^2$ parameterize the varying input.

With the help of Claim 1 (anm_claims.pdf) and defining $V:=m(\tilde{Y})+Z$ with $\tilde{Y}\sim \tilde{Q}$, $Z\sim \mathcal{N}(0,v)$, $Z\perp \perp \tilde{Y}$, we can show that $P_V=K(\tilde{Q})$, with the distribution

$$V\sim \mathcal{N}(\mu (1-\kappa )+\kappa \tilde{\mu },{\kappa }^2{\tilde{c}}^2+v).$$

Since $C(\tilde{Q})=\mathcal{N}(\tilde{\mu },{\tilde{c}}^2-b^2)$ while $K(\tilde{Q})=P_V=\mathcal{N}(\mu (1-\kappa )+\kappa \tilde{\mu },{\kappa }^2{\tilde{c}}^2+v)$, the two maps differ whenever $\tilde{\mu }\ne \mu$; moreover for ${\tilde{c}}^2<b^2$ the map $C$ is undefined while $K$ is not, and for $\tilde{\mu }=\mu$ the variances agree only at ${\tilde{c}}^2=a^2+b^2$. The maps agree exactly at $\tilde{Q}=P_Y$, where both return $P_X$.

Link to original