Problem 3.6

Problem 3.6. Recall Eq (3.4):

$$P_C^{\mathcal{C};do(E:=2)}=\mathcal{N}(0,1)\ne P_{C|E=2}^{\mathcal{C}},$$

where $C:=N_C,E:=4C+N_E$ and $N_C,N_E\sim \mathcal{N}(0,1)$.

Show that $P_{C|E=2}^{\mathcal{C}}$ is Gaussian:

$$C|E=2\sim \mathcal{N}(8/17,{\sigma }^2=1/17)$$

Per Bayes’ theorem $p(C|E=2)=p(E=2|C)p(C)/p(E=2)$, and our goal is to massage the right-hand side into the standard Gaussian form

$$p(C|E=2)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(C-\mu {)}^2}{2{\sigma }^2}\right),\mu =8/17,{\sigma }^2=1/17.$$

(0) First let us remind ourselves how the change of variables work.

When we have a continuous measure $\mu \ll \mathrm{Leb}$ then we can write $\int d\mu =\int p(x)dx$ where $p(x)$ is the density function with respect to the Lebesgue measure. We can write $F(x):=P(X<x)$. Now let us look at some transformation $g:X\mapsto g(X)$ and we care about $G(x):=P(g(X)<x)$.

Let $(S,\Sigma )$ be measurable and $h:S\to \mathbb{R}$ function and $\mathcal{B}:=\mathcal{B}(\mathbb{R})$. We say that $h$ is $\Sigma -$measurable or $\Sigma \setminus \mathcal{B}$-measurable if every Borel set has it’s original in $\Sigma$.

Now let us look at measure spaces $(S,\Sigma ,\mu )$ and $(S^{\prime },{\Sigma }^{\prime })$. Let us define a generalisation for $\mu h^{-1}$.

Let us have a function $T:S\to S^{\prime }$ such that $T^{-1}\in \Sigma$ for every $B\in {\Sigma }^{\prime }$. Now define

$$\mu T^{-1}:{\Sigma }^{\prime }\to [0,\infty ],\mu T^{-1}(A^{\prime })=\mu (T^{-1}{A}^{\prime }).$$

If $(S^{\prime },{\Sigma }^{\prime })=(\mathbb{R},\mathcal{B})$ then $\mu T^{-1}=\mu h^{-1}$. If $f\in m{\Sigma }^{\prime +}$, then for every $A^{\prime }\in {\Sigma }^{\prime }$ we have

$${\int }_{T^{-1}{A}^{\prime }}f(Ts)\mu (ds)={\int }_{A^{\prime }}f(s^{\prime })\mu T^{-1}(d{s}^{\prime })\text{ (*) and}$$ $$f\in {\mathcal{L}}_1(S^{\prime },{\Sigma }^{\prime },\mu T^{-1})\text{ iff }f\circ T\in {\mathcal{L}}_1(S,\Sigma ,\mu )\text{ and equation (*) holds.}$$

Now specialise this machinery to densities — which is all the “Jacobian” ever was. Let $X$ be a random variable on $(\Omega ,\mathcal{F},\mathbb{P})$ with law $P_X:=\mathbb{P}{X}^{-1}$ on $(\mathbb{R},\mathcal{B})$, and suppose $P_X$ has a density $p_X$ with respect to Lebesgue measure $\lambda$, i.e. $d{P}_X=p_{X}d\lambda$. Take $T:\mathbb{R}\to \mathbb{R}$ a $C^1$ diffeomorphism and set $Y:=T(X)$. Pushforwards compose, so the law of $Y$ is

$$P_Y=\mathbb{P}{Y}^{-1}=\mathbb{P}(T\circ X{)}^{-1}=(\mathbb{P}{X}^{-1})T^{-1}=P_X{T}^{-1}.$$

For any $A^{\prime }\in \mathcal{B}$, the image-measure identity (*) with $f=1_{A^{\prime }}$ gives

$$P_Y(A^{\prime })=P_X(T^{-1}{A}^{\prime })={\int }_{\mathbb{R}}{1}_{A^{\prime }}(Tx)p_X(x)\lambda (dx).$$

So far there is no Jacobian: the abstract change of variables (*) is exact for any measurable $T$. The factor enters only when we re-express this integral against $\lambda$ itself, via the substitution $y=Tx$. That substitution is not imported from calculus as a black box — it is justified, within this same framework, by the following lemma.

Lemma (density of the pushforward of Lebesgue measure). For a $C^1$ diffeomorphism $T:\mathbb{R}\to \mathbb{R}$, the pushforward $\lambda T^{-1}$ is absolutely continuous with respect to $\lambda$, with

$$\frac{d(\lambda T^{-1})}{d\lambda }(y)=|(T^{-1}{)}^{\prime }(y)|,\text{i.e.}(\lambda T^{-1})(dy)=|(T^{-1}{)}^{\prime }(y)|\lambda (dy).$$

Proof. Take $T$ increasing (the decreasing case is identical up to a sign). For an interval $A^{\prime }=(c,d]$ we have $T^{-1}{A}^{\prime }=(T^{-1}c,T^{-1}d]$, so by the fundamental theorem of calculus (valid since $T^{-1}\in C^1$),

$$(\lambda T^{-1})(A^{\prime })=\lambda (T^{-1}{A}^{\prime })=T^{-1}(d)-T^{-1}(c)={\int }_c^d(T^{-1}{)}^{\prime }(y)dy={\int }_{A^{\prime }}|(T^{-1}{)}^{\prime }|d\lambda .$$

For $T$ decreasing the endpoints swap and $(T^{-1}{)}^{\prime }<0$, which is exactly what produces the absolute value. Both sides are measures in $A^{\prime }$ agreeing on the $\pi$-system of intervals, hence agree on all of $\mathcal{B}$ by uniqueness of measures. $\square$

Now combine the two. Apply (*) with $f=g\circ T^{-1}$, so that $f(Tx)=g(T^{-1}Tx)=g(x)$:

$${\int }_{T^{-1}{A}^{\prime }}g(x)\lambda (dx)={\int }_{A^{\prime }}g(T^{-1}y)(\lambda T^{-1})(dy)={\int }_{A^{\prime }}g(T^{-1}y)|(T^{-1}{)}^{\prime }(y)|\lambda (dy),$$

Trying to derive the first equation in prev eq.

$${\int }_{T^{-1}{A}^{\prime }}g(x)\lambda (dx)={\int }_{A^{\prime }}=g(T^{-1}y)\lambda T^{-1}(dy)$$

the last equality by the lemma. This is the substitution rule $\lambda (dx)=|(T^{-1}{)}^{\prime }(y)|\lambda (dy)$ in differential shorthand; the only analytic input was the fundamental theorem of calculus applied to $\lambda T^{-1}$ on intervals. Taking $g=p_X$ turns our expression for the law of $Y$ into

$$P_Y(A^{\prime })={\int }_{A^{\prime }}{p}_X(T^{-1}y)|(T^{-1}{)}^{\prime }(y)|\lambda (dy).$$

Since this holds for every $A^{\prime }\in \mathcal{B}$, uniqueness of the Radon–Nikodym derivative identifies the density of $Y$:

$$p_Y(y)=\frac{d{P}_Y}{d\lambda }(y)=p_X(T^{-1}y)|(T^{-1}{)}^{\prime }(y)|.$$

The factor $|(T^{-1}{)}^{\prime }(y)|$ is precisely the Radon–Nikodym derivative $\frac{d(\lambda T^{-1})}{d\lambda }$ — the density with which $T$ distorts Lebesgue measure. That is the whole content of the word “Jacobian”: not a separate rule for densities, but the price of measuring against a $\lambda$ that $T$ does not preserve.

For the affine map $T(x)=ax+b$ (with $a\ne 0$) we have $T^{-1}(y)=(y-b)/a$ and $|(T^{-1}{)}^{\prime }(y)|=1/|a|$, so

$$p_{aX+b}(y)=\frac{1}{|a|}{p}_X\left(\frac{y-b}{a}\right).$$

With $a=4$, $b=0$ this is the $p_{4X}(x)=\frac{1}{4}{p}_X(x/4)$ used next; for $X\sim \mathcal{N}(0,1)$ it returns $\mathcal{N}(b,a^2)$.

(1) Recall the convolution operation. For standard normal rvs X,Y we have

$$p(X+Y=x)=\int dap(X=a)p(Y=x-a)=(\frac{1}{\sqrt{2\pi }{\sigma }_X{\sigma }_Y})\exp -\frac{(a-{\mu }_X{)}^2}{2{\sigma }_X^2}\exp -\frac{(x-a-{\mu }_Y{)}^2}{2{\sigma }_Y^2}$$

X+Y should have $\mu ={\mu }_X+{\mu }_Y$ and ${\sigma }^2={\sigma }_X^2+{\sigma }_Y^2$

Let it be standard normal so we simplify to

$$p(X+Y=x)=\int dap(X=a)p(Y=x-a)=\int da\frac{1}{2\pi }\exp (-\frac{a^2}{2})\exp (-\frac{(x-a{)}^2}{2})$$

Complete the square in $a$ to separate the $x$ part from the integration variable:

$$-\frac{1}{2}\left[a^2+(x-a{)}^2\right]=-\left[a^2-ax+\frac{x^2}{2}\right]=-(a-\frac{x}{2}{)}^2-\frac{x^2}{4}$$

so

$$p(X+Y=x)=\frac{1}{2\pi }\exp (-\frac{x^2}{4})\int da\exp (-(a-\frac{x}{2}{)}^2)$$

The remaining integral is a plain Gaussian, $\int da\exp (-(a-\frac{x}{2}{)}^2)=\sqrt{\pi }$, independent of $x$. Hence

$$p(X+Y=x)=\frac{1}{2\pi }\sqrt{\pi }\exp (-\frac{x^2}{4})=\frac{1}{2\sqrt{\pi }}\exp (-\frac{x^2}{4})=\frac{1}{\sqrt{2\pi \cdot 2}}\exp (-\frac{x^2}{2\cdot 2})$$

which is exactly $X+Y\sim \mathcal{N}(0,2)$, confirming $\mu ={\mu }_X+{\mu }_Y=0$ and ${\sigma }^2={\sigma }_X^2+{\sigma }_Y^2=2$.

(2) Apply the affine rule from (0) with $a=4$, $b=0$:

$$p(4X=x)=\frac{1}{4}p(X=x/4)=\frac{1}{4}\cdot \frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x/4-\mu {)}^2}{2{\sigma }^2}\right)=\frac{1}{\sqrt{2\pi }(4\sigma )}\exp \left(-\frac{(x-4\mu {)}^2}{2(4\sigma {)}^2}\right)$$

which is $\mathcal{N}(4\mu ,(4\sigma {)}^2)=\mathcal{N}(4\mu ,16{\sigma }^2)$: the $\frac{1}{4}=|(T^{-1}{)}^{\prime }|$ from (0) turns the prefactor into $\frac{1}{\sqrt{2\pi }(4\sigma )}$, exactly the normalization for standard deviation $4\sigma$.

Now assemble the three pieces. Since $E=4C+N_E$ with $N_E\sim \mathcal{N}(0,1)$, conditioning on $C$ shifts the mean by $4C$, so $E\mid C\sim \mathcal{N}(4C,1)$. The prior is $C\sim \mathcal{N}(0,1)$, and the evidence $E=4C+N_E\sim \mathcal{N}(0,16+1)$ is a plain number (here the $\frac{1}{\sqrt{17}}$ is the $a=\sqrt{17}$ factor from the affine rule in (0)):

$$p(E=2\mid C)=\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{1}{2}(2-4C{)}^2\right),p(C)=\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{1}{2}{C}^2\right),$$ $$p(E=2)=\frac{1}{\sqrt{2\pi \cdot 17}}\exp \left(-\frac{2^2}{2\cdot 17}\right)=\frac{1}{\sqrt{34\pi }}\exp \left(-\frac{2}{17}\right).$$

Multiply likelihood and prior and complete the square in $C$:

$$p(E=2\mid C)p(C)=\frac{1}{2\pi }\exp \left(-\frac{1}{2}\left(17C^2-16C+4\right)\right),$$

and since $17C^2-16C+4=17{\left(C-\frac{8}{17}\right)}^2+\frac{4}{17}$ (using $17\cdot \frac{64}{289}=\frac{64}{17}$ and $4-\frac{64}{17}=\frac{4}{17}$),

$$p(E=2\mid C)p(C)=\frac{1}{2\pi }\exp \left(-\frac{17}{2}{\left(C-\frac{8}{17}\right)}^2\right)\exp \left(-\frac{2}{17}\right).$$

Now divide by $p(E=2)$ — the algebra closes on its own, no appeal to “it must be a normalizer.” The $\exp (-2/17)$ factors cancel exactly, and the prefactor collapses:

$$\frac{1/(2\pi )}{1/\sqrt{34\pi }}=\frac{\sqrt{34\pi }}{2\pi }=\frac{\sqrt{34}}{2\sqrt{\pi }}=\frac{\sqrt{17}}{\sqrt{2\pi }},$$

so

$$p(C\mid E=2)=\frac{\sqrt{17}}{\sqrt{2\pi }}\exp \left(-\frac{17}{2}{\left(C-\frac{8}{17}\right)}^2\right)=\frac{1}{\sqrt{2\pi \cdot \frac{1}{17}}}\exp \left(-\frac{{\left(C-\frac{8}{17}\right)}^2}{2\cdot \frac{1}{17}}\right).$$

This is exactly the density of $\mathcal{N}\left(\frac{8}{17},\frac{1}{17}\right)$, correctly normalized by direct computation — the dividing $p(E=2)$ supplies precisely the $\sqrt{17}/\sqrt{2\pi }$ that a ${\sigma }^2=1/17$ Gaussian requires.