Starting point: Classical entropy and mutual information, density operators, partial traces and the protocols that use Bell pairs; spectral representations and matrix logarithms are developed in context.
Keep in view: Compare the entropy of a Bell pair with the entropy of either subsystem.
The protocols ran on Bell pairs. Each pair is described by a single pure vector, yet each half is as uncertain as a fair coin. For classical sources we measured uncertainty by entropy and correlation by mutual information. We have no such numbers for quantum states.
The obvious attempt is to measure the state and take the entropy of the outcomes. But the answer depends on the measurement: plus gives a fair coin in Z and certainty in X. What single number measures the uncertainty of a density operator? Asking for the least uncertain measurement leads to the eigenvalues, and their entropy is the von Neumann entropy. Applying it to joint and reduced states gives quantum mutual information and conditional entropy, and the Bell pair makes the conditional entropy negative. Comparing two states gives relative entropy. One theorem about relative entropy then yields the Holevo bound, the general ceiling on information extracted from an encoding.
11.1 State Entropy
Which probabilities should the entropy of a quantum state use? We first try outcome entropies of different measurements on a family of states and find that they disagree. The disagreement singles out one basis, the eigenbasis of the density operator, whose probabilities are its eigenvalues. Their entropy is the von Neumann entropy, which is zero for every pure state.
where \(r=1\) is pure plus and \(r=0\) is \(I/2\). How uncertain is it? A Z measurement is fair for every \(r\), so its outcome entropy is always 1 bit. That cannot be the right answer. Someone who prepared pure plus has no uncertainty about the state at all.
An X measurement does better. Multiply \(\rho_r\) by \((1,1)^T/\sqrt2\): both entries are scaled by \((1+r)/2\). Multiply by \((1,-1)^T/\sqrt2\): the factor is \((1-r)/2\). So plus and minus are eigenvectors of \(\rho_r\), with eigenvalues \((1\pm r)/2\), and these are also the X-measurement probabilities. At \(r=1\) they are 1 and 0, with no uncertainty, as they should be.
Does every density operator have such a basis? A density operator is Hermitian, so by the linear-algebra fact we used for discrimination it has an orthonormal basis of eigenvectors, \(\rho\ket j=\lambda_j\ket j\). Each eigenvalue is a test probability, \(\lambda_j=\bra j\rho\ket j\geq0\), and their sum is the trace, 1. So
\[\rho=\sum_j\lambda_j\ket j\bra j.\]
The state behaves like an ensemble of orthogonal states \(\ket j\) with probabilities \(\lambda_j\). A measurement in this basis has outcome probabilities \(\lambda_j\). Any other basis has outcome probabilities that are averages of the eigenvalues, and averaging probabilities never lowers entropy. We quote this last fact without proof. So the eigenbasis gives the least uncertain complete measurement, and its probabilities, the eigenvalues, belong to the state itself rather than to a choice of readout.
This is the von Neumann entropy. In the second form, the logarithm of a matrix means taking the logarithm of each eigenvalue in the eigenbasis. A zero eigenvalue contributes zero, as in the classical case.
Check the cases we know. A pure state has eigenvalues 1 and 0, so \(S=0\). The state \(I/2\) has eigenvalues \(1/2,1/2\), so \(S=1\). For \(\rho_r\) with \(r=1/2\), the eigenvalues are \(3/4\) and \(1/4\), so \(S=h_2(1/4)\approx0.8113\), although every Z measurement on it is fair.
So \(S\) is not the entropy of every measurement. A Z measurement on pure plus gives a fair record, 1 bit of outcome entropy, while \(S=0\). The outcome entropy describes the record of one chosen measurement. The von Neumann entropy describes the state.
Problems
Problem 11.1. What is the entropy of diag(3/4,1/4)?
Worked solution
It is −(3/4)log₂(3/4)−(1/4)log₂(1/4)≈0.811 bits.
11.2 Correlation Measures
Now apply the entropy to the Bell pair: the whole has entropy 0, and each half has entropy 1. How should we compare the uncertainty of a whole with the uncertainty of its parts? Classical information theory compared them through mutual information and conditional entropy. We rebuild those from joint and marginal entropies, carry them over to quantum states, and compute them for three pairs that look identical locally. The conditional entropy of the Bell pair turns out negative, and we ask what that can mean.
Classically, mutual information was \(I(X:Y)=H(X)-H(X\mid Y)\). Can we write it using only entropies of joint and marginal distributions, which have quantum versions? From \(p(x,y)=p(y)\,p(x\mid y)\) we get \(-\log_2p(x,y)=-\log_2p(y)-\log_2p(x\mid y)\). Averaging gives \(H(XY)=H(Y)+H(X\mid Y)\). So
\[I(X:Y)=H(X)+H(Y)-H(XY).\]
For a joint quantum state \(\rho_{AB}\), write \(S(A)\) for the entropy of the reduced state \(\rho_A\), \(S(B)\) for that of \(\rho_B\), and \(S(AB)\) for that of \(\rho_{AB}\). The quantum mutual information is
\[I(A:B)=S(A)+S(B)-S(AB).\]
Compute it for three pairs whose halves are each \(I/2\), so that \(S(A)=S(B)=1\) in all three. The Bell pair is pure, with one eigenvalue 1, so \(S(AB)=0\) and \(I=2\). The equal mixture of 00 and 11 has two eigenvalues \(1/2\), so \(S(AB)=1\) and \(I=1\). Two independent fair qubits have joint state \(I/4\), with four eigenvalues \(1/4\), so \(S(AB)=2\) and \(I=0\).
Classically, two bits can share at most one bit of mutual information, since \(I\leq H(X)\leq1\). The Bell pair has two. That does not mean measuring one half yields two classical bits; any measurement of a qubit gives at most one. It measures correlation that only certain protocols can use. Dense coding is one of them: one qubit plus a shared Bell pair carried two bits.
The same identity suggests a conditional entropy, \(S(A\mid B)=S(AB)-S(B)\). Classically, \(H(X\mid Y)\) is an average of entropies and can never be negative: a pair is at least as uncertain as one of its parts. What happens quantumly?
Equal local entropies, different joint correlations
Joint state
\(S(A)\)
\(S(B)\)
\(S(AB)\)
\(I(A:B)\)
\(S(A\mid B)\)
Bell pair
1
1
0
2
−1
Equal mixture of 00 and 11
1
1
1
1
0
Independent fair qubits
1
1
2
0
1
For the Bell pair, \(S(A\mid B)=0-1=-1\). The whole is less uncertain than its part. For the classically correlated mixture it is 0: knowing B's bit leaves no uncertainty about A's. For independent qubits it is 1. A negative value is a signature of entanglement in a pure joint state, and it has no classical counterpart. Conditioning on a recorded classical bit always leaves a nonnegative entropy.
Can a negative number of bits mean anything? It can, in a resource task. Suppose Alice and Bob share many independent copies of \(\rho_{AB}\), and Alice wants to hand her parts to Bob, so that he ends up holding the whole state. They may use classical communication and shared ebits. This task is called state merging. We quote a theorem of Horodecki, Oppenheim and Winter (2005): in the limit of many copies and vanishing error, the net number of ebits needed per copy is \(S(A\mid B)\). A negative value means the merge leaves them with ebits to spare. For the Bell pair, Bob already holds a purification of Alice's half, so merging gains one ebit per copy instead of costing any.
Problems
Problem 11.2. Why is the Bell pair’s joint entropy zero?
Worked solution
Its joint density matrix is a rank-one projector with one eigenvalue one. The mixed local spectra do not determine the joint spectrum.
11.3 Information Inequalities
Entropy measures one state. How should we measure how different two states are, in a way that respects information? We find the classical answer in the cost of using the wrong code, and carry it over to the relative entropy of two density operators. A theorem that channels cannot increase relative entropy then gives, in three steps, the Holevo bound on information extracted from an encoding, which we evaluate for zero and plus.
Suppose Bob designs a code for a source with distribution \(q\), but the source actually follows \(p\). How many extra bits does he pay? His codeword for \(x\) has length about \(-\log_2q(x)\). The average is \(-\sum_xp(x)\log_2q(x)\), while the best code would average \(H(p)\). The excess is
\[D(p\|q)=\sum_xp(x)\log_2\frac{p(x)}{q(x)}.\]
This is the relative entropy of \(p\) with respect to \(q\). It is zero only when \(p=q\). It is infinite if some \(x\) has \(p(x)>0\) but \(q(x)=0\), since Bob's code has no word for a message that actually occurs.
The quantum version replaces the distributions by density operators:
When both are diagonal in one basis, it reduces to the classical formula for their eigenvalues. It is finite when the support of \(\rho\), the span of its eigenvectors with nonzero eigenvalues, lies inside the support of \(\sigma\).
Take \(\rho=\ket0\bra0\) and \(\sigma=I/2\). Then \(D(\rho\|\sigma)=0-\log_2\tfrac12=1\). Reverse them: now \(\sigma\) assigns probability zero to \(\ket1\), which the true state \(I/2\) contains, so the relative entropy is infinite. It is not symmetric, so it is not a distance in the geometric sense. It measures how badly one state imitates another. One useful identity shows the link to correlation: \(I(A:B)=D(\rho_{AB}\|\rho_A\otimes\rho_B)\), how far a joint state is from the product of its parts.
Earlier we argued that no channel makes two states easier to tell apart. Does relative entropy obey the same rule? We quote the theorem that it does (Lindblad, 1975, and Uhlmann, 1977): for every channel \(\mathcal E\),
The examples here do not prove it. What it gives us is a general limit on reading an encoding. Alice encodes label \(x\), with probability \(p_x\), in \(\rho_x\). Put the label and the system into one joint state, \(\rho_{XQ}=\sum_xp_x\ket x\bra x\otimes\rho_x\). Three steps follow.
First, compute its mutual information. Its eigenvalues are \(p_x\) times the eigenvalues of \(\rho_x\), so \(S(XQ)=H(X)+\sum_xp_xS(\rho_x)\). The reduced states are the label distribution and the average \(\bar\rho=\sum_xp_x\rho_x\). Hence
\[I(X:Q)=S(\bar\rho)-\sum_xp_xS(\rho_x)=\chi.\]
Second, Bob's measurement is a channel on \(Q\) whose output is his classical record \(Y\). Third, mutual information is a relative entropy to the product of the parts, and a channel on \(Q\) maps that product to a product. By the theorem, \(I(X:Y)\leq I(X:Q)\). So every measurement satisfies the Holevo bound \(I(X:Y)\leq\chi\), and \(\chi\) is called the Holevo information of the encoding. For a qubit, \(\chi\leq S(\bar\rho)\leq1\), which is the one-bit ceiling quoted when we first scored encodings in bits.
Evaluate it for zero and plus with equal priors. The average matrix is \(\bar\rho=\bigl(\begin{smallmatrix}3/4&1/4\\1/4&1/4\end{smallmatrix}\bigr)\). Its trace is 1 and its determinant is \(1/8\), so its eigenvalues solve \(\lambda^2-\lambda+1/8=0\) and are \((1\pm1/\sqrt2)/2\). Their entropy is about 0.6009 bits. Both component states are pure and contribute nothing, so \(\chi\approx0.601\).
The best single-qubit measurement extracts about 0.399 bits, below the bound. So for this encoding the bound is not reached one qubit at a time. Neither number is Alice's 1 bit of label entropy, because her two preparations overlap. Whether the gap to 0.601 can be closed by measuring many qubits together is a question about long block codes.
Entropy now measures the uncertainty of a state. Does it also count something physical, such as the number of qubits a quantum source really needs?
Problem 11.3. Does χ=0.601 mean each trial returns a fractional bit string?
Worked solution
No. It bounds average mutual information between the input label and the recorded outcome. Individual outcomes are ordinary labels; information is a statistical quantity.
Transfer problem. Fully dephase Alice’s half of a Bell pair. Calculate the changes in joint entropy, quantum mutual information, and conditional entropy.
Worked solution
The output is the equal mixture of 00 and 11. Local entropies stay at one each, while joint entropy rises from zero to one. Mutual information falls from two to one and conditional entropy rises from −1 to zero. The preserved Z correlations do not preserve all quantum correlations.
Summary
The von Neumann entropy \(S(\rho) = -\mathrm{Tr}(\rho\log_2\rho) = -\sum_i \lambda_i\log_2\lambda_i\) equals zero for pure states and reaches \(\log_2 d\) for the maximally mixed state \(I/d\).
Unlike classical conditional entropy, quantum conditional entropy \(S(A|B) = S(AB) - S(B)\) can be negative when \(AB\) is entangled, while quantum mutual information \(I(A:B) = S(A)+S(B)-S(AB)\) can reaches \(2\log_2 d\).
Strong subadditivity \(S(ABC) + S(B) \le S(AB) + S(BC)\) and monotonicity of relative entropy \(D(\rho\|\sigma)\) under quantum channels anchor all fundamental quantum coding bounds.
Questions and worked problems
These exercises draw on the whole chapter. Attempt each question before opening a hint, explain your reasoning, then compare it with the worked solution.
Find the von Neumann entropy of \(\rho=\frac12\begin{pmatrix}1&r\\r&1\end{pmatrix}\) for \(r=0.6\), and compare it with the Shannon entropy of its computational-basis outcomes.
Hint 1
The eigenvalues are \((1\pm r)/2\).
Hint 2
\(h_2(0.2)\approx0.722\).
Worked solution
The eigenvalues are \(0.8\) and \(0.2\), so \(S(\rho)=h_2(0.2)\approx0.722\) bits. A computational-basis measurement gives fair outcomes, with Shannon entropy \(1\) bit. The measurement record is more uncertain than the state, because the computational basis is not the eigenbasis; measuring in the \(\pm\) basis would give outcome entropy \(0.722\), matching \(S\).
For the pure state \(\cos\frac\pi8\ket{00}+\sin\frac\pi8\ket{11}\), find \(S(A)\), \(S(B)\), \(S(AB)\), \(I(A:B)\), and \(S(A\mid B)\).
Hint 1
The reduced state of \(A\) is \(\mathrm{diag}(\cos^2\frac\pi8,\sin^2\frac\pi8)\approx\mathrm{diag}(0.854,0.146)\).
Hint 2
A pure joint state has \(S(AB)=0\).
Worked solution
\(S(A)=S(B)=h_2(0.854)\approx0.601\) bits, and \(S(AB)=0\) because the joint state is pure. So \(I(A:B)\approx1.202\) bits and \(S(A\mid B)\approx-0.601\). This partially entangled state sits between a product (\(I=0\), \(S(A\mid B)=0\)) and a Bell pair (\(I=2\), \(S(A\mid B)=-1\)). For pure joint states, \(S(A)\) measures the entanglement.
Show that \(D(\rho\|I/2)=1-S(\rho)\) for any qubit state \(\rho\), and evaluate it for \(\ket0\bra0\), \(\mathrm{diag}(\frac34,\frac14)\), and \(I/2\).
Hint 1
\(\log_2(I/2)=-I\).
Hint 2
\(\mathrm{tr}(\rho\log_2\rho)=-S(\rho)\).
Worked solution
Since \(\log_2(I/2)=-I\), \(D(\rho\|I/2)=\mathrm{tr}(\rho\log_2\rho)+\mathrm{tr}\rho=1-S(\rho)\). For \(\ket0\bra0\) it is \(1\); for \(\mathrm{diag}(\frac34,\frac14)\), \(1-0.811\approx0.189\); for \(I/2\), \(0\). Relative entropy to the maximally mixed state measures how far a state is from complete uncertainty, in bits.
Two pure states with overlap magnitude \(s\) are used with equal priors. (a) Show that the average state \(\bar\rho\) has eigenvalues \((1\pm s)/2\), so the Holevo quantity is \(\chi=h_2\big(\frac{1+s}{2}\big)\). (b) Evaluate for \(s=0\), \(s=1/\sqrt2\), and \(s=1\). (c) Check part (a) for \(\ket+\) and \(\ket{y_+}\).
Hint 1
\(\bar\rho\) has trace \(1\) and determinant \(\frac14(1-s^2)\).
Hint 2
Use the decomposition \(\ket\phi=s\ket\psi+\sqrt{1-s^2}\ket{\psi_\perp}\) after fixing the phase.
Worked solution
(a) In the basis \(\ket\psi,\ket{\psi_\perp}\), \(\bar\rho=\frac12\begin{pmatrix}1+s^2&s\sqrt{1-s^2}\\s\sqrt{1-s^2}&1-s^2\end{pmatrix}\), with trace \(1\) and determinant \(\frac14(1-s^2)\). The eigenvalues solve \(\lambda^2-\lambda+\frac14(1-s^2)=0\), giving \((1\pm s)/2\). Pure components contribute no entropy, so \(\chi=h_2(\frac{1+s}{2})\). (b) \(s=0\): \(1\) bit; \(s=1/\sqrt2\): \(h_2(0.854)\approx0.601\); \(s=1\): \(0\). (c) \(\frac12(\ket+\bra++\ket{y_+}\bra{y_+})=\frac14\begin{pmatrix}2&1-i\\1+i&2\end{pmatrix}\) has eigenvalues \(\frac14(2\pm|1-i|)=\frac{2\pm\sqrt2}{4}\), as predicted with \(s=1/\sqrt2\).
A noisy Bell pair is \(\rho=(1-p)\ket{\Phi^+}\bra{\Phi^+}+pI/4\). (a) Find its eigenvalues. (b) Compute \(S(AB)\), \(I(A:B)\), and \(S(A\mid B)\) for \(p=0.2\) and \(p=0.5\). (c) Interpret the sign of the conditional entropy.
Hint 1
\(\ket{\Phi^+}\) has eigenvalue \(1-p+\frac p4\); the three orthogonal Bell states each have \(\frac p4\).
Hint 2
Each marginal is \(I/2\).
Worked solution
(a) \(1-\frac{3p}{4}\) once and \(\frac p4\) three times. (b) Each marginal is \(I/2\), so \(S(A)=S(B)=1\). For \(p=0.2\): eigenvalues \(0.85,0.05,0.05,0.05\), \(S(AB)\approx0.848\), \(I\approx1.152\), \(S(A\mid B)\approx-0.152\). For \(p=0.5\): eigenvalues \(0.625\) and three of \(0.125\), \(S(AB)\approx1.549\), \(I\approx0.451\), \(S(A\mid B)\approx+0.549\). (c) At \(p=0.2\) the conditional entropy is still negative, a signature of entanglement usable in state merging; at \(p=0.5\) it is positive. Noise raises the joint entropy above the marginal entropy and removes this quantum advantage (although a state with positive \(S(A\mid B)\) can still be entangled).
For the pure state \(\frac{1}{\sqrt3}(\ket{00}+\ket{01}+\ket{11})\), find both reduced density matrices, show that they have the same eigenvalues, and compute \(S(A)=S(B)\).
Hint 1
\(\rho_A=\begin{pmatrix}2/3&1/3\\1/3&1/3\end{pmatrix}\); compute \(\rho_B\) the same way, pairing equal Alice labels.
Hint 2
Both have trace \(1\) and determinant \(\frac19\).
Worked solution
\(\rho_A=\begin{pmatrix}\frac23&\frac13\\\frac13&\frac13\end{pmatrix}\) (the partial trace). For Bob, \((\rho_B)_{00}=|\alpha_{00}|^2+|\alpha_{10}|^2=\frac13\), \((\rho_B)_{11}=\frac23\), and \((\rho_B)_{01}=\alpha_{00}\alpha_{01}^*+\alpha_{10}\alpha_{11}^*=\frac13\). Both have trace \(1\) and determinant \(\frac29-\frac19=\frac19\), so both have eigenvalues \(\frac12\big(1\pm\sqrt{5}/3\big)\approx0.873,0.127\), and \(S(A)=S(B)\approx0.550\) bits. For every pure joint state the two parts have equal entropy, even when their matrices look different.
Compute \(D(\ket0\bra0\,\|\,\rho_p)\) and \(D(\rho_p\,\|\,\ket0\bra0)\) for \(\rho_p=\mathrm{diag}(1-p,p)\) with \(p=\frac14\). Interpret the asymmetry in terms of testing the hypothesis “the state is \(\rho_p\)” against “the state is \(\ket0\)”.
Hint 1
\(D(\ket0\bra0\|\rho_p)=-\log_2(1-p)\).
Hint 2
The support of \(\rho_p\) includes \(\ket1\), which \(\ket0\bra0\) excludes.
Worked solution
\(D(\ket0\bra0\|\rho_p)=\mathrm{tr}[\ket0\bra0(\log_2\ket0\bra0-\log_2\rho_p)]=-\log_2\frac34\approx0.415\) bits. The reverse is infinite: \(\rho_p\) has weight on \(\ket1\), which \(\ket0\bra0\) never produces. Operationally, a single outcome \(1\) refutes “\(\ket0\)” with certainty, so that hypothesis can be rejected quickly when \(\rho_p\) is true; but when the true state is \(\ket0\), every outcome is consistent with \(\rho_p\), and only the slow accumulation of zeros, at about \(0.415\) bits per copy, separates them.
A student estimates the entropy of a state by measuring in the computational basis and computing the Shannon entropy of the outcomes. Show with \(\rho=\frac12\begin{pmatrix}1&\frac12\\\frac12&1\end{pmatrix}\) that this overestimates \(S(\rho)\), and state which measurement gives the right value.
Hint 1
The eigenvalues are \(\frac34\) and \(\frac14\).
Hint 2
Mixing probabilities (as the diagonal does) can only increase Shannon entropy.
Worked solution
The computational-basis outcomes are fair, giving \(1\) bit, but \(S(\rho)=h_2(\frac14)\approx0.811\) bits. The outcome entropy of a measurement in any orthonormal basis is at least \(S(\rho)\), with equality in the eigenbasis, here \(\{\ket+,\ket-\}\). Off-diagonal entries represent order that a mismatched measurement scrambles into apparent randomness. A pure state such as \(\ket+\) shows the extreme case: \(S=0\), yet its computational-basis record has \(1\) bit of entropy.
The three trine states \(\ket{\psi_k}\), used with equal priors, have average state \(I/2\), so \(\chi=1\) bit. Compute the mutual information for two measurements: (a) the trine POVM \(E_k=\frac23\ket{\psi_k}\bra{\psi_k}\); (b) the “anti-trine” POVM \(E_k=\frac23\ket{\psi_k^\perp}\bra{\psi_k^\perp}\), where \(\ket{\psi_k^\perp}\) is orthogonal to \(\ket{\psi_k}\). Compare both with \(\chi\).
In both cases the outcomes are uniform, so \(H(Y)=\log_23\).
Worked solution
Both POVMs give uniform outcomes, \(H(Y)=\log_23\approx1.585\). (a) \(H(Y\mid X)=H(\frac23,\frac16,\frac16)\approx1.252\), so \(I\approx0.333\) bits. (b) Each input gives two outcomes with probability \(\frac12\), so \(H(Y\mid X)=1\) and \(I=\log_23-1\approx0.585\) bits, the known optimum for this ensemble. Both fall short of \(\chi=1\). The best measurement for guessing (a) is not the best for information (b), and the Holevo quantity is an upper bound that a single-copy measurement need not reach.
Check data processing for \(\rho=\ket+\bra+\) and \(\sigma=I/2\) under the phase-flip channel with probability \(p\). (a) Compute \(D(\rho\|\sigma)\). (b) Show that after the channel, \(D=1-h_2(p)\). (c) Confirm that the value never exceeds the original and explain the cases \(p=\frac12\) and \(p=1\).
Hint 1
\(\sigma\) is unchanged by the channel.
Hint 2
The output \(\rho'\) has eigenvalues \(1-p\) and \(p\).
Worked solution
(a) \(D(\ket+\bra+\|I/2)=1-S(\ket+\bra+)=1\). (b) The channel leaves \(I/2\) unchanged and maps \(\ket+\bra+\) to \((1-p)\ket+\bra++p\ket-\bra-\), with eigenvalues \(1-p,p\); by the earlier identity, \(D=1-h_2(p)\). (c) \(h_2(p)\ge0\), so \(D\le1\), consistent with the data-processing inequality. At \(p=\frac12\) the output is \(I/2\) and \(D=0\): the states are identical. At \(p=1\) the output is the pure \(\ket-\), and \(D=1\) again, because a known flip loses no distinguishability.