Entropy and Data Processing

3.1 Entropy and relative entropy

3.1.1 Spectra, support, and the reference state

The entropy of a state is \(S(\rho)=-\tr\rho\log_2\rho=-\sum_i\lambda_i\log_2\lambda_i\), with \(0\log0=0\). It measures the spread of the spectrum, not the spread of outcomes in an arbitrary basis. A pure qubit has zero entropy even if an incompatible measurement gives a uniformly random bit. The maximal entropy in dimension \(d\) is \(\log_2d\), reached by \(I/d\).

Relative entropy compares a state with a reference: \(D(\rho\|\sigma)=\tr\rho(\log_2\rho-\log_2\sigma)\) when \(\operatorname{supp}\rho\subseteq\operatorname{supp}\sigma\), and infinity otherwise. It is nonnegative but generally asymmetric and does not obey a triangle inequality. For diagonal states it becomes the classical expression \(\sum_i p_i\log_2(p_i/q_i)\). The support condition is operational: assigning zero probability to an event that actually occurs is a decisive modeling failure.

To evaluate a noncommuting example, diagonalize each matrix separately. The term \(\tr\rho\log\sigma\) depends on how the eigenvectors overlap; subtracting two lists of eigenvalues is not enough. For the special reference \(I/d\), however, \(\log_2(I/d)=-(\log_2d)I\), giving \(D(\rho\|I/d)=\log_2d-S(\rho)\). This identity measures deviation from maximal mixing and will later provide a resource monotone.

3.2 Correlations and conditioning

3.2.1 Negative conditional entropy is consistent

For a bipartite state, define \(I(A:B)=S(A)+S(B)-S(AB)\) and \(S(A|B)=S(AB)-S(B)\). Expanding the logarithm of \(\rho_A\otimes\rho_B\) shows \(I(A:B)=D(\rho_{AB}\|\rho_A\otimes\rho_B)\). Thus mutual information is nonnegative and vanishes precisely for product states. It counts total correlation, including classical correlation; it is not an entanglement measure.

A Bell pair has entropies \(S(A)=S(B)=1\), \(S(AB)=0\), giving \(I(A:B)=2\) and \(S(A|B)=-1\). Dephasing one half yields \((|00\rangle\langle00|+|11\rangle\langle11|)/2\), with joint entropy one. Its mutual information is one and conditional entropy zero. The local states have not changed, but coherent correlations have been lost.

The central theorem is data processing: for a channel \(\mathcal N\), \(D(\mathcal N(\rho)\|\mathcal N(\sigma))\le D(\rho\|\sigma)\). We state the full operator inequality without proof; its noncommuting proof is substantially deeper than the diagonal case. Applying it to a channel on one half proves that local processing cannot increase mutual information. Applying it to \(\rho_{ABC}\) and \(\rho_A\otimes\rho_{BC}\), then tracing out \(C\), gives strong subadditivity:

See Watrous, Chapter 5 for entropy inequalities and their proofs. The derivation of strong subadditivity here depends on the stated data-processing theorem, rather than proving that theorem.

3.3 Many-copy testing

3.3.1 Relative entropy as an error exponent

Return to the asymmetric test in Chapter 1, now with hypotheses \(\rho^{\otimes n}\) and \(\sigma^{\otimes n}\). Fix a tolerated miss probability \(0<\epsilon<1\), independently of \(n\), and minimize the false-acceptance probability \(\beta_{n,\epsilon}\). Quantum Stein’s lemma states that \(-\lim_{n\to\infty}n^{-1}\log_2\beta_{n,\epsilon}=D(\rho\|\sigma)\) for finite-dimensional states with finite relative entropy. Collective measurements are allowed. The Ogawa–Nagaoka strong-converse result completes the quantum Stein theorem. This is a theorem statement, not a proof; the asymptotic testing discussion in Preskill’s notes provides further context.

Why should a logarithm appear? For commuting states, probabilities of independent strings multiply. Their likelihood ratio is a product, and its logarithm is a sum. Under the first hypothesis, the average log likelihood approaches \(\sum_i p_i\log_2(p_i/q_i)\). This gives an intuition for the exponent; controlling error tails and noncommuting measurements needs additional arguments.

For the biased coin and fair coin above, the exponent is approximately 0.278072 bits per sample. That does not assert the exact finite-sample formula \(\beta=2^{-0.278072n}\); subleading terms and the allowed miss probability matter. Nor does it describe equal-prior minimum-error testing, where both errors are penalized symmetrically. Keeping these tasks separate prevents an entropy formula from being applied to the wrong experiment.