John R. Pierce (2)

(Original Review, 1980-12-05)

Final answer to question, "How many joules to send a bit?"

The unit of information is determined by the choice of the arbitrary scale factor K in Shannon's entropy formula:

{ s(Q|X) = -K SUM(p*ln(p)) }

If K is made equal to 1/ln(2), then S is said to be measured in "bits" of information. A common thermodynamic choice for K is kN, where N is the number of molecules in the system considered and k is 1.38e-23 joule per degree Kelvin, Boltzmann's constant. With that choice, the entropy of statistical mechanics is expressed in joules per degree. The simplest thermodynamic system to which we can apply Shannon's equation is a single molecule that has an equal probability of being in either of two states, for example, an elementary magnet. In this case, p=.5 for both states and thus S=+k ln(2). The removal of that much uncertainty corresponds to one bit of information. Therefore, a bit is equal to k ln(2), or approximately 1e-23 joule per degree K. This is an important figure, the smallest thermodynamic entropy change that can be associated with a measurement yielding one bit of information.

The amount of energy needed to transmit a bit of information when limited by thermal noise of temperature T is:

E = kT ln 2 (Joules/bit)

This is derived from Shannon's initial work (1) on the capacity of a communications channel in a lucid fashion by Pierce (2), although it is not obvious that he was the first to derive it. This limit is the same as the amount of energy needed to store or read a bit of information in a computer, which Landauer derived (3) from entropy considerations without the use of Shannon's theorems. Pierce's book is reasonably readable. On page 192 he derives the energy per bit formula (Eq. 10.6), and on page 200 he describes a Maxwell Demon engine generating kT ln 2 of energy from a single molecule and showing that the Demon had to use that amount of energy to "read" the position of the molecule. Then on page 177 Pierce points out that one way of approaching this ideal signalling rate is to concentrate the signal power in a single, short, powerful pulse, and send this pulse in one of many possible time positions, each of which represents a different symbol. This is essentially the concept behind the patent (4) which led me to ask the original question. My thanks to those who helped with their replies.

REFERENCES

1. C. E. Shannon, "A Mathematical Theory of Communication", Bell
System Tech. J., Vol. 27, No. 3, 379-423 and No. 4, 623-656
(1948); re-printed in: C. E. Shannon and W. Weaver, "The
Mathematical Theory of Communication", University of Illinois
Press, Urbana, Illinois (1949).
2. J. R. Pierce, "Symbols, Signals and Noise", Harper, NY (1961)
3. R. Landauer, "Irreversibility and Heat Generation in the
Computing Process," IBM J. Res. & Dev., Vol. 5, 183 (1961).
4. R. L. Forward, "High Power Pulse Time Modulation
Communication System with Explosive Power Amplifier Means",
U. S. Patent 3,390,334 (25 June 1968).

[2018 EDIT: This review was written at the time as I was running my own personal BBS server. Much of the language of this and other reviews written in 1980 reflect a very particular kind of language: what I call now in retrospect a “BBS language”.]… (mer)