First, consider all the possible “worlds” that agree with the constraints but are equally likely based on indifference. In each such world a particular frequency distribution is “realized”. Then, examine which distributions are going to come up most often if you keep drawing randomly from the bag of worlds; this of course is a basic problem solved by the multinomial distribution.

If my remarks are not clear, think about his broken windows example. N windows have been broken into an integer number of pieces and all that we know is the average number of pieces (seems like a rather strange situation to me, but who am I to criticize textbook examples). If we assume some upper integer limit on the number of pieces per window, we can easily imagine a concrete world in which the “first” window was broken into p_1 pieces, the “second” window into p_2 pieces and so on until p_n. Now, if we enumerate all the possible worlds (and there’s a finite number of them, based on our assumptions about the number of windows and pieces), some of them will agree with the average number of pieces constraint, most will not. Then, we conceptually put these matching worlds into a “bag”, sample from this bag and examine the relative frequency of each number of pieces in each drawn world. What the maxent principle says is that an overwhelming number of draws from the bag will have the relative frequencies very close to most other draws, and that the most frequent frequency distribution can be calculated by maximizing entropy (why this correspondence holds is not explained very well in the book, I find).

Why are we willing to accept the maxent frequency distribution, which is after all based on a thought up generative sampling model? So far, the only good answer I understand is that other distributions would have to be also based on thought up generative sampling models – ones that are even more ridiculous (arbitrary) than the maxent one. Sometimes I wonder if it is the only answer.

As for CS people not knowing about Jaynes: I think it is “Jaynes’s fault” – he assumes that his reader has a working knowledge of calculus (and often also “orthodox” statistics and history) to follow his reasoning. This may be true for physicists, but certainly isn’t true for CS students. The funny thing is Jaynes has inspired me to improve my maths education. There’s something magnetic in the way he explains stuff and deals with critics.

Good to hear from you; an interesting response! I wasn’t really expecting much activity on this thread, since this blog is centered around computer science, and if you are a computer scientist, it is quite unlikely that you have come by Jaynes’ book… I myself came by it by pure coincidence.

But you have! Great; and thanks for the reference to Cox’ original work, I will definitely read that when I get the time and opportunity

Regarding your problem with MaxEnt, I think Jaynes gives a really satisfactory explanation on this. One can think of the “frequentist thing” as a degenerate special case of proper Bayesian reasoning in the case where the prior information says nothing, i.e., when we use the principle of insufficient reasoning. Now, in the case of MaxEnt we have actual prior information, say in the form of average values that the solution must satisfy. Frequentist theory (conventional statistics) cannot make use of this prior information, but MaxEnt can: intuitively, one gets a prior which is as uniform as possible while respecting the given constraints. So the frequency correspondence is not a bad thing it is good; it is what makes MaxEnt as noncommittal as possible.

If you read section 11.8, page 365 in the 2003 edition, there is an interesting exposition on ‘frequency correspondence’: “…the probability distribution which maximises entropy is numerically identitical with the frequency distribution which can be realized in the greatest number of ways (which is vastly greater than it’s competitors).

If you haven’t read Cox’s original exposition, it is worth a trip to library. His little “algebra of probable inference” book is short and neat, maybe even easier to follow than Jaynes.

Anyway, I see one problem with the explanatory approach taken by Cox/Jaynes (which I started transcribing for the average Joe non-mathematician in my blog). Their discussion of the probability rules and their uniqueness in context of logical propositions is nice and dandy, but it leaves a lingering question of where these proposition sets are supposed to come from in practice. When Jaynes explains his maximum entropy principle, it looks very much like he is doing a “frequentist” thing after all to arrive at the atomic probabilities. Basically he’s counting possibilities and weighing more complex propositions by counting the number of atomic propositions that imply them and weighing this against other complex propositions.