• Probability is about information. Forget notions of “random” experiments, e.g., the outcome of throwing a die is about the laws of physics – there is no magical randomness built into the die causing it to come up with each side equally often “in the long run” (what ever that means). If we know the laws of physics and have enough information about the experiment we might assign other probabilities to the outcomes. This means that if new information is obtained the probability changes!

• In the name of science, we shall not make any unwarranted conclusions, and we promise to use all the information at hand, not deliberately discarding information (for what ever reason).

• We use the laws of probability theory.

The first point is where the problem lies. In fact the probabilities are just a formalization of the information we have. This is why we always condition the probability on some information e.g., P(A | I) is the probability of proposition A given the information I.

Let’s solve Monty Hall using the principles presented in Jaynes’ book. Notice that these techniques are completely general and can solve many different problems of this nature; the key in Monty Hall is not to make any unjustified conclusions, but simply consider all the information at hand!

Lets name the doors A, B and C, and let us assume that we initially selected door A (the argument applies for any other initial choice as well). Let us write I for the information we have initially (before opening any doors) – this includes the rules of the game. More precisely I is the conjunction of statements:

* We have chosen door A.

* The host knows where the car is, and may only open a door that does not contain the car.

* The host may not open the door we selected (A) (even if it does not contain the car).

We formalize the information I in probability equations below. Let us consider three mutually exclusive and exhaustive propositions

A = the car is behind door A;

B = the car is behind door B;

C = the car is behind door C

By the principle of indifference, given only the initial information, I, we have

P(A | I) = P(B | I) = P(C | I) = 1/3

Notice also that so far intuition is completely clear.

Let us now assume that the host opens door C. Let H be the statement:

H = the host opened door C

(the argument is similar for any other choice). The important point here is that we are given additional information beyond our prior information, I. Now let us first derive the correct solution, we then analyze where peoples intuition usually goes wrong.

By the rules of probability theory (specifically Bayes theorem) we have

P(A | HI) = P(A|I) P(H | AI)/P(H|I)

The quantity P(A|HI) is what we are interested in: the probability of the car being behind door A given all the information available to us.

We know P(A|I) is 1/3 so we can focus on P(H|AI) and P(H|I) – notice these are both probabilities about the host’s actions – probabilities that are very relevant to the problem (by the equation above), but also probabilities that most people would not consider to analyse (because probability is taught the way it is)! The former is the probability of the host opening door C given the car is behind door A; the latter is the a-priori probability of the host opening door C given only the information I. The former, P(H|AI), is the most simple: given we know that the car is behind door A what is the probability of the host opening door C. Since we have no information about which door the host opens when the car is behind the door we selected, the principle of indifference applies and we have:

P(H|AI) = P(not H|AI) = 1/2

Now consider P(H|I): the a-priori probability of the host opening door C. Since propositions A, B and C are mutually exclusive and exhaustive, the rules of probability theory imply

P(H | I) = P(H | AI)P(A|I) + P(H | BI)P(B | I) + P(H | CI)P(C | I)

We know that P(H | CI) = 0 because by information I, the host may not open door C if the car is behind door C. We already figured out P(H | AI) and P(A|I) so the first term is 1/2*1/3 = 1/6. Also we know P(B | I) = 1/3. Now we can focus on P(H | BI): the probability of host opening door C when the car is behind door B and we have chosen door A. But here the rules of the game are clear: the host must open door C in this case since he may not open A and may not reveal the car. Hence P(H | BI) = 1. We get:

P(H | I) = 1/6 + 1*1/3 = 3/6 = 1/2.

We now have all the quantities needed to calculate P(A | HI):

P(A | HI) = 1/3 * [(1/2)/(1/2)] = 1/3. Hence P(B | HI) = 2/3 and we should switch doors!

The technique is completely mechanical. I have not used any clever arguments or mathematical ingenuity: only the rules of the game, the principle of indifference and the laws of probability theory. All calculations could be performed by a machine with these inputs.

So where does our intuition go wrong? Two places, I believe. First because of schooling our intuition tells us that somehow we can only reason about “random” events. Instead we should focus on information e.g., ‘the car is placed “at random” behind a door’ versus “we are given the information that there is a car behind one door and sheep behind the two others.” Once we free ourselves to consider probabilities as expressing the information we have available, it is natural that the probabilities change when we obtain more information. This is crucial. The second, I believe, is a consequence of the first: most people seem to identify the following two statements in the Monty Hall problem:

* the host opens door C

* the car is not behind door C

But these two pieces of information are not equivalent when we know the rules of the game (i.e. given our prior information). Considering them equal would violate our commitment to science, to be objective and to consider all the information at hand

To illustrate let us analyse a variant of Monty Hall where “H” means the latter instead.

Given this alternate piece of information, our first step is still to apply Bayes theorem so:

P(A | HI) = P(A|I) P(H | AI)/P(H|I)

We still have P(A|I) = 1/3. P(H|I) is the probability that the car is not behind door C given our initial information. Since H = not C, and since A, B, C are mutually exclusive and exhaustive, the rules of probability theory imply

P(H | I) = P(not C|I) = P(A or B|I) = P(A|I) + P(B|I) = 2/3

Now P(H | AI) is calculated by classical logic: since A implies not C we have P(H | AI) = 1. Hence:

P(A | HI) = 1/3 * 1/[2/3] = 1/2

Which is the false result that most people’s intuition prefer. This is the correct conclusion, but for a different game than Monty Hall, namely the game without a host where you simply get an additional information about where the car is not.

Notice that we easily analysed both variants of the game using simple probability theory, and in its interpretation as an extension of classical logic, the intuition follows along nicely. The key is to use all the information at hand, not discarding any information that could be of relevance to the problem (i.e. we got the information that host selected C, not that door C had a sheep behind it).

Now these legacy applications are of different ages and have been developed in various technologies, with varying quality and standards-awareness. Now these requirements brings a bunch of problems, and I won’t start complaining too much here. Instead I’ll just focus on one issue: character encodings! One of the aspects of this scenarios is that the applications are using different character-encodings, e.g., ISO-8859-1 and UTF-8. This gives the following requirements.

• Cross page (cross encoding) navigation. For example, our client app often has to navigate from a UTF-8 encoded page to an ISO-8859-1 encoded page, transferring properly encoded parameters from the first page to the second.
• Localization. The client app is available in various languages; together with the requirement that it must exist in UTF-8 and ISO-8859-1 encoded pages, this means that the actual JavaScript being sent on the wire must be available in different languages and encodings, based on the browsers settings.

You’d expect these things to be pretty basic stuff for an HTTP based app; something that would be easy in a version 7 browser.

Cross page (cross encoding) navigation

For various reasons irrelevant here, our client app navigates to an application by dynamically creating a form with method “GET”, and plugging in the URL and the parameters for the receiving application, i.e.,

<form method="GET" action="URL">
<input type="hidden" name="pname" value="pvalue" >
…
</form>

There is an implicit attribute called ‘accept-charset’ which according to HTML 4.01 satisfies:

“This attribute specifies the list of character encodings for input data that is accepted by the server processing this form. The value is a space- and/or comma-delimited list of charset values. The client must interpret this list as an exclusive-or list, i.e., the server is able to accept any single character encoding per entity received. The default value for this attribute is the reserved string “UNKNOWN”. User agents may interpret this value as the character encoding that was used to transmit the document containing this FORM element.”

The problem with IE7 is that it ignores the accept-charset property entirely on forms. In fact, it instead does the default behaviour for “UNKNOWN”, i.e., “the character encoding that was used to transmit the document containing this form element.” Actually, to be precise it doesn’t even do that either: It turns out the IE7 encodes the form according to the current value of document.charset. Fortunately it is possible to set document.charset using JavaScript, this means that it is possible to control the encoding of form input data by setting document.charset just before the form submits. And this works. Problem solved. Kind of …

Unfortunately, this hack turns out to trigger a somewhat strange and nasty bug: After form submission, IE navigates to the desired page and input data is encoded according to document.charset, which is just what we want. However, if you click the ‘back’-button on the next page, IE navigates back to the original page, but the document.charset property somehow persists which means that the page is not interpreted with the correct encoding. OK, so we should clean up after ourselves: One would think that restoring the document.charset to it’s original value after calling form.submit() would work: it doesn’t.

After some experimentation I found that the following works. To submit a form with input data encoded in a particular encoding:

1. on document load, store the original page encoding
2. on form submit change the value of document.charset to the desired encoding, e.g., “ISO-8859-1″
3. on ‘beforeunload’ restore document.charset to it’s original value.

Script character encodings

Our JavaScript client app sometimes lives in a UTF-8 encoded page, sometimes in an ISO-8859-1 encoded page. The client app is loaded by including a <script> tag that points to a servlet that delivers the JavaScript code (customized for the requesting user and his preferred locale). By default when IE loads script, interprets the bytes it receives as characters in the encoding of the current page. However, it doesn’t tell the server which encoding that is. More specifically, our servlet receives a request for the JavaScript client with these HTTP headers:

Accept : */*
Referer : http://localhost:10045/wps/portal/sn
Accept-Language : da
UA-CPU : x86
Accept-Encoding : gzip, deflate
User-Agent : Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; MS-RTC LM 8; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Connection : Keep-Alive
Cache-Control : no-cache

Furthermore, even if we explicitly tell IE which encoding we have chosen, it will still ignore that and use the page encoding. More specifically, even if we return this HTTP:

Content-Type: application/javascript; charset=UTF-8

IE ignores the Content-Type charset. In other words: if we choose as a default (when no Accept-Charset header is present) to deliver the script in UTF-8 encoded, it breaks for ISO-8859-1 pages, and vice versa. So what to do? IE apparently doesn’t speak HTTP.

It turns out that one should use a script tag with a charset property to tell IE how to interpret the script. Ironically this means that on UTF-8 encoded pages one would need to say <script charset=”ISO-8859-1″>. In other words, it is not the HTTP headers describing the resource that describes its encoding. This must be know in advance by all documents that link to the resource by embedding a charset property. Yet another reason that these are tough times to be RESTful…