Drupal - Why did the Apache directive to handle a clean URL get changed?

There was something seriously broken, but it only affected a handful of cases due to fringe nature of the issue. Essentially, what happens is roughly (%26 == &):

Request: /lost%26found
Rewrite: /index.php?q=lost&found

The issue here is obvious when you see it this way: you end up with two $_GET variables, q = 'lost' and found which isn't set properly. Since the q variable is truncated, you end up with the wrong page or a 404 error.

Drupal 7 fixes this by simply redirecting to index.php and then parsing the original query string manually itself, bypassing Apache and PHP's parsing of it entirely.

Using the rewriting rule used in Drupal 6 there would be a problem when the path contains escaped & or % that are changed back to respectively & and % from the "mod_rewrite" Apache module. The solution would be using the following rewriting rule, but it would require Apache 2.2.8 or higher, when Drupal 7 is compatible with any Apache 2.x versions.

RewriteRule ^(.*)$ index.php?q=$1 [L,QSA,B]

Between being compatible only with Apache 2.2.8 and higher, or being compatible with all the Apache 2.x versions, they preferred to the compatible with all the Apache 2.x versions.

To understand why unescaping & and % is a problem, suppose a third-party module is able to handle a path such as title/12&id=13; the path would be changed to index.php?q=title/12&id=13 instead of index.php?q=title/12%26id=13. The effect is that the module would get title/12 as path, instead of title/12&id=13.