So I got some update tests fail as we upgrade to Symfony 2.1. The failures were coming from
update_parse_xml complaining about malformed XML. I have quickly found a
drupal_http_request in the calling function and dumped the URLs it tried to fetch into a file. The test apparently calls a page provided by the
update_test.module. As the first attempts to fetch these pages with cURL failed miserably, I peeked into
update_test_mock_page to see what XML file it is trying to transfer as a response and found every file name it tried to work with contained
At this point I still was fetching every URL I had at hand in a log file with
for i in `cat /tmp/log1` ; do j=`echo $i | md5sum|cut -c 1-32`; curl -H Accept:text/xml -s $i > /tmp/1/$j; done but I have singled out just
http://localhost/d8/update-test/drupal/8.x as it seemed to be broken enough.
As the mock page function is very short it was easy to figure out why it was #broken#: it needed configuration to be set accordingly, which I have done with a simple
drush ev 'config("update_test.settings")->set("xml_map", array("drupal" => "0"))->save();'. Re-running my cURL command, however, showed well formed XML. Getting the same XML with
drupal_http_request showed XML with some number before and after the XML, however.
So I googled curl raw http response and found that netcat will do nicely. So I dumped the request
drupal_http_request runs and fed it into netcat. Reminder: HTTP requests have two line breaks at their end. Finally I was able to reproduce the brokenness with netcat. This was important because now I had a very fast way to reproduce the problems. Running the update test itself required some three minutes (this would later decrease to below a minute) and running
drupal_http_request via drush was quite slow too.
As I looked into
file_transfer it seemed strange to me that there were no Content-Length header nor a Connection Close. As I added them, things got fixed. But the story doesn't end here -- it felt strange and wrong to fix it this way. Digging further, I knew that
send() in the HTTP Foundation Response has changed (because it caused other bugs in the same issue) and I began to comment out this and that in that function just to see what happens. Somewhere by now I have realized comparing the many responses that the broken replies had a
Transfer-Encoding: Chunked header in them while the unbroken ones had a
Content-Length header. Very strange, as this header didn't appear anywhere in D8. I have found out that the
flush(); PHP command makes a
Transfer-Encoding: Chunked appear if
Content-Length is not sent. I suspect this has more to do with Apache than PHP because the PHP source code doesn't seem to contain anything do this.
Much to my surprise commenting out
sendHeaders worked. So I began to binsearch in that function, commenting out this and that and found to my utter shock that commenting out
header(sprintf('HTTP/%s %s %s', $this->version, $this->statusCode, $this->statusText)); alone was enough. I started replacing the variables in sprintf manually and found that HTTP/1.0 worked but
$this->version was set to 1.1. I have looked up the
Transfer-Encoding header in Wikipedia and it was indeed a HTTP 1.1 mechanism and those mysterious numbers were the chunk sizes. Well, why doesn't
drupal_http_request support this mechanism, then? Another googling, for Transfer-Encoding drupal_http_request finds a comment by Damien Tournoud: "If a server replied with chunked encoding to a HTTP/1.0 request, fix the server.". Ah ha! Here is the fix.
There is extremely little a debugger would have helped here because the PHP codebase didn't quite contain any PHP bugs.
Commenting on this Story is closed.