Two months ago Dries asked for a queue API in his DC DC keynote. I wrote one immediately after the keynote :) and two months and 131 followups later, it's committed. Most help came from Derek Wright (dww) with his Condor experience and James Gilliland (neclimdul). It was an amazing ride, getting this patch right. And its a groundbreaking patch because it prepares for integration with the likes of Amazon SQS -- it's a different thinking to what core did so far, where everything was stored in a database and consistency was taken as granted. With the queue API, all bets are off: there is no guaranteed order, just a best effort to keep a FIFO. The number of items is never guaranteed to be accurate. This allows us scaling to new heights. My example in the issue was media conversion -- if you have a busy site where media is uploaded and needs to be converted, this queue is pretty much a must: upload servers put the name of the files to be converted in queues and conversion servers read them. It does not matter if it does not happen in strict a FIFO order as long as conversion happens in a reasonable time. The number of queue items changes so rapidly that reporting an exact number is pointless, by the time it reaches your browser, it's outdated anyways, so why bother with the exact count? If you have geographically spread servers then this arrangement allows you to upload and convert even if half of the conversion servers can only see some of the upload servers. Of course, this needs database support too, but you can have geographically distributed database slaves and INSERTs can go in a queue too so that when the master reappears, the new media can be loaded there.

Commenting on this Story is closed.