Working with Python and RabbitMQ
I recently installed RabbitMQ to handle some message queuing needs at Readernaut and thought I’d share how everything came together. If you’d like to learn more about RabbitMQ please read the excellent Rabbits and Warrens.
To use RabbitMQ with python you need amqplib because Rabbit uses the AMQP standard. To make amqplib a little easier to use I needed a simple script that did three things:
- Easy way to connect to RabbitMQ.
- Easy way to pull stuff out of the queue.
- Easy way to throw stuff into the queue.
There’s a project called Carrot which handles all this and much more but it’s a little too complex for what I’m doing. All I needed was something very small, straightforward, and EASY.
After staring at Carrot for a few days I decided to distill down what I needed into a single script I’m calling Flopsy (this is what happens when you’re coding and watching a movie about Beatrix Potter, right?). So here we go:
Step 1: Set global variables:
AMQP_SERVER = 'localhost'
AMQP_PORT = 5672
AMQP_USER = 'guest'
AMQP_PASSWORD = 'guest'
AMQP_VHOST = '/'
Step 2: Create a consumer (a script that consumes messages from the queue):
>>> from flopsy import Connection, Consumer
>>> consumer = Consumer(connection=Connection())
>>> consumer.declare(queue='books', exchange='readernaut', routing_key='importer', auto_delete=False)
>>> def message_callback(message):
... print 'Recieved: ' + message.body
... consumer.channel.basic_ack(message.delivery_tag)
>>>
>>> consumer.register(message_callback)
>>> consumer.wait()
Step 3: Create a publisher (a script that publishes messages to the queue):
>>> from flopsy import Connection, Publisher
>>> publisher = Publisher(connection=Connection(), exchange='readernaut', routing_key='importer')
>>> publisher.publish('Test message!')
>>> publisher.close()
And there we go! Flopsy weighs in at under 80 lines of code and it can be found on Github. Most of the above will make a lot more sense after reading the Rabbits and Warrens article and investigating RabbitMQ.
Update: I just realized, this isn’t at all Django specific so my title is a little misleading (changing). That said, you could most certainly use Flopsy with straight up Python :)
Comments
kevin http://www.montylounge.com
Again, love the insights into readernaut. We should do another callcast soon! Thanks for the post.
Nathan Borror http://playgroundblues.com
I should point out there is another python AMQP library on the market besides amqplib, txAMQP which uses Twisted. I’ve heard good things but I didn’t want to install yet another library, Twisted, to use it at the time.
Nathan Borror http://playgroundblues.com
@kevin I’m going to try to do more :) We totally should, I really enjoyed the last callcast.
majek http://popcnt.org
Maybe you’re looking for that: http://github.com/bwhitman/py-amqplib-wrapper/tree/master
Yoan http://yoan.dosimple.ch/blog/
I wrote a simple example app using both amqplib and txamqp: http://bitbucket.org/greut/up/
amqplib fits perfectly for the producer side, especially if it’s inside a web application but as a client I’d prefer to use txAMQP. It feels more robust.
Jeremy Dunck
Does channel block for some timeout, or is this a fast loop, burning CPU?
Nathan Borror http://playgroundblues.com
@Jeremy It doesn’t seem to burn CPU when it’s not processing messages. CPU usage drops to 0 in top. Let me know if you experience otherwise.
Grégoire Cachet
How do you run your consumer in practice ? Do you have a daemon (like an /etc/init.d/ script) running on the server ? Any monitoring in case the process crashes ?
Nathan Borror http://playgroundblues.com
@Grégoire I just have a daemon running. No monitoring other than me eavesdropping now and then.
I have some other monitoring needs and I should probably install a proper monitoring tool, Nagios is pretty popular I guess.
Ask Solem http://github.com/ask/carrot
Hi Nathan!
I adopted your register scheme in carrot. I know documentation for carrot is scarce, but it works pretty much the same way.
I updated some documentation as well: http://ask.github.com/carrot/reference/carrot.messaging.html
The intro will probably be moved into its own document soon.
(btw, you probably should generate a unique consumer tag, since right now I don’t think it’s possible to use two consumers simultaneously without setting the consumer tag explicitly.)
Ask Solem http://github.com/ask/carrot
Oh, btw. I’m also working on a branch for carrot that supports different backends. For one, it has a python Queue module backend for use in unit tests. http://github.com/ask/carrot/tree/backends
Vitaly Babiy http://howsthe.com
Great post just started to play with rabbitmq and django using celery.