Fast and efficient process management

A great tool to run scripts based on AMQP input.

About AMQPipe

AMQPipe is a high performance application the reads messages as fast as it can from a RabbitMQ message queue, runs your scripts or applications to process these messages, and publishes the results back to RabbitMQ. With AMQPipe you make maximum use of your hardware, because AMQPipe runs the scripts in parallel (with a configurable limit).

AMQPipe is your perfect solution for data processing. Are you converting webpages into PDF documents? Do you process incoming emails or pageviews? With AMQPipe you ensure that your hardware is used to the max. The input is loaded from RabbitMQ, sent by AMQPipe to as many of your scripts that can simultaneously run, and the output is published back to RabbitMQ again (where it can then for example be picked up for further processing).


AMQPipe is written in C++ so that it can read messages from RabbitMQ as fast as possible, and so that it can very efficiently manage child processes. If you want an even higher performance, you can choose not to only run scripts, but also to write your own C++ plugins with the very simple C++ API. This will give an even further boost to your performance.


AMQPipe relies on RabbitMQ for loading input as well as publishing results. Perhaps the most important factor in choosing RabbitMQ is flexibility. RabbitMQ's routing options allow you to have full control over the input and output streams. Do you want to filter input, or fork of an output queue? Everything is possible.


AMQPipe can run all sorts of scripts and applications, as long as the executable bit is set. You can for example run PHP, Perl, Python and Bash scripts, or even native applications. All you need is to set the executable bit. And if this is not flexible enough, with the C++ API you can even write native plugins.


To prevent that your scripts run out of control, you can set limits on memory usage and/or runtime. AMQPipe prevents that one of your scripts will consume more resources than is healthy for your system.