UPDATE: Watch the full panel replay here.
Earlier this week, I moderated a panel at the first RabbitMQ Summit in London. Organized by Erlang Solutions and 84codes, the event brought together about 150 users and members of the RabbitMQ community. The sessions were fantastic. I learned more about how RabbitMQ works and best practices in those eight hours than the last year.
The organizers will post the session recordings soon. In the meantime, here are my notes from moderating the panel with four RabbitMQ experts:
- Gavin Roy, VP at AWeber and Author of "RabbitMQ in Depth"
- Karl Nilsson, on the RabbitMQ team at Pivotal
-
Carl Hörberg, CEO of CloudAMQP, the leading hosted RabbitMQ provider
- Daniel Carwin, product manager of RabbitMQ at Pivotal
Use Cases for RabbitMQ
Once upon a time, RabbitMQ was the clear, go-to choice for open source messaging. But with the emergence of Apache Kafka, understanding when to use RabbitMQ has become more nuanced. To help demystify when to use RabbitMQ, I asked the panelist for good and bad use cases for RabbitMQ.
Good use cases for RabbitMQ
- Integrating different systems, especially when some are legacy, different languages, etc.
- Event streams
- When uncertainty is high: you know you need messaging, but not sure what exactly
- Building big, microservices architectures
Bad use cases for RabbitMQ
- Large, opaque files (e.g. videos, images)
- As a data store
- Large, long-running queues (UPDATE: clarifying that this refers to queues that keep a permanent backlog and are never emptied. Large and/or long-running aren't inherently problematic on their own…)
What are common pitfalls when using RabbitMQ?
One of RabbitMQ's strengths is its flexibility. But that can also be a challenge and lead to some misuse. I asked the panel what were the most common pitfalls they see among RabbitMQ users.
Common RabbitMQ pitfalls
- Memory challenges from long queues or too many connections
- Over-reliance on RPC (remote procedure call)
- Turning on all guarantees (e.g. mirroring, persistence, etc.) all at once. There is a cost in performance and stability for these various guarantees…
- Stop using Celery: it abstracts too much away
What RabbitMQ features do users overlook?
RabbitMQ has been around for ten years and is hardened to work in a variety of production environments. In many places, it's up and running and "just works". Not surprisingly, there are many features that have come along, but go overlooked. For example, even though RabbitMQ 3.7 came out a year ago, it seemed like many folks weren't aware of the new features. I asked the panelist for their opinion on the most overlooked RabbitMQ features.
Most overlooked RabbitMQ features
- Dead lettering and TTL (time to live)
- Consistent pattern exchange
- Event exchange
- Automated reply-to feature
Thinking differently about how to scale RabbitMQ
There are different ways to scale RabbitMQ today. Deciding how you should scale RabbitMQ depends on the use case. I asked the panelists to opine a bit on what they would change about the way RabbitMQ scales. On the other hand, I also asked them what they would keep about how RabbitMQ scales today.
What to (potentially) change about how RabbitMQ scales
- Separate the management from the messaging processing
- Have nodes of different types (e.g. nodes for connections)
- Queue location when scaling horizontally
- Rebalancing queues*: managing too many high volume queues on the same node
*On a related note, Jack Vanlightly highlighted the Rebalanser Project in his talk: GitHub.com/rebalanser
What works well about how RabbitMQ scales
- Because it's based on Erlang, it scales up very well
- Looking forward to Raft, the algorithms that will be used to reach consensus in a RabbitMQ cluster. This work is in progress and I recommend watching this talk from SprignOne Platform to learn more.
- Flexibility to enter the cluster from anywhere
Roadmap highlights
Michael Klishin gave an overview of what the core RabbitMQ engineering is working on. I recommend RabbitMQ users watch that replay when it's available. I also asked the panel which features and capabilities on the roadmap had them most excited.
Most anticipated roadmap features
- Quorum queues (see note above about Raft)
- Churn statistics
- Introducing native protocols
- All of the above!
AMQP and the future of RabbitMQ
Gavin Roy kicked off the day with an in-depth look at RabbitMQ's core underlying protocol, AMQP. How AMQP works defines many of the capabilities—and limitations—of RabbitMQ. For example, I learned how short-lived connections are very inefficient for RabbitMQ. This is because of the complex handshake required by AMQP to open that connection.
Given the opinions and expertise of the panel, I asked whether RabbitMQ should move away from AMQP. I also asked how that might be accomplished. I'll note that much of this discussion was not roadmap per se. Rather, the panelists were free to blue sky ideas about RabbitMQ's relationship to an underlying protocol.
The panelists discussed:
- Not a fan of AMQP 1.0 and think the spec should be forked (UPDATE: Note that this is not necessarily the opinion of the core RabbitMQ team)
- The exchange model is good… – Finding ways to support GRCP and HTTP 2/3
- How translating from MQTT to AMQP adds overhead. Note that this is how RabbitMQ works today, but Michael Klishin noted a protocol-agnostic core in his roadmap session…
- A first step is to make all protocols more equal
- The identity of other protocols is fundamentally different… would have to be generic (to be agnostic). Some potential for a loss in fidelity with this approach, but it's a simplification.
State of the RabbitMQ Community
This was the first RabbitMQ Summit in the technology's eleven-year history. And yet, the popularity of the technology and ample documentation and plug-ins indicate a vibrant community. I asked the panelists what they thought worked well in the RabbitMQ community today. I also asked where they thought the community could do a better job.
RabbitMQ community strengths
- Mailing list
- The first RabbitMQ Summit!
- Community contributions and the CLA process
- Early feedback (to the core RabbitMQ team)
- RabbitMQ.com (open to pull requests)
- Management interface
RabbitMQ community challenges
- Need better onboarding
- Need more operator-friendly docs and tutorials
- Exposing best practices and helping people find the right patterns
On that last note, there were a ton of best practices shared at the event. We're also hosting a webinar next month on the "10 things every developer using RabbitMQ should know." We'll be organizing those resources to make them more findable for the community on RabbitMQ.com. If you have any other ideas on how to make the RabbitMQ community better, or need clarification on any of these points, please leave a comment or jump on the mailing list.
Check back for links to more of the session replays. In the meantime, you can find the Twitter coverage of the event here: https://twitter.com/i/moments/1062010422944038912
**UPDATE** Watch the full panel replay below:
