Controversial database topics are a guaranteed success on reddit, because everyone has an opinion on those topics. More importantly, many people have a dogmatic opinion, which always triggers more debate than pragmatism.
So, recently, I posted a link to an older article titled The Database As Queue Anti-Pattern by Mike Hadlow, and it got decent results on /r/programming:
Mike’s post was pretty much against all sorts of queueing in the database, which matches the opinions I have heard from a couple of JavaZone speakers in a recent discussion, who all agreed that messaging in the database is “evil.”
… and I’m saying: No. Messaging in the database is not an anti-pattern, it is (can be) totally OK. Let’s consider why:
KISS and YAGNI
First off, if you don’t plan on deploying thousands of message types and millions of messages per hour, then you might have a very simple messaging problem. Since you’re already using an RDBMS, using that RDBMS also for messaging is definitely an option.
Obviously, many of you will now think:
If all you have is a hammer, everything looks like a nail
… and you’re right! But guess what, the reasons for only having a hammer can be any of:
- You don’t have the time to equip with a sophisticated tool box
- You don’t have the money to equip with a sophisticated tool box
- You actually don’t really need the sophisticated tool box
Think about these arguments. Yes, the solution might not be perfect, or even ugly…
But we’re engineers and as such, we’re not here to debate perfection. We’re here to deliver value to our customers and if we can get the job done with the hammer, why not just forget our vanity and simply use that hammer to get the job done.
Transactional queues
In a perfect world, queues are transactional and guarantee (to the extent allowed by underlying theory) atomic message delivery or failure – something that we’ve been taking for granted with JMS forever.
At GeeCON Krakow, I had a very interesting discussion with Konrad Malawski regarding his talk about Akka Persistence. If we remove all the hype and buzzwords (e.g. reactive programming, etc.) from Akka, we can see that Akka is just a proprietary alternative to JMS, which looks more modern but is lacking tons of features that we’re used to having in JMS (e.g. like transactional queue persistence).
One of the interesting aspects of that discussion with Konrad Malawski is the fact that a 100% message delivery guarantee is a myth (details here). Which leads to the conclusion:
Messaging is really hard
It is, indeed! So if you really think you need to embed a sophisticated MQ system, beware of the fact that you will have to learn how it really works and how to correctly operate it.
If you’re using RDBMS-backed queues, you can get rid of this additional transactional complexity, because your queue operations participate in the transactions that you already have with your database. You get ACID for free!
No additional operations efforts
What developers very often underestimate (we can’t say this enough) are the costs incurring to your operations team when you add new external systems to yours.
Having just one simple RDBMS (and your own application) is a very very lean and simple architecture. Having an RDBMS, an MQ, and your application is already more complex.
There are a lot of excellent DBA out there who know what they’re doing when operating productive databases. Finding excellent “MQA” is much harder.
If you’re using Oracle: Use Oracle AQ
Oracle has a very sophisticated built-in queueing API called Oracle AQ, which can interoperate with JMS.
Queues in AQ are essentially just tables that contain a serialised version of your message type. If you’re using jOOQ, we’ve blogged about how to integrate Oracle AQ with jOOQ, recently.
RDBMS-centric applications can be much easier
We’ve blogged about that before as well: Why Your Boring Data Will Outlast Your Sexy New Technology.
Your data might just survive your application. Consider Paypal replacing Java with JavaScript (it could also have gone the other way round). In the end, however, do you think that Paypal also replaced all their databases? I don’t. Migrating from Oracle to DB2 (different vendor), or from Oracle to MongoDB (different DBMS type) is mostly motivated by political decisions rather than technical ones. Specifically, people don’t migrate from RDBMS to NoSQL databases entirely. They usually just implement a specific domain with NoSQL (e.g. document storage, or graph traversal)
Assuming that the above really applies to you (it may, of course, not apply): If your RDBMS is in the middle of your system, then running queues in your RDBMS to communicate between system components is quite an obvious choice, isn’t it? All system parts are already connected to the database. Why not keep it that way?
Conclusion
The arguments listed here are all pretty obvious and pragmatic. At some point, they no longer hold true, as your messaging demands are really big enough to justify the integration with a sophisticated MQ system.
But many people have strong opinions about the “hammer / nail” argument. Those opinions may be correct but premature. Very often in software engineering, it is entirely acceptable and sufficient to work with just one tool. The hammer of software: The RDBMS.
| Reference: | Using Your RDBMS for Messaging is Totally OK from our JCG partner Lukas Eder at the JAVA, SQL, AND JOOQ blog. |
Recently attended an Atlanta Java Uses Group (AJUG) meeting where we discussed using Redis as a persistent/clustered messaging queue. RIAK would work just as well. If one is using an RDBMS already, this might be a good solution. I know PostGreSQL has a noSQL component and one could possibly use that to act as a transactional message queue.
Good article…. gets me to thinking.
All these choices of persisting messages. Well, the RDBMS solution is really the path of least resistance unless you’re using Oracle or SQL Server, which have more sophisticated out-of-the-box solutions. So, if you’re not using an RDBMS because you have more sophisticated requirements, why not just default to JMS, then – which is also transactional?