Tech Explainer - Understanding message queues & event-driven systems
Updated: Oct 3
Why do we need message queues?
A message queue is a software system that allows applications to communicate by sending and receiving messages. These messages are stored in a queue, which acts as a buffer for messages that need to be processed. The messages are stored in a first-in, first-out (FIFO) order, meaning that the first message added to the queue is the first message that is processed. This allows for asynchronous communication between applications, where one application can send a message and another can receive it at a later time.
Message queues are useful for a number of reasons. They can help to decouple different components of an application so that they don't have to be aware of each other. This can make the application more flexible and easier to maintain. They also allow for better scalability and fault tolerance, since messages can be stored in a queue and processed at a later time, rather than requiring all applications to be online at the same time.
In simple terms, please!
To understand the need for message queues one of the most important concepts to understand is the difference between synchronous and asynchronous processes/messaging.
Synchronous messaging
Synchronous messaging or processes simply mean at the same time. We’re engaging with synchronous messaging in our daily lives.
When we ask someone a question we need to wait for the answer and if the answer takes a while, well we just have to wait for the response.
The same takes place within many systems you use daily. Such as requesting a web page, once we’ve made the request we need to wait until until the web page is returned.
Asynchronous messaging
A prime example of asynchronous messaging is email. When sending an email to multiple recipients, you are placing it in their "queue" for them to access and read at their convenience. As the sender, you have no way of knowing when all recipients have read the message unless you receive a failure notification, such as an incorrect email address.
However, even if one email address is incorrect, you can still assume that the rest of the recipients have received the message.
Asynchronous messaging is also prevalent in instant messaging systems, both for human users and for systems. For instance, in a platform like Slack, you can create a channel to "publish" and “event” to, in Slack’s case a message, which others can "subscribe" to and receive notifications for. Additionally, it is possible to build integrations that allow other systems to "consume" specific messages and trigger further actions based on that message (the terms in bold above will be important later).
As you can see in the above image that the event is merely published to a queue and is not concerned about what consumers may do with that message.
Event-driven architecture
Event-driven architecture is a way of designing systems where the different components communicate with each other by sending and responding to events. An event is a message that describes something that has occurred, such as a button being clicked or a file being uploaded.
The components in an event-driven system can respond to these events in a way that is loosely coupled, meaning that they don't need to know the details of how the other components work. This allows for more flexibility, scalability and maintainability in the system. In simple terms, Event-Driven Architecture is a way of designing a system where different components communicate with each other by sending and responding to events, rather than requesting and receiving data.
The structure of event-driven message queues One of the most used, or at least well-known, event-driven tools is Kafka and we will look at the main components of that system:
Topics: Topics are the channels that messages are sent to and received from. Think of them as categories or streams of messages. For example, a topic could be "Order_Updates" for an e-commerce system, "Temperature_Readings" for an IoT application, or "Logs" for a logging system.
Producers: Producers are the applications or systems that send messages to topics. In the e-commerce example, the website would be a producer that sends order details to the "Order_Updates" topic.
Brokers: Brokers are the servers that handle the storage and routing of messages. When a producer sends a message to a topic, it is sent to one or more brokers, which store the message and make it available to consumers.
Consumers: Consumers are the applications or systems that read messages from topics.
To sum up…
Message queue / Event-driven systems are a powerful tool for building distributed systems that need to communicate and exchange data in a reliable, efficient, and scalable way. Kafka-style message queue systems, in particular, use a publish-subscribe model, where messages are sent to topics and consumed by subscribers. A broker is a key component of a message queue system, it contains one or more topics and manages the storage and retrieval of messages, and it is also responsible for routing messages to the appropriate consumers. Topics can also have multiple partitions for parallel processing and high availability.