One of the fundamental aspects of microservice architecture is data possession. Encapsulation of the data and logic prevents the tight coupling of services. Since they only expose information via public interfaces (like stable REST API) and cover inner implementation details of data storage, they can evolve their schema independently of 1 another.
A microservice should be an autonomous unit that can fulfill most of its assignments with its own data. It can also ask other microservices for missing pieces of information required to complete its tasks and, optionally, store them as a denormalized copy in its storage.
Inevitably, services also have to exchange messages. Usually, it’s essential to ensure that the sent message reaches its destination and losing it could yield critical business implications. Proper implementation of communication patterns between services might be 1 of the most critical aspects when applying microservices architecture. It’s quite easy to drop the ball by introducing undesirable coupling or unreliable message delivery.
Let’s consider a simple scenario of service A having just completed processing some data. It has dedicated the transaction that saved a couple of rows in a relational database. Now it needs to notify service B that it has completed its task and new information is available for fetching.
The easiest resolution would be just to send a synchronous REST request (most probably POST or PUT) to service B immediately after a transaction is dedicated.
This approach has some drawbacks. Arguably, the most necessary 1 is a tight coupling between services caused by the synchronous nature of the REST protocol. If any of the services is down because of maintenance or failure, the message will not be delivered. This kind of relationship is called temporal coupling because both nodes of the system have to be available throughout the whole period of the request.
Introducing an additional layer — message broker — decouples both services. Now service A doesn’t need to know the exact community location of B to send the request, just the location of the message broker. The broker is responsible for delivering a message to the recipient. If B is down, then it’s the broker’s job to hold the message as lengthy as necessary to successfully pass it.
If we take a nearer gaze, we might notice that the dispute of temporal coupling persists even with the messaging layer. This time, though, it’s the broker and service A that are using synchronous communication and are coupled together. The service sending the message can assume it was properly received by the broker only if it gets back an ACK response. If the message broker is unavailable, it can’t obtain the message and won’t reply with an acknowledgement. Messaging systems are very often sophisticated and durable distributed systems, but downtime will nonetheless happen from time to time.
Failures very often are quick-lasting. For example, an unstable node can be restarted and become operational again after a quick period of downtime. Accordingly, the most straightforward way to increase the chance for the message to get through is just retrying the request. Many HTTP clients can be configured to retry failed requests.
But there’s a catch. Since we’re never sure whether our message reached its destination ( maybe the request got through, but just the response was lost?), retrying the request can cause that message to be delivered more than once. Thus, it’s crucial to deduplicate messages on the recipient side.
The same principle can be utilized to asynchronous communication.
For example, Kafka producers can retry the delivery of the message to the broker in case of retriable errors like
NotEnoughReplicasException. We can also configure the producer as idempotent and Kafka will routinely deduplicate repeated messages.
Unfortunately, there’s bad news: even retrying events doesn’t guarantee that the message will reach its goal service or the message broker. Since the message is stored only in memory, then if service A crashes before it’s able to successfully transfer the message, it will be irretrievably lost.
A situation like this can leave our system in an inconsistent state. On the 1 hand, the transaction on service A has been successfully dedicated, but on the other hand, service B will never be notified about that event.
A fine example of the consequences of such failure might be communication between 2 services when the first 1 has deducted some loyalty points from a user account and now it needs to let the other service know it should send a sure prize to the customer. If the message never reaches the other service, the user will never get their gift.
So maybe the resolution for this dispute will be sending a message first, waiting for ACK, and only then committing the transaction? Unfortunately, it wouldn’t help much. The system can nonetheless fail after sending the message, but just before the commit. The database will detect that the connection to the service is lost and abort the transaction. Nevertheless, the destination service will nonetheless receive a notification that the data was modified.
That’s not all. The commit can be blocked for some time if there are other concurrent transactions holding locks on database objects the transaction is trying to modify. In most relational databases, data altered inside a transaction with an isolation level equal to or stronger than the read dedicated (default in Postgres) will not be visible until the completion of a transaction. If the goal service receives a message before the transaction is dedicated, it may try to fetch new information but will only get the stale data.
Additionally, by making a request before the commit, service A extends the period of its transaction, which may probably block other transactions. This sometimes may pose a dispute too, for example, if the system is under high load.
So are we doomed to live with the fact that our system will get into an inconsistent state occasionally? The old-school approach for ensuring consistency across services would use a pattern like distributed transaction (for example 2PC), but there’s also another neat trick we could utilize.
The dispute we’re facing is related to an issue that we can’t atomically both perform an external call (to the message broker, another service, etc.) and commit the ACID transaction. In the pleased path scenario, both tasks will succeed, but problems start when 1 of them fails for any reason. I will try to elaborate how we can overcome these issues by introducing a transactional outbox pattern.
As the first step, we need to introduce a desk that shops all messages that are meant for delivery — that’s our message outbox. Then instead of immediately doing requests, we just save the message as a row to the new desk. Doing an INSERT into the message outbox desk is an operation that can be a part of a regular database transaction. If the transaction fails or is rolled back, no message will be continued in the outbox.
In the second step, we should create a background worker process that, in scheduled intervals, will be polling data from the outbox desk. If the process finds a row containing an unsent message, it now needs to publish it (send it to an external service or broker) and mark it as sent. If delivery fails for any reason, the worker can retry the delivery in the next round.
Marking the message as delivered includes executing the request and then a database transaction (to update the row). That means we are nonetheless dealing with the same problems as before. After a successful request, the transaction can fail and the row in the outbox desk won’t be modified. Since the message status is nonetheless pending (it wasn’t marked), it will be re-sent and the goal will get a message twice. That means that the outbox pattern doesn’t forestall duplicate requests — these nonetheless have to be handled on the recipient side (or message broker).
The main improvement of the transactional outbox is that the intent to send the message is now continued in durable storage. If the service dies before it’s able to make a successful delivery, the message will stay in the outbox. After restart, the background process will fetch the message and send the request again. Eventually, the message will reach its destination.
Ensured message delivery with possible duplicated requests means we’ve got an at-least-once processing guarantee and recipients won’t lose any notifications (until in case of some catastrophic failures inflicting data loss in the database). Neat!
Not surprisingly, though, this pattern comes with some weak points.
First of all, implementing the pattern requires writing some boilerplate code. The code for storing the message in the outbox should be hidden under a layer of abstraction, so it won’t intrude with the whole codebase. Additionally, we’d need to implement a scheduled process that we’ll be getting messages from the outbox.
Secondly, polling the outbox desk can sometimes put significant stress on your database. The query to fetch messages is usually as plain as a simple SELECT statement. Nevertheless, it needs to be executed at a high interval (usually below 1s, very often way below). To reduce the load, the check frequency can be decreased, but if the polling happens too rarely, it will impact the latency of message delivery. You can also decrease the number of database calls by simply increasing the batch size. Nonetheless, with a huge number of messages selected, if the request fails, none of them will be marked delivered.
The throughput of the outbox can be boosted by increasing the parallelism. Multiple threads or instances of the service can be each choosing up a bunch of rows from the outbox and sending them concurrently. To forestall different readers from taking up the same message and publishing it more than once, you need to block rows that are just being handled. A neat resolution is locking a row with the SELECT … FOR UPDATE SKIP LOCKED statement ( SKIP LOCKED is available in some relational databases, for example in Postgres in Mysql). Other readers can then fetch other unblocked rows.
Last, but not least, if you’re sending massive quantities of messages, the outbox desk will very quickly bloat. To hold its size under control, you can create another background process that will delete old and already sent messages. Alternatively, you can simply remove the message from the desk just after the request is acknowledged.
A more sophisticated approach for getting data from the outbox desk is called database log tailing. In relational databases, every operation is recorded in WAL (write-ahead-log). It can be later queried for new entries regarding rows inserted into the message outbox. This kind of processing is called CDC (capture data change). To use this technique, your database has to offer CDC capabilities or you’d need to use some kind of framework (like Debezium).
I hope I have convinced you that the outbox pattern can be a worthy asset to increase the robustness and reliability of your system. If you need more information, a exceptional source to learn more about transaction outbox and its various applications is the book titled Microservices Patterns by Chris Richardson.
A properly implemented outbox pattern ensures that the message will ultimately reach its goal. To get this guarantee end to end with messaging system, the broker also needs to assure at-least-once delivery to the consumer (like Kafka or RabbitMQ do).
In most circumstances, we do not only want to simply deliver the message. It’s also necessary to ensure that the task that was triggered by the message is completed. Hence, it’s essential to acknowledge the message receipt only after the task’s completion! If the task fails (or the whole service crashes), the message will not be acked and will get re-delivered. After receiving the message again, the service can retry processing (and then ack the message if the task is completed).
If we do things the other way around: ack the message first and only then start processing, we’ll lose that at-least-once guarantee. If the service crashes when performing the task, the message will be effectively lost. Since it is already acknowledged, there will be no retries of delivery.
The at-least-once processing guarantee means that the recipient will from time to time get repeated messages. For that reason, it’s crucial to actively eliminate duplicate messages. Alternatively, we can make our process idempotent, which means that no matter how many times it is re-run, it should not further modify the system’s state.
Message delivery can be retried in case of explicit failure of processing. In this case, the recipient could return NACK (unfavorable acknowledgement) to the broker or reply with an error code when synchronous communication is used. This is a clear sign that the task was not successful and the message has to be sent again.
The more interesting scenario happens when the work takes more time to finish than anticipated. In such a situation, the message, after some time, can be considered as lost by the sender (for example, an HTTP request can hit the timeout, visibility timeout of SQS can pass, etc.) and scheduled for redelivery. The recipient will get the message again even if the first task is nonetheless alive. Since the message is not yet acked, even a proper deduplication mechanism won’t forestall triggering multiple concurrent processes caused by these repeated messages.
Timeouts can be fine-tuned so they take into consideration the prolonged time when the task is handled. On the other hand, finding out the appropriate value is sometimes difficult, especially when finishing an task by a client takes an unpredictable amount of time.
Repeated work is very often not a huge deal with properly working duplicate elimination. We can simply get the message and reject the result after task completion (or upsert if it’s idempotent). But what if the processing includes some costly action? For example, after receiving the request, the client could spin up an additional VM to handle the task. To avoid unnecessary waste of sources, we could adopt another communication pattern: the transaction inbox.
The inbox pattern is quite identical to the outbox pattern (but let’s say it works backwards). As the first step, we create a desk that works as an inbox for our messages. Then after receiving a new message, we don’t start processing right away, but only insert the message to the desk and ACK. Finally, the background process picks up the rows from the inbox at a handy pace and spins up processing. After the work is complete, the corresponding row in the desk can be updated to mark the task as complete (or just removed from the inbox).
If received messages have any kind of unique key, they can be deduplicated before being saved to the inbox. Repeated messages can be caused by the crash of the recipient just after saving the row to the desk, but before sending a successful ack. Nevertheless, that is not the only potential source of duplicates since, after all, we’re dealing with an at-least-once guarantee.
Again, the throughput of processing can be improved by increasing parallelism. With multiple workers concurrently scanning the inbox desk for new tasks, you need to remember to lock rows that were already picked up by other readers.
Another possible optimization: instead of waiting for the worker process to select the task from the inbox, we can start it asynchronously right after persisting it in the desk. If the process crashes, the message will nonetheless be available in the inbox.
The inbox pattern can be very helpful in case the ordering of messages is necessary.
Sometimes, the order is assured by a messaging system (like Kafka with idempotence configuration turned on), but that is not the case for every broker. Similarly, HTTP requests can interleave if the client doesn’t make them in a sequence in a single thread.
Fortunately, if messages contain a monotonically increasing identifier, the order can be restored by the worker process while reading from the inbox. It can detect missing messages, hold on until they arrive, and then handle them in sequence.
What are the disadvantages? Similar to the outbox pattern: increased latency, additional boilerplate, and more load on the database.
Very often, the recipient service can cope without the inbox. If the task doesn’t take lengthy to finish or completes a predictable amount of time, it can just ack the message after processing. Otherwise, it might be worthwhile to spend some effort to implement the pattern.
As I stated at the beginning of this article: setting up proper and reliable communication channels between microservices is not a piece of cake! Thankfully, we can accomplish a lot by using correct patterns. It’s always necessary to consider what guarantees your system needs and then apply appropriate solutions.
And always remember to deduplicate deduplicate your messages 😉