Decoupling Microservices with AWS EventBridge Pipes
AWS introduced a new service called EventBridge Pipes. It is used to create point-to-point integrations between event producers and consumers with optional transform, filter and enrich steps.
In combination with an EventBridge and DynamoDB Streams this makes a perfect match to decouple serverless microservices. Decoupling is an important principle in a microservice architecture because it allows you to build, deploy, and manage your microservices independently.
In this blog post we will look at an example to first see how decoupling of services can be done without EventBridge Pipes and then look at some problems you will encounter. In the next step we look at how EventBridge Pipes can be used to solve the problems.
The time before EventBridge Pipes
In our example an order service uses DynamoDB to persist its data. Changes of the order service should be exposed to other services like a order-history service, payment service and inventory service.
One way to achieve this is to use DynamoDB Streams and propagate all changes from the order service to the order-history, payment service and inventory service.
One problem of this solution is that each consumer of DynamoDB Streams contributes to the read capacity and may result in throttling which is why AWS recommends only two consumers at a time. This is why this solution does not scale well when more and more services get interested in the data of the order service.
Therefore it would be better to use an EventBridge event bus to distribute the events. However there is still a problem if the events of DynamoDB Stream are directly send to an event bus: an important purpose of a microservice architecture is to isolate each service but in our example the database model of the order service is now coupled to all consumers because each change in the data model in the order service might require changes in all consuming microservices. Also not all persist actions should necessarily be propagated to other services.
This means before sending the DynamoDB Stream events to the event bus we need to filter the interesting events and transform the events to domain events that will then form the API to other services. And here is where EventBridge Pipes comes into play.
Using EventBridge Pipes
EventBridge Pipes is a service to connect sources to targets. You can add optional filtering and define an optional enrichment or transformation.
This is exactly what we need to add the missing peace to our proposed solution. Instead of directly connect EventBridge to DynamoDB Streams we add a Pipe in between for filtering and transformation.
This way the data model of the order service stays independent and interested services can register to the event bus to listen for interesting events.
What about ordering of events?
EventBridge does not guarantee ordering and exactly once delivery. Instead it offers an at least once delivery. This means that events can be out of order and delivered more than once. If you really need an exactly once semantic you can use SNS FIFO instead of EventBridge. It has most of the capabilities as EventBridge including content based filtering but guarantees ordering and exactly once delivery. However in most cases I would rather recommend to build the services in such way that it can handle duplicate and out of order events because this may make your system more robust.
Conclusion
EventBridge Pipes is a service that receives events from a variety of targets including DynamoDB, which provides optional filtering, enrichment and transformation and sends the events to targets including EventBridge event bus.
This can be used in services that use DynamoDB for persistence where changes should be propagated to other services. Instead of using DynamoDB Streams directly to connect services EventBridge Pipes can be used to filter the events and transform them to domain events before sending them to an event bus for distribution to other services. If ordering is an issue SQS FIFO can be used instead of an event bus.
This way services can synchronize data while the data model stays decoupled.