Building Event-Driven Systems with PostgreSQL Logical Replication and Drasi | POSETTE 2026

Key Takeaways

Hello everyone. My name is Deirdre I'm part of the global black belt team in Microsoft covering EMEA focusing on Postgres and other open source databases. This session is building event driven systems with Postgres logical application and tracing. In this session I'll show you three ways to tap into that stream and build event driven architectures directly on Postgres. We will run live benchmarks compare latency numbers and look at working code that you can deposit today. As I mentioned, we will go through building event driven architecture directly on Postgres. We will look into live benchmarks, working code and practical guidance. So let's start. In the common scenarios when you have e-commerce application, you need to most probably you need to send an email when a high high value order is paid. How do you detect that? And how you detect that change with when it happens. Most teams start with pulling and running a query every few seconds, but you miss changes between polls. You waste queries when nothing changed. Some use database triggers, but triggers run inside transaction. They block rights. They are hard to debug. Other do the double write, save to the database and then send to Kafka for example. But what happens when DB succeeds and Kafka fails? You get data inconsistency, the number one of cause of bugs in microservice architecture. Going to the next slide how we can solve that with the CDC. And in Postgres, every insert, update and delete is already being recorded in the in the wall. The wall is right ahead log. Every row change in the wall, we just need to read it. No application code changes. Guaranteed delivery, capture inserts, updates, and deletes. Um there is no dual right risk there. That's what we will see today. So, let's get a little bit deeper deeper here. With the uh wall pipeline, we have five steps. Application writes, wall records every row changes, logical decoding convert binary wall into JSON or uh PG output depends on how you set up your environment, consumer reads, and then downstream action. Postgres retain wall segments until the consumer confirm it has processed them. This is powerful, but also dangerous. We will come back to that. Let's go to the next slide. Sorry about this. So, let's see how replication is is working. So, um this is um uh this is the configuration of the um um wall, how we can configure that. So, this is three three items that we need to configure. Um first thing, we need to um change the wall level to equal logical. In some um um applications, um in some um providers, it requires a restart. If you're running on a VM, it requires a restart. Basically, this um configuration wall equal logical distance Postgres to include row level data in the wall, not just uh physical page changes. The second thing is the publication. Um this define which tables to stream changes from. In this demo, we're using e-commerce app, like I said, so we have um orders table because it is data base for e-commerce. Uh think of the publication as a filter at the Postgres level. The third item and the last one is the replication slot. Each consumer gets its own slot. The slot tracks the position of wall. Again, the wall is right head lock. Now, let me show you the architecture of the demo that we will go through. I know this is a little bit busy slide, but bear bear with me. At the very top, we have here PostgreSQL flexible server. You can run um that in different PostgreSQL if you want to uh if you want to take the code. You can run that as a Docker con- container. And then you have here the um green box with three CDC um streams. You have a wall stream, you have um a Debezium in the blue box, and then you have the Kafka cluster. These are the three scenarios that we will go through. And then you have we have a um a load and dashboard. We are using low cost with 50 users simulating and generating traffic that will go through um the um uh the benchmark table that we have. I have also Prometheus and Grafana. And we will not go through that to save some time today. Um the PostgreSQL is the source of truth, and um you can uh change that and um if you wanted to run your own environment. Also, remember if you're running a um external database, latency might be a little bit um uh a bit higher. Um let's go to the next slide. Okay, this is too fast. Um start with the um uh first option. Um oh, okay. I forgot to talk about the um the the Debezium option here. Um in the this in the blue um um blue box, um we have a continuous query filtering. And in the um uh the Debezium with Kafka, we have also Kafka Connect and Apache Kafka before reaching to the consumer. So, um let's go to the to the um the next slide, which is the putting things um side by side um to talk about the the environment that we will go through. So, we have same possible resource, three different batches, all right all writing to the same benchmark events table. And this is how we get a fair comparison. Same load, same data, but different architecture for each scenario that we have. Uh the first one is wall to JSON. Um let me describe how this is go. The first one we have four steps. Um wall to plugin then to consumer, to database, no middleware, direct connection right way. Um let me switch to my terminal here. And um um let me give you a first what's the Docker containers that we have. So, we have again um the um um Kafka Kafka Connect and and also the some exporters for the monitoring. On Kafka and Debezium, we have the broker, we have the connect and the UI, and also we have the Stream It dashboard that we will see the results of the environment that we are running. Um before that I wanted to show you the database that we have. So, this is the benchmark um table. We don't have a lot of things to to show because we didn't run anything yet. Um now um checking the how the orders looks like. This is the source of truth that we have. So, we have the customer ID, we have the amount, we have the status. And um the the status can be new, shipped, and paid. And this is will be important because we will talk about filtering later on. Let's see what's the distribution of the data that we have today. So, um we have um different status for the canceled, new, paid, and shipped orders. The It changes from It amounts to range from 20 to 2,000. Okay, let's start with the first one uh first uh option that we have, and this is wal2json. So, now we're starting wal2json and also starting the uh low cost. And when we start low cost, it will um generate some traffic read from the order table to the benchmark. This would take a couple of um um um um uh uh minutes. We have 2 minutes run for each workload. So, as you can see here, wal2json um uh is is running. The median latency is um 2 um milliseconds, and low cost is starting to um um uh push the data in inside the the benchmark table. So, now this is moving. This is This is good. This is a live demo that we're having here, and we'll do that for the different Now, there's the Visium, and there there is a little bit later on. Um let me continue going through the um um presentation. And um when we go to through through the presentation, I'll I'll cover a couple of things to give you a little bit deep dive on how things are running. Um with the wal2json plugin, it's confirm um converting the binary wall records into JSON. It is a custom out um um output plugin installed on the server. Key benefit here right away is simplicity. It just um what Postgres has. And um one disk, it is one slot, one consumer, no replay, and no no fan out. If you have five uh if you have five downstream services, then you need five replication slots. This is five times the wall cost. Drassy solves this differently. One slot, but multiple queries, multiple reactions inside Drassy. Fan out without extra slots. In the BCM, it is solved with them with the Kafka. One slot into Kafka, then unlimited number of consumers and groups. So, this is a critical design choice. So, let's jump into the wall JSON use cases. And this is the good fit when it comes to all trail. You wanted to stream every change to comply the store or cash invalidation for example. Or you wanted to do search sync. So, you keep elastic cluster and post elastic and sync or real-time ETL. Also, if you wanted to have single service webhook firing HTTP called per change for example. But wall to JSON will not be suitable if you have multiple consumers that need the same stream or you you need replay. I think our a benchmark finished already. Let's go through it. Let's read now the the latest numbers. The latency is around 2.5 milliseconds. So, this is this is this is good. And that's as close to the middle as you can get with logical replication. No broker, no middleware, just post a grace and and Python. And when we go to post a grace here, let's see the number of records that we have. So, it's around 3,000 500. So, this is this is good. Let's jump into the next scenario that we have. And this is with the BCM and Kafka. So, in the BCM scenario, we have six steps. Wall to PG output to Kafka connect, then to Kafka consumer to database. Two extra hops. Let's see what the cost of that. I'll I'll back to the um terminal and fire up the um the uh the scenario of the um uh Debezium. And I'm I'm launching that as we did before, what will happen that will um uh bring up the environment for Debezium and the same time open up uh Locust. And then we will see um the uh the the whole changes happening and uh the traffic going to the benchmark table. So, Locust is up and running now. And let's see how it how it looks like on our um um um uh Locust environment. Um It's starting now. Okay. So, it's started. Um more traffic is coming. Um we have couple of response time. And also the number of users that we have is uh 50 as as as we mentioned. Um Quick comparison here between Walter Jason and um uh Debezium. You'll see big difference between them. You can see um Walter Jason is low here while the um um Debezium is uh is there. This is because of the two hops that we have. This is around six 600 milliseconds or 500 milliseconds. So, it is way bigger than uh the um uh Walter Jason. So, um let's go to the um um uh presentation to get into the um um details. So, um Debezium is the industry standard the change this for the change that data capture. Um it reads a PostgreSQL wall using PG output um PG output um uh protocol, but instead of going direct to change the it goes through And flow of the flow goes to Kafka Connect into Apache Kafka topic and our Python consumer then reads from from Kafka. Um you get schema of evolution, multiple consumers groups for even replay. That's a lot of capabilities, but it comes with cost of infrastructure. And um going through the the the full division pipeline includes Kafka Connect, schema management, and connector configuration. Um it is the right choice when you already have Kafka or when you have when you when you need multiple teams consuming the same event stream. Let's go to the use cases of Debezium. Of course, it's very obvious microservice event bus. Many teams, many consumers, one stream that you that you want to have. Event replay for new services joins replay and replays from offset zero. Cross system sync. And also is very important to know that if you're using if you wanted to have one consumer then Kafka would be overkill. And if you wanted also sub minute second latency. You don't want to operate Kafka. This is one of the the reasons why you don't want to use Debezium plus Kafka. Okay, I'm going to the Kafka UI. Here we see the um our topic. Okay, doesn't doesn't work. But our local stream finished. Still running. If I updated the the stream that benchmark that is from the table that we created couple of seconds ago. We'll see that we have a division and wall to JSON. I'll cancel wall to JSON. See that is going through the around 500. Um So, now with the division it is finished. It's around 500 milliseconds. This this is a good going through the P sequel. Now we see the same amount of events that that happens. Both of them doesn't have any kind of uh filtering. It is only the events that uh we got from the orders table. Okay, I'll move to the the um next um um to topic here and which which is going through the Drici. We we we we need to to not only the Kafka thing is not slow, but it's three times um higher. This is direct path. And the trade-off that you get durability, find out, and the entire Kafka ecosystem. Now, what if you don't want to every change, just to the the changes that matters to you. This is where Drici comes in. So, with Drici um is a open-source data change processing platform for Microsoft. It is now a CNCF sandbox project. And it lets you define continuous continuous queries that react to a data changes in real time without writing stream code. Um Drici has four steps. It is the same as wall to JSON, but step three is different. It instead of shipping all events, Drici runs a continuous query. It only emits events that match our filter. So, let's go to the terminal again and launch the um the Drici use case and it will do the same. It will bring up low cost and it will write to the benchmark table that we that we have. So, this will take a couple of seconds. And we go to the CDC again uh for the the benchmark. And you see two things here coming up, Drassy and Drassy filters. A filter because Drassy we can see the filtered option, which is the $500 um orders and two status. So, I'll exclude the Bisium because the Bisium is too high. I will skew the the the whole environment. So, you see the Bisium and Drassy and Drassy filter playing in the same ballpark. It's around 2.2.5. Um low-cost still um running here. So, um until that finish, I'll go through the the presentation once once again to go through the Drassy deep dive. So, Drassy use one replication slot, just one replication slot. Uh but it runs multiple continuous queries against the single stream that we have. We have two queries right now. One capture all the orders, and one filter high-value orders, like we said. Uh that is amounts with $500 or more for the orders that paid or shipped. Each query can trigger different reaction, webhook, gRPC, HTTP, all from one slot. With you'll need two slots and two separate consumers. This is twice the wall cost. Drassy evaluates every change against all active queries. If it match, it emits. If not, it will filter it will be filtered out. Drassy supports Cypher and SQL queries. It is Kubernetes native for the source queries and reaction. They are all in declarative config. So, it makes things easier to add additional uh runtime tools without redeployment. There is also some trade-off. It's a new project and there is no durable even log. If you need replay, you need to add Kafka. Let's go to the use cases of um uh Drassy um for fraud detection, for example, SLA monitoring, dynamic business rules, and also a multi-query fan fan out in uh condition notification. Um it is similar to the case that we have where you wanted to email only VIP customers with high value value value orders. Um Drassy will not be suitable if you wanted durable even log. Um you will need to add um Kafka and I said or Kafka plus Drassy. And if you wanted to have a broad ecosystem connectors, um Debezium, for example, has more than uh 50 um uh async connectors which make it um very rich. So, um let's go to the um the terminal. Hopefully, Drassy will um uh will be um uh finished. Um if we go through the approaches, um you will find that we have around five 5,000 um events. We have uh 1,500 or 1,600 um uh uh filtered filtered orders. So, now with the with all of this, what we can see here with um with Drassy, we can see the orders that we have and um see the grouping of the filtered and unfiltered ones. Um Let's do this. So, we see with the Debezium and um Drassy, they are the same event numbers, but the filter is around uh half and the captured and the um the average milliseconds. As you can see here, um uh the is around 500 ms. Debezium is around 3 ms. Volt JSON is 2 ms. When we read the demo results, what you see here is the same query that we we run. And if we compare that with the CDC pipeline that we have for for the benchmark, you'll find that um Debezium is going up. I'll filter out Debezium. Sorry, I filter out Debezium here and add the Debezium. You'll find that they are around the same timing, but you add a filtering with with Debezium. So, going to the the the results here reading the benchmark, um we have three approaches. Same workload, 50 users each, 2 minutes run the latest first. It is the the the benchmark that I run for this slides, it was 1.5 because I used Docker container, not external Postgres. So, Postgres externally will will add a little bit of latency. It is around 2.5 to 3 ms for both Volt JSON and Debezium. With Debezium, it is 500. Um Kafka, of course, add the broker time plus the consumer pull interval. This is 300 times slower. But slower is not worse. It depends on what you need. Debezium has more because it writes both filtered and and unfiltered. So, about 500 5,000 um high-value orders that we filtered out. Only Debezium can do that filtering. Volt JSON and Debezium show zero filtering. And comparing also replication slots. You'll find Walrus isn't is one slot per consumer. If you wanted five consumers, this means five slots and this is five times the cost on water retention. Drassi has one slot multiple queries, multiple reactions, fan out happens inside the Drassi. With the busy and one slot into Kafka fan out by consumer groups. This is a big deal in a production system. And on infrastructure and infrastructure infrastructure complexity, Walrus isn't just possible this Drassi is using possible this and Drassi server. With the busy and you have possible this plus Kafka plus Zookeeper plus Kafka Connect, which is a little bit a lot. On code side, Walrus isn't is around 150 lines of code. The busy and is around 120. And Drassi around 180 including the the the declarative language that we were using. There is no single winner here. It depends on your requirements. Let's go to the most important thing here if you're running in a production environment. All three approaches use replication slots. If your consumer dies, possible this keeps accumulating wall. It will not release these segments until these slots confirm it has processed them. This is strongest part part but also it is dangerous. And here's what happens. If consumer pauses for network issue or crash or new deployment, slots stop advancing. Possible this retain all wall since the last acknowledgement. And our later, your disk fill up. Possible this goes to read only. All rights will fail. I've seen 64 gigabytes of wall accumulate in 1 hour under moderate load. Single forgotten slot can take down the production environment. So how we can protect our systems against it? Three things. The first the first one is max slot wall keep size. We keep that we make sure that it is not minus zero. So, we can free the the the wall. This is first thing. If you're running fixed per server, it is minus one. So, make sure that you you change that. The second thing is monitoring and query PG replication slots regularly and check the wall lock column and alert when exceeds your threshold. Third one is drop stall slots. If a consumer has has been dying for an hour, drop that slot. It is safer than filling up the disk. Let's go to the last slide that we have and this is around the guidance and what which which solution to pick. So, you use wall to JSON when you have single consumer, one slot, one reader. You want simple setup, audit log, cache invalidation, but remember, no fan out. Each new consumer needs its own slot. Debezium used plus Kafka, you have multiple consumers. You need multiple consumers that needs this the same event. You need event replay or microservice microservice event bus. When to use Debezium you only care about specific data pattern, high priority alerts, you want it to have rules and to change at run time. And acquisition flow need fan out or replay, Debezium plus Kafka. Need server side filtering, this is Debezium. Single consumer, all events, simple as possible setup, this is wall to JSON. If you're not sure, start with wall to JSON. It is the easiest setup and easiest to to to migrate from. So, you can add the Kafka or the last you later. Also, you don't always need a streaming platform. Sometimes the database that you have is already enough. Um you can download all the code for this demo from my GitHub repo. You can see the link down here. And the um And hope this session was beneficial to you. And thank you.