Understanding event driven architecture

Key Takeaways

there is an increasing need to make processed consumable data available in real time and one way to make that happen is by using event-driven architecture the topic of this episode of cloud data engineering Academy time waits for no one let's get started Building Solutions in the world of Big Data you often face common challenges firstly you often need to connect different tools and processes together to accomplish your task in addition the world is moving faster and faster and reaction time windows are getting shorter as companies strive to get faster the data processing must keep up or risk being outdated and stale using an invent driven architecture allows you to tend to these challenges however there are many considerations necessary for Building A system that works with adventur and architecture the processing mindset is different and it may introduce additional components previously not used or replace existing components so let's take a closer look event-driven architecture can be described as a system or a pattern that reacts to a state the state can come in the form of a notification or a data payload you can also denote these states as messages in this event-driven architecture you have a producer whose responsibility is to generate the messages you also have a broker whose role is to ingest the messages lastly you have a third role a consumer the role of the consumer is to receive the message and react upon it these roles are not mutually exclusive for example something that consumes a message may also produce a new message to be forwarded onwards this architecture allows you to seamlessly connect different processes and functions asynchronously each process and function can contribute in their best way to the end goal without needing to share anything such as Hardware or States since they are disjointed processes they can scale independently and also fit a variety of use cases event-driven architecture looks very similar to the architecture of a real-time data processing pipeline by extending the architecture you can also enable a real time or near real time approach to data processing let's take a look at how we can build an vent driven architecture on Google Cloud any service that can deliver a message to an endpoint can be a producer these could be your servers your users Etc you then have a broker that ingests these messages here we have Google cloud pubsub a fully managed scalable messaging service your producers write messages to a pubsub topic and pubsub will deliver the messages to one or more subscriptions your consumers will then pull from the subscriptions to receive the messages you have many choices for consumers on Google Cloud if you're looking for a simple function you can use cloud functions to ingest those messages or you can use cloud run which can deploy a containerized app for large scale data processing Frameworks you have data proc and data flow to run your spark Hadoop Flink or beam jobs there is also event Arc which can help you deliver new States based on event triggers such as job completion these triggers will create a state that will be delivered to a service of your choosing there are a lot of different services that can be integrated with event Arc you may use any combination of products to achieve your goal pubs up with data flow to process a streaming realtime data pipeline or event Arc with Cloud run when there's a particular log published from a service logged to Cloud audit logs these Google cloud services will help you build an event driven architecture that will allow you to connect tools of different nature act upon your events or data in real time and Bridge asynchronous events easily that's it for now thanks for watching click the links below to learn more about event driven architectures on Google Cloud see you next time