Event-driven architectures (EDAs) have become a cornerstone in the design and development of highly scalable, decoupled, and extensible systems. From small-scale applications to massive enterprise ecosystems, EDAs are the default du jour for asynchronous events and data flow. But what comes next?
In this article, we’ll argue that while capable of meeting the challenges of Internet scale, much of the work that goes in to implementing and maintaining Event-Driven systems is wasted on code dedicated to understanding, reconciling, and distributing messages, rather than on user-facing features that enhance value.
Before the proliferation of microservices and event-driven systems, monolithic architectures were the go-to design for software applications. Monolithic systems are easy to develop, test, and deploy, but they have inherent limitations. As applications grow, the complexity of managing state, data, and scalability becomes increasingly challenging.
With the advent of the Internet and the exponential increase in users, data, and processes, it became crucial to build systems that could scale easily and adapt to changes. Enter the concept of decoupling, a design principle aimed at separating the functionalities of a system into distinct, independent components.
Event-driven architectures emerged as an answer to the dual challenge of decoupling (and multiplying) different parts of an application with the need to keep all parts (”Microservices”) in sync with each other. The solution offered by Event-Driven architectures is for each microservice to publish messages to a central queue, and for other microservices to pick up messages of relevance, interpret the message, and incorporate the message contents into the microservice’s local state.
Although Event-Driven architectures provide a way to decouple various parts of an application such that those parts can communicate asynchronously, each microservice or application needs to contain logic for understanding events so that each microservice can itself independently replay the state of the system and hope to arrive at the correct state.
The Pub-Sub pattern can be considered one of the most significant milestones in the adoption of event-driven architectures. With Pub-Sub, systems could publish messages (events) to particular channels or topics, and any number of subscribers could receive these events asynchronously. Technologies like MQTT and RabbitMQ popularized this approach.
The rise of cloud computing and serverless architectures fueled the adoption of EDAs. Cloud providers like AWS, Azure, and Google Cloud offer native services that support event-driven paradigms (AWS Lambda, Azure Functions, Google Cloud Functions).
The need for real-time analytics and Internet of Things (IoT) devices also spurred the widespread adoption of event-driven architectures. They allowed for immediate, asynchronous handling of events, making them ideal for these applications.
One of the overlooked drawbacks of event-driven architectures is the code complexity required for each application or service to understand a stream of event messages. Unlike request-response patterns where the expectations are explicit and often tightly-coupled, EDAs necessitate that each participating component comprehends the semantics, structure, and implications of incoming events. This means applications need to have built-in logic to handle various types of events, which often includes:
This results in each application having a significant amount of code just dedicated to understanding and reacting to events, which can complicate development and debugging efforts.
Event-driven architectures have been a significant advancement in the way applications are built and scaled. However, they come with their own set of complexities around event ordering, state management, and data consistency, as previously discussed. The task of event processing often burdens application code with additional logic to handle these complexities. Mycelial offers a fundamentally different approach that abstracts away many of the challenges associated with event-driven systems, enabling seamless data orchestration with minimal engineering effort.
While traditional event-driven architectures may require developers to have in-depth knowledge of various messaging protocols like MQTT, RabbitMQ, or Kafka, Mycelial provides an abstraction layer across diverse inter-application communication protocols. This removes the need for developers to be protocol experts, simplifying the architecture and accelerating development.
One of the most substantial pain points in event-driven architecture is managing the state across different services. Mycelial handles device messaging and state reconciliation regardless of the protocol or network conditions. For instance, in the US Navy case study, Mycelial enabled real-time data synchronization between unmanned and manned vessels, whether they were connected or not.
Traditional event-driven tools like Kafka or Apache NiFi can be resource-intensive. Mycelial, with its minimal memory overhead and single-binary deployment, is designed for SWaP (space, weight, and power)-constrained environments. This makes it particularly useful for edge computing scenarios where resource utilization is a critical factor.
In a typical event-driven system, implementing features like data prioritization or local AI/ML inferencing would involve substantial code complexity. Mycelial simplifies this by enabling on-device processing and intelligent data distribution based on your AI's priorities, as seen in the Walmart case study for voice-powered in-home shopping.
Mycelial aims to eliminate most of the cost and complexity of maintaining Cloud infrastructure for edge applications. With features like the Mycelial Router for visual data pipeline setup and Mycelial Catalog for plug-and-play AI/ML models, the need for specialized knowledge in maintaining and scaling your architecture is dramatically reduced.
Mycelial is more than just another tool in the event-driven landscape; it is a paradigm shift. By abstracting away the intricacies commonly associated with event-driven architectures, Mycelial allows developers to focus more on the business logic and less on the underlying plumbing. As data continues to grow in volume, velocity, and variety, tools like Mycelial that simplify the complexities of data orchestration and event processing are not just convenient—they're essential.
Let’s explore the challenges of Event-Driven Architectures in the context of a fleet of aerial drones accompanied by a group of Humvees. If we’re designing an Event-Driven or Message-Driven System, we might have events that look like the list below:
Handling these events using traditional event-driven architectures could be intricate and prone to errors. For example, what happens if a BatteryStatusEvent is processed late and a drone runs out of power mid-mission? What if a ThreatAlertEvent doesn't reach all Humvees in real-time?
Today, most folks don’t think about the impact of all the event-handling code we have to write. We’ve gotten used to it.
But let’s take a look at all the code we would need to write for each application in our integrated Drone + Humvee solution.
If we’re using Kafka as a message queue, we would need to write a function to handle each event like so:
The code comments above highlight the 5 central challenges of Event-Driven Architectures:
Possibly the thorniest challenge is Event Ordering. Look below at the complexity of logic needed IN EACH APPLICATION to make sure that Events are valid and processed in the correct order.
This example illustrates the significant complexities involved in handling events using a traditional event-driven architecture. Contrast this with using a tool like Mycelial, which abstracts away much of this complexity, automating data orchestration, state management, and event synchronization across a distributed system.
Mycelial aims to ease the complexities that inherently come with event-driven architectures, particularly those related to message reconciliation, ordering, and state management. One of Mycelial’s strongest advantages lies in the abstraction layer it provides across diverse inter-application communications protocols. Mycelial handles device messaging and state reconciliation regardless of the protocol or network conditions, which means applications don't have to worry about the details of message synchronization.
Let's return to the example above where multiple aerial drones are collecting data and communicating back and forth with a collection of Humvees. Both the drones and the Humvees are equipped with local SQLite databases. Mycelial is configured to keep these databases in sync across the network.
Instead of complex event ordering and validation code, with Mycelial all you need to do is query the local SQLite database for the latest state. Mycelial takes care of keeping the data up to date.
SQL Query for Receiving Latest Drone Data:
SQL Query for Receiving Latest Humvee Data:
Because Mycelial synchronizes all databases, running this query on any drone's or Humvee's local database will yield the most current data from all peers. Mycelial's state reconciliation works in the background and ensures that you're querying a database that represents the latest state of the entire network, without having to explicitly handle message ordering, duplication, or loss.
Similarly, updating the local state and propagating it across the network is as straightforward as executing a SQL INSERT statement.
SQL Statement to Update Drone's Local State:
SQL Statement to Update Humvee's Local State:
Mycelial would automatically propagate this new state to every node in the network, ensuring that when any device queries its local database, it gets this newly inserted data as well. The complexities associated with event-driven architectures—like message ordering, network partition handling, and state reconciliation—are abstracted away by Mycelial.
We believe Mycelial provides an OSS alternative not only to popular messaging frameworks like MQTT or RabbitMQ, but an architectural alternative that offers the possibility of huge efficiency gains for developers and data engineers.