Architecting Systems Like A Rock Band

In Software, orchestration often means control, synchronization, mediation and scheduling of decoupled application services in order to fulfill groups of tasks as part of a business processes. In highly distributed systems, orchestrating could be a nightmare to develop and maintain. If not designed carefully, it can lead to a scenario where the orchestrator becomes the bottle neck to future architecture changes. In traditional SOA, the orchestrator is often called the service bus or enterprise service bus. Even Micro-services architecture has an orchestrator. It hides behind the name "Gateway" and "Control Plane".

In an Orchestra, if the conductor leaves the orchestra in the middle of a symphony, chaos could occur eventually. The percussion section could become out-of-sync with the woodwinds and the brass section overtime. This is because each musician and each section rely heavily on a set of rules and patterns in the sheet music and of the tempo and the instructions of the conductor. Remove the music sheets and the orchestra will simply stop playing as the musician will not know what to play next. But not in a rock band.

Rock bands don't need a conductor and often times they don't need music sheets. This is because each musician do not rely on a single person to give them instructions. Instead, they listen to each other. They listen to the drum beat for the tempo. They listen to the singer to know which section they are in the song. They listen to the guitarist to know when to bring down their instruments volume to bring up the guitar solo, and so on and so forth. Each musician also knows what they need to do and when they need to do their thing without being told when and how. They decide when to do things based on what they hear. And because they don't follow music sheets, they can improvise and make the music sound better than expected. They can recover quickly from failure, i.e. a broken guitar string or a thrown away drum stick or improvise when this happens so, but they never stop playing just because of it. This is why each live performance is a bit unique as opposed to the musicians in an orchestra. In other words, musicians in a rock band react to events instead of being conducted.

Event-Driven Architecture

Broker Event-Driven Architecture

Event-Driven Architecture is recently catching the attention of software architects and engineers Today because of its simplicity and versatility. Components are loosely coupled. The interface contract definition does not have to be defined up front and can be easily changed. There are no more request-response (synchronous type or communication) and messages are pushed and not pulled. When pulling data a service needs to know upfront where to pull the data from. But when you push events/data, one does not need to know to which specific service you need to push the data to. There are many advantages of this architecture except if the system is too large, say 1000 services/actors, in which case the risk of loosing the enterprise-wide event broker becomes unacceptable.

The Rock Band Architecture

Similar to EDA, in a rock band architecture, the role of orchestration is delegated and distributed among each of the services. Independent services listen for events and react to those events whenever necessary. Services also broadcast events but they don't need to know how other services would react to it or would use those messages, or if other services are even listening. Much like a rock band, each musician is responsible for playing his own instrument but they also communicate by listening to each others instruments and looking at each other from time-to-time so that collectively, they can deliver the music as a one pleasing rock symphony. 

But there is a fundamental difference. Rock Bands are small. When you put 1000 rock musicians together and play the same song, it's fun for sure but it does not sound as good as a single 5-pieace rock band on a small stage. This is because it is so difficult to synchronize a huge number of musicians and that is the reason why orchestras need a conductor.

Sound travels at 342 m/s. So if you put a drummer and a bassist 100 meters apart, they will be de-synchronized by 1/3 of a second and that's what you are hearing in the video. The drums are not crisp and the bass guitar lost its punch.

Similar principles apply to Event-Driven Architecture. If two-closely relate services need to send messages to each other but there are 5 topics to go through before reaching the other end of the channel, you are loosing time and it's not efficient. 

Another important aspect to consider when deciding an architecture is how well it fits in the enterprise organization. Teams maintaining and building applications in a huge organization don't often speak to each other and the bureaucratic processes behind this are counter productive. No matter how you try to break the silos between teams, sections and departments, you can only do too much. So when different silos are sharing a middleware such as an Event Broker, what organizations usually do is to create another silo, such as a dedicated middleware team, to maintain that middleware. And from then on, teams and departments has to go through them when they want to establish a communication interface. That's a new layer of red tape.

The Rock Band Architecture is a Federated Event-Driven Architecture. Because bands sound better when they are the only ones playing the music in a small stage closer to the audience.

Rock Band Architecture - Federated Event-Driven Architecture

Instead of one single event broker, each silo has it's own event broker. This maximizes the freedom of the teams to choose how to implement their event broker. This also reduces the work on coordination between the silos. Overall, this reduces the effort required for dependency management.

The Event Broker

You don't need an orchestrator or a conductor in a Rock Band. What the system needs is a channel of communication where services can broadcast to and listen for events. The Event Broker. A rock band only needs a stage and their instrument monitors (the big speakers and guitar amps you see on stage). Events are not some generic free-form messages but are specific, small, real-time and contextual. The messaging channel for the event broker must have the following properties:

1. Events must be broadcasted in real-time or near real-time, otherwise the band would be out of sync.
2. The communication channel must also use a protocol that is platform agnostic so that services can be built in any platform and developed in any language
3. It supports any kind of endpoints, i.e. HTTP, HTTP/2, TCP, UDP, MQTT, etc...
4. It implements a publish-subscribe or consumer-producer messaging pattern

The Players

The services in RBA are the band members or the "Players". Each player plays a specific role that makes up the entire system or subsystem. Service can be in any form of software component (or even hardware) that is either subscribed to one or more topic or is publishing to topics or doing both. Services in RBA are similar to the "Actors" of the Actor-Model architecture pattern, they are completely isolated and are not aware of each others existence but players in RBA are much richer and performs more than just primitive calculations compared to Actors. They are also "stateful" and are always up and running. A player can be as big as an entire application or as small as a serverless function. Players are monitored so that in the event of a crash, there is something or someone that can do something about it like a player service automatic restart or an severity 1 incident raised to the helpdesk. One way to do this is to design the player service so that they are sending heartbeats regularly to a specific topic on the event broker.

The Band

A Band is a group of Players that are connected to a common Event Broker. These player services are grouped functionally into a Band. The grouping can be done in an organization context or by a more detailed context like a subsystem or even an application. The goal is to minimize dependencies between systems and/or organizational unit.

Edge Services

The services that are publishing  messages outside of it's own band are called Edge Services. I wanted to call it the "Groupie", but my wife doesn't approve. There can be one or more Edge Services in a band. Edge services must only publish messages to other band's event broker and must not subscribe to other band's event broker. This simplifies the service as they don't have to maintain a persistent connection with an event broker outside its perimeter.

Summary

Enterprises continue to implement SOA architectures and have also begun adapting event-driven architectures (SOA 2.0) as the standard for many of their applications. But organizations are also becoming more and more agile. Being able to decentralize event brokers in a federated manner can reduce or eliminate organizational dependencies between different silos which in turn would produce a more empowered, agile team who can make important architectural decisions on their own with minimal/isolated risk of impacts.