The architecture behind our first beta

Two more weeks in, and Scylla keeps moving forward. A quick look today at our current architecture — the one that will carry us through our first Beta!

Background

For a long time, Scylla was built as a single block: an API that had to handle gRPC calls, business logic, database interactions, and so on.

But adding agents, the last step before this Beta, made us rethink our architecture. Connecting one or several agents to the API involved much more code than we expected: the agent pool (mpsc channels, gRPC connections) and its management (heartbeats, crash handling, …) represented a non-negligible amount of code and complexity — and, on top of that, very little of it had anything to do with the role of an API.

The problem

An almost obvious answer started to take shape: extract this agent “orchestration” logic into a dedicated crate. Except that this would only move the problem somewhere else: how would the API communicate with this external service?

We needed both to split the code properly and to find a reliable, independent way to handle communication between all these services.

Our solution

As you can see on the diagram, our architecture is now built around two main components: the core and the broker.

Concretely, we isolated a core: a crate that gathers the business logic — essentially a contract for interacting with the database — and is imported by every other service (API, recorder, etc.). The rest of the code is built around it, with each service importing this shared kernel rather than reimplementing its own rules.

That said, the services have no direct link between them, and that’s by design: we want a structure that scales in O(1), not O(n).

This is where the broker plays a key role. If we decide to add a new service, it only has to connect to the broker, listen for the events it cares about, and publish the ones other services need. From there, we can chain cascading sequences of events between services without any extra complexity.

Finally, when it comes to using the core, agents are an exception. Since their role is to execute pipelines, this would represent an attack vector, so preventing them from accessing the database was a must. We therefore introduced a buffer service, the recorder, whose job is to listen to agents and persist their data into the database on their behalf. This indirection lets us keep a controlled entry point to the db, while leaving agents free to evolve independently.

What’s next?

Even though this new architecture has helped us answer real problems, we are aware of its limits and of the new challenges it raises, especially around security. This is just a step, and we are actively working on it for the next Beta!

Stay tuned!

• Follow us on LinkedIn
• Join our Discord server