In this short post I will explain some of the things I’ve learned building Akka.NET.
I will describe some of the friction points I have noticed and why I personally don’t use features like Akka Cluster to build entire systems.
Some of these thoughts might be obvious, some might be naive, but they do reflect my current view on building distributed systems.
Location transparency and Message contracts
Actors are supposed to be location transparent.
DCom, Corba and .NET remoting were all based on a local model, trying to make remote calls appear as local, in process calls, all failed for this very reason.
Never try to make the explicit implicit.
The Actor Model is remote first and can be optimized when used locally.
We constantly fall into the trap of trying to make local message objects somehow become wire friendly.
Making something that is designed to work local only and try to shoehorn that into a world of network calls will result in problems.
Messages should be designed with a remote first mindset, explicit contracts that can support versioning and be maintained over time.
In Akka.NET we have tried very hard to make serialization just work magically for any message type, currently using Json.NET serializer and soon the Wire serializer.
This gives you low friction when getting started with Akka.NET, but it is the wrong design to use for real systems.
You should swap that out for a custom serializer using e.g. ProtoBuf, where your message contract are explicit and there are no magic or unexpected behavior involved.
Distributed monolith and the Unix philosophy
Actor model frameworks and languages does not play nicely between platforms.
Erlang, Pony, Akka, Akka.NET, Project Orleans, Service Fabric ActFab, Orbit, none of those can communicate with any of the others.
If you base your entire infrastructure on such framework or language, you are building a distributed monolith.
You pay a very high price when you decide to build your systems this way, none of the components or services in your systems can be replaced with another tech stack, you are forever bound to use the same stack until you replace the entire thing or surgically cut one of the parts out of it.
Instead, architect your systems as isolated islands, bounded contexts, and connect them using standard protocols and contracts, HTTP, JSON, XML, AMQP etc.
Then inside each of those isolated islands, feel free to use any of the above technologies.
The above does not just apply to actor model frameworks, it applies to any RPC or micro-service framework that have their own special service discovery, clustering support or protocols.
The Unix philosophy should be applied.
Use tools like Consul, Etcd, Zookeeper for service discovery, Docker using Swarm, Rancher, Kubernetes for deployment and clustering.
This gives you a lot more flexibility and options.
If it turns out that one of your choices didn’t work out, there are plenty of others to solve the same problem without completely redesigning your system.
There are of course cases where your system might have special requirements, such as extremely high throughput and/or latency requirements, then other rules apply.
Maybe you need to build a distributed monolith for such reasons, but it should not be the default when designing a new system.
More on this in the next post.
“Erlang, Pony, Akka, Akka.NET, Project Orleans, Service Fabric ActFab, Orbit, none of those can communicate with any of the others.”
This isn’t true. Erlang can talk with most other environments. Its pretty trivial to get erlang to talk to jvm based stuff (akka or otherwise) as there are OTP adapters available for most platforms.
No Greg, in order to natively talk to Akka, you need to port the JavaSerializer to Erlang as some of the built in messages or part of messages in Akka uses that. you would also need to make your Erlang system simulate Akka heartbeats in a way that is compatible with Akka in order to not get kicked out from Akka Remote/Cluster.
Feel free to prove me wrong by linking such adapter that fully supports connections from Erlang to Akka Remote or Cluster.
If your suggestion is to do it without using the built in protocols, but rather using HTTP or plain sockets using Akka.HTTP or Akka.IO. then yes, that is exactly my point, integrate systems using platform agnostic concepts.
you are talking against your own baby?
There are pros and cons to most things. I am simply highlighting some of the backsides of Akka / Akka.NET.
Akka.NET is a solid framework with a huge user-base.
That does not mean that I think that every single design choice in there is the best.
I am spending my time fixing those mentioned issues in my new library now.
A remake if you will, a conceptual 2.0 from the experiences gained building Akka.NET.
Same concept, but removing the bits that I think were wrong with Akka.