Architecture of Rancher’s Docker-machine Integration

As you may have seen, Rancher recently
announced
our integration
with docker-machine. This
integration will allow users to spin up Rancher compute nodes across
multiple cloud providers right from the Rancher UI. In our initial
release, we supported Digital Ocean. Amazon EC2 is soon to follow and
we’ll continue to add more cloud providers as interest dictates. We
believe this feature will really help the Zero-to-Docker _(and
Zero-to-Rancher)_ experience. But the feature itself is not the focus
of this post. In this post, I want to detail the software architerture
employed to achieve this integration. First, it’s important to
understand that everyhting in Rancher is an API resource with a process
lifecycle. Containers, images, networks, and accounts are all API
resources with their own process lifecycles. When you deploy a machine
in the Rancher UI, you’re creating a machine resource. It has three
life cycle processes: 1. Create 2. Bootstrap 3. Delete The create
process is kicked off when the user creates a machine in the UI. When
the create process completes, it auotmatically kicks off the bootstrap
process. Delete (perhaps obviously) occurs when the user chooses to
delete or destroy the host. Our integration with machine is achieved
through a microservice that hooks into Rancher machine lifecycle events
and execs out to the docker-machine binary accordingly. You can check
out the source code for this service here:
https://github.com/rancherio/go-machine-service.
Logically, the interaction looks like this:

…Sorry for the bad graphic. Anyway… When you spin up Rancher
with docker run rancher/server ... with the default configuration, the
Rancher API, Rancher Process Server, DB, and Machine Microservice are
all processes living inside that container (and in fact, the API and
process server are the same process). The docker-machine binary is in
the container as well but only runs when it is called. You may at this
point be wondering about that event bus. In Rancher, we keep eventing
dead-simple and above all follow this principle:

There is no such thing as reliable messaging.

So, that “event bus” consists of the microservice making a POST
request to the /subsribe API endpoint. The response is a stream of
newline-terminated json events, similar in concept to the docker event
stream. The
process server is responsible for firing (and refiring) events until it
receives a reply event (another API POST) indicating the event was
handled. Further event handlers are blocked until the current event
handler replies successfully. The microservice is responsible for
handling the events, replying, and acting idempotently so that refires
can occur without ill-effect. So when the machine microservie receives a
create event, it translate the machine API resource’s prooperties into
a docker-machine cli command and execs out to it. Since the machine
creation process is long lived, the service monitors the standard out
and error of the call and sends corresponding status updates to the
Rancher server. These are then presented to the user in the UI. When
docker-machine reports that the machine was successfully created, the
microservice will reply to the original event it received from the
Rancher server. The successful end of the create event will cause the
process server to automatically kick off the bootstrap event, which
makes it way right back down to the machine microservice. When that
event is received, we’ll again exec out to docker-machine to get the
details needed to connect to the machine’s docker daemon. We do this by
executing the docker-machine config command and parsing the response.
With the connection parameters in hand, the service fires up a rancher
agent on the machine via docker run ... rancher/agent .... This is the
exact same command that a user would run if they wanted to manaully join
a server to Rancher. When that container is up and running, it will
report into the Rancher server and start hooking into container
lifecycle events in much the same way that this service hooks into
machine lifecycle events. From there, it’s business as normal for the
Rancher server and the machine’s rancher-agent. That about does it for
the technical architecture of our docker-machine integration. There are
a lot more interesting but minor technical detail to share, but I
didn’t want to go too far off into the weeds in this post. I’ll write
up some follow up post sharing those details in the not-too-distant
future. Finally, shout out (and thanks) to Evan
Haslett, Ben
Firshman, and the rest of the docker-machine
team and community for the help along the way. We look forward to more
exciting work with the docker-machine, including getting RancherOS in
there. If you’d like to learn more about Rancher, please schedule a
demo and we’ll walk you through the latest features, and our
future roadmap. Note: This post also appears
on Craig’s personal blog
here.
Feel free to check out that blog for more software engineering
insights.