Creating Microservices Deployments on Kubernetes with Rancher – Part 2

In a previous
article
in this series we looked at the basic
Kubernetes concepts including
namespaces, pods, deployments and services. Now we will use these
building blocks in a realistic deployment. We will cover how to setup
persistent volumes, how to setup claims for those volumes and then mount
those claims into pods. We will also look at creating and using secrets
using the Kubernetes secrets management system. Lastly, we will look at
service discovery within the cluster as well as exposing services to the
outside world.

Sample Application

We will be using
go-auth
as a sample application to illustrate the features of Kubernetes. If you
have gone through our Docker
CI/CD series of
articles then you will be familiar with the application. It is a simple
authentication service consisting of an array of stateless web-servers
and a database cluster. Creating a database inside Kubernetes is
nontrivial as the ephemeral nature of containers conflicts with the
persistent storage requirements of databases.

Persistent Volumes

Prior to launching our go-auth application we must setup a database for
it to connect to. Prior setting up a database server in Kubernetes we
must provide it with a persistent storage volume. This will help in
making database state persistent across database restarts, and in
migrating storage when containers are moved from one host to another.
The list of currently supported persistent volume types are listed
below:

• GCEPersistentDisk
• AWSElasticBlockStore
• AzureFile
• FC (Fibre Channel)
• NFS
• iSCSI
• RBD (Ceph Block Device)
• CephFS
• Cinder (OpenStack block storage)
• Glusterfs
• VsphereVolume
• HostPath (Testing only will not work in multihost clusters)

We are going to use NFS-based volumes, as NFS is ubiquitous in network
storage systems. If you do not have an NFS server handy, you may want to
use Amazon Elastic File Store to quickly
setup a mountable NFS volume. Once you have your NFS volume (or EFS
volume) you can setup a persistent volume in Kubernetes using the
following spec. We specify the hostname or IP of our NFS/EFS server, 1
GB of storage with read, write many access mode.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-volume
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: us-east-1a.fs-f604cbbf.efs.us-east-1.amazonaws.com
    path: "/"

Once you create your volume using *kubectl create -f
persistent-volume.yaml, *you can use the following command to list your
newly created volume:

$kubectl get pv
NAME                CAPACITY   ACCESSMODES   STATUS      CLAIM     REASON    AGE
mysql-volume          1Gi         RWX      Available                         29s

Persistent Volume Claim

Now that we have our volume, we can create a Persistent Volume Claim
using the spec below. A persistent volume claim will reserve our
persistent volume, and can then be mounted into a container volume. The
specifications we provide for our claim are used to match available
persistent volumes and bind them if found. For example, we specified
that we only want a ReadWriteMany volume with at least 1 GB of storage
available:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-claim
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

We can see if our claim was able to bind to a volume using the command
shown below:

$kubectl get pvc
NAME          STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
mysql-claim   Bound     nfs       1Gi       RWX           13s

Secrets Management

Before we start using our persistent volume and claim in a MySQL
container we also need to figure out how to get a secret such as
database password into Kubernetes Pods. Luckily, Kubernetes provides a
secrets management
system for this purpose.
To create a managed secret for Database password, create a file called
password.txt and add your plain text password here. Make sure there are
no newline characters in this file as they will become part of the
secret. Once you have created your password file, use the following
command to store your secret in Kubernetes:

$kubectl create secret generic mysql-pass --from-file=password.txt
secret "mysql-pass" created

You can look at a list of all current secrets using the following
command:

$kubectl get secret
NAME         TYPE      DATA      AGE
mysql-pass   Opaque    1         3m

##

MySQL Deployment

Now we have all the requisite pieces, we can setup our MySQL deployment
using the spec below. Some interesting things to note: in the spec, we
use the strategy recreate, which means that an update of the
deployment will drop all containers and create them again rather than
using a rolling deploy. This is needed because we only want one MySQL
container accessing the persistent volume. However, this also means that
there will be downtime if we redeploy our database. Secondly, we use the
valueFrom and secretKeyRef parameters to inject the secret we
created earlier into our container as an environment variable. Lastly,
note in the ports section that we can name our port and in downstream
containers we will refer to the port by its name, not its value. This
allows us to change the port in future deployments without having to
update our downstream containers.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: go-auth-mysql
  labels:
    app: go-auth
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: go-auth
        component: mysql
    spec:
      containers:
      - image: mysql:5.6
      name: mysql
      env:
      - name: MYSQL_ROOT_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mysql-pass
            key: password.txt
      ports:
      - containerPort: 3306
        name: mysql
      volumeMounts:
      - name: mysql-persistent-storage
        mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-claim

##

MySQL Service

Once we have a MySQL deployment we must attach a service front end to it
so that it is accessible to other services in our application. To create
a service we can use the following spec. Note that we could specify a
cluster IP in this spec if we wanted to statically link our application
layer to this database service. However, we will use the service
discovery mechanisms in Kubernetes to avoid hard-coding IPs.

apiVersion: v1
kind: Service
metadata:
  name: go-auth-mysql
  labels:
    app: go-auth
    component: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: go-auth
    component: mysql
  ClusterIP: 10.43.204.178

In Kubernetes, service discovery is available through Docker link style
environment variables. All services in a cluster are visible to all
containers/pod in the cluster. Kubernetes uses IP Tables rules to
redirect service requests to the Kube proxy which in turn routes to the
hosts and pods with the requisite service. For example, if you use
kubectl exec CONTAINER_NAME bash into any container and run env, you
can see the service link variables as shown below. We will use this
setup to connect our go-auth web application to the database.

$env | grep GO_AUTH_MYSQL_SERVICE
GO_AUTH_MYSQL_SERVICE_PORT=3306
GO_AUTH_MYSQL_SERVICE_HOST=10.43.204.178

Go Auth Deployment

Now that we finally have our database up and exposed, we can finally
bring up our web layer. We will use the spec shown below for our web
layer. We will be using the usman/go-auth-kubernetes image, which uses
an initialization
script
to add the database service Cluster IP to the /etc/hosts. If you use the
DNS add in Kubernetes, you can skip this step. We also use the secrets
management feature in Kubernetes to mount the mysql-pass secret into the
container. Using the the args parameter, we specify the db-host
argument as the mysql host we setup in /etc/hosts. In addition, we
specify db-password-file to so that our application connects to the
mysql cluster. We also use the livenessProbe element to monitor our web
service container. If the process has problems, Kubernetes will detect
the failure and replace the pod automatically.

* *

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: go-auth-web
spec:
  replicas: 2 # We want two pods for this deployment
  template:
    metadata:
      labels:
        app: go-auth
        component: web
    spec:
      containers:
      - name: go-auth-web
        image: usman/go-auth-kubernetes
        ports:
        - containerPort: 8080
        args:
          - "-l debug"
          - run
          - "--db-host mysql"
          - "--db-user root"
          - "--db-password-file /etc/db/password.txt"
          - "-p 8080"
        volumeMounts:
        - name: mysql-password-volume
          mountPath: "/etc/db"
          readOnly: true
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 30
          timeoutSeconds: 1
      volumes:
      - name: mysql-password-volume
        secret:
          secretName: mysql-pass

Exposing Public Services

Now that we have setup our go-auth service, we can expose the service
with the following spec. We specify that we are using the service type
NodePort which exposes the service on a given port from Kubernetes range
(30,000-32,767) on every Kubernetes host. The host then uses the
kubeproxy to route traffic to one of the pods in the go-auth deployment.
We can now use round-robin DNS or an external loadbalancer to route
traffic to all Kubernetes nodes for fault tolerance and to spread load.

apiVersion: v1
kind: Service
metadata:
  name: go-auth-web
  labels:
    app: go-auth
spec:
  type: NodePort
  ports:
  - port: 8080
    nodePort: 30000
    protocol: TCP
    name: http
  selector:
    app: go-auth
    component: web

With our service exposed, we can use the go-auth REST API to create a
user, generate a token for the user, and verify the token using the
following commands. These commands will work even if you kill one of the
go-auth-web containers. They will also still work if you delete the
MySQL container (after a time when it gets replaced).

curl -i -X PUT -d userid=USERNAME 
     -d password=PASSWORD KUBERNETES_HOST:30000/user
curl 'http://KUBERNETES_HOST:30000/token?userid=USERNAME&password=PASSWORD'
curl -i -X POST 'KUBERNETES_HOST:30000/token/USERNAME' 
     --data "IHuzUHUuqCk5b5FVesX5LWBsqm8K...."

Wrap up

With our services setup, we have both a persistent MySQL service and
deployment, as well as a stateless web deployment for go-auth service.
We can terminate the MySQL container and it will restart without losing
state (although there will be temporary down time). You may also mount
the same NFS volume as a read-only volume for MySQL slaves to allow
reads even if the master is down and being replaced. In future articles,
we will cover using Pet Sets and Cassandra-style application layer
replicated databases to have persistent layers which are tolerant to
failure without any downtime. For the stateless web layer we already
support failure recovery without downtime. In addition to our services
and deployments, we looked at how to manage secrets in our cluster such
that they can be exposed to the application only at run time. Lastly, we
looked at a mechanism by which services discover each other.

Kubernetes can be daunting with its plethora of terminology and
verbosity. However, if you need to run workloads in production under
load, Kubernetes provides a lot of the plumbing that you would otherwise
have to hand-roll.