Lately I’ve been studying Kubernetes to replace a custom PaaS implementation. While running minikube locally is trivial, I would like to know what would entail building a cluster from zero. I’ve used Terraform before to provision single hosts and felt like I could still use this for a three-node cluster.

This tutorial uses Terraform to provision a three-node CoreOS cluster on DigitalOcean.

Generate a discovery URL

curl -w "\n" "https://discovery.etcd.io/new"

The -w "\n" parameter tells curl to append a newline after the response.

The response will be similar to this: https://discovery.etcd.io/eeecbdf5d73feb93ee53d05916809f90. You will need the random string (eeeec…) when you prepare a cloud-init config file.

Prepare a cloud-init config file

DigitalOcean supports cloud-init to bootstrap a node. We will use cloud-init to setup the etcd service for each node. Etcd is a distributed key-value database used for service discovery (i.e., publishing truths across the cluster).

Below is an example of a CoreOS cloud-init file (#cloud-config needs to be on the first line):

#cloud-config

coreos:
  etcd2:
    discovery: https://discovery.etcd.io/eeecbdf5d73feb93ee53d05916809f90
    advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
    initial-advertise-peer-urls: http://$private_ipv4:2380
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://$private_ipv4:2380
  fleet:
    public-ip: $private_ipv4
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start

Install terraform

First-time installation (I’m using Apple):

brew install terraform

Upgrading:

terraform --version

Terraform v0.7.7

Your version of Terraform is out of date! The latest version
is 0.7.11. You can update by downloading from www.terraform.io

Prepare terraform configuration

I’ve split up my terraform config into several files.

All DigitalOcean-specific configuration goes to this file. I’ve set do_token to be a variable to be provided at runtime:

variable "do_token" {}

provider "digitalocean" {
  token = "${var.do_token}"
}

The master node’s configuration:

resource "digitalocean_droplet" "master" {
  image="coreos-stable"
  name="master"
  region="sfo2"
  size="1gb"
  ssh_keys=["de:ad:be:ef"]
  user_data="${file("user_data/cluster.yml")}"
  private_networking=true
}

resource "digitalocean_record" "master" {
  domain="example.com"
  type="A"
  name="master"
  value="${digitalocean_droplet.master.ipv4_address}"
}

The first worker’s configuration:

resource "digitalocean_droplet" "worker-1" {
  image="coreos-stable"
  name="worker-1"
  region="sfo2"
  size="1gb"
  ssh_keys=["de:ad:be:ef"]
  user_data="${file("user_data/cluster.yml")}"
  private_networking=true
}

resource "digitalocean_record" "worker-1" {
  domain="example.com"
  type="A"
  name="worker-1"
  value="${digitalocean_droplet.worker-1.ipv4_address}"
}

Generate execution plan

terraform plan -out cluster.tfplan

The file cluster.tfplan will be a binary file and could be any name.

Run execution plan

I will only need to provide my API key when running commands:

terraform apply cluster.tfplan

Terraform will then talk to the DigitalOcean API to provision three servers and add the required DNS entries. At the end of the run, terraform will display something like this:

terraform-apply

Smoke test

Shell into a node and list all machines:

fleetctl list-machines

You should see something similar:

fleetctl-list-machines

Commit your changes

Terraform allows infrastructure to be versioned. You will need to add the configuration files and the generated state file to a git repository.

See also