Terraform Hands-on |

Terraform

I guess, that wast majority of all possible readers of this blog knows Terraform. As their site states: Terraform - Use Infrastructure as Code to provision and manage any cloud, infrastructure, or service. There is plenty of articles about Terraform itself, so I’ll not go to the details… If you are still interested in or for any wired reason you like my style, you can watch my (F5 related) Terraform introduction here: Slovak version, English version.

My use-case

I run all of my publicly facing services in ScaleWay cloud in one Docker server. Typically those services generates zero traffic 🔀, which means also zero CPU 📉 utilization. It means I don’t need a big HW resources and a DEV1-S type serves me well. Only recently I deployed a Minecraft server for my kids 👨‍👩‍👦‍👦 and this container is utilizing the server a lot. CPU goes up to 100% and RAM is full.

This Minecraft container has brought to idea increase the underlying HW which is powering my Docker - hope you understand that this is very necessary and important step to take 😉.

On the other hand I’m quite lazy to migrate all (7!) the running containers. So this gives me the opportunity to automate the whole think on one hand and learn something new on the other.

This article evolves as I proceed with this project…

High level design

I need to keep the state of the Terraform project, so I’ll use Terraform Cloud as the backend (if I’ll be to lazy I’ll relay on iCloud 🤷🏻‍♂️)
One folder with Terraform project and sub-folder for each Docker container, that I want to deploy using TF Provisioners
Some projects have to be pulled directly from Github
I need to think about how to manage the DNS records, some of my domains are in Cloudflare with is easy to manage by TF, but rest is not

Services

Ok, here goes list of my very important (= nobody is using them) services:

Traefik 🔀🔒

not the real end-user service, but rather my web traffic router and manager for the SSL certificates from Let’s Encrypt
I use older 1.7 version, and the configuration is just one file traefik.toml
Also, I need to keep in mind, that it requires also acme.json file for storing the certificates, this one is ignored from git
This one needs to start as first container, as it is the gateway for the rest, and some issues with Network dependencies might occur - need to do some testing on this.

This very important site - lubos.klokner.sk 🌍

Should be easy as the whole website is in private git repository
I just need to run one command to generate the static content and then to start the NGINX

Minecraft server for my kids 🧱

the main reason for all this
here is the question what to do with the actual data, I guess I’ll lazy and just drop the content of server

YouTransfer 📁

A file upload service, that I use mainly for sharing files with my (F5) customers
Here in configuration is two sensitive parameters, one to unlock the configuration changes and the second is for the email account that is used for the email notifications
The whole code is hosted again in private repo on Github, so it should be ok

Some demo apps 👨🏻‍💻

such as Juice Shop, F5 Demo HTTPD…
I assume, that I just create sub-folders for each one, and the configuration with be part of the code

Default web for geck2.vault-tec.sk ♺

here I have really no idea, why I’m running it

Let’s get started!

How I failed…

I put some effort into this, but it seems that TF is really NOT suitable for configuring machines… the plan was to build the Instance in Scaleway, assign DNS in Cloudflare and up to this point it is ok, easy and working…

variable "scaleway_access_key" {}
variable "scaleway_secret_key" {}
variable "hostname" {
  default = "docker-sw1"
}
variable "env" {
  default = "test"
}
variable "username" {
  default = "root"
}

variable "cloudflare_email" {}
variable "cloudflare_token" {}
variable "base_domain" {}

provider "scaleway" {
  access_key = var.scaleway_access_key
  secret_key = var.scaleway_secret_key
  zone = "fr-par-1"
  region = "fr-par"
  organization_id = "xyz"
}

resource "scaleway_instance_ip" "public_ip" {}

resource "scaleway_instance_server" "docker" {
  name = format("%s-%s.%s", var.hostname, var.env, var.base_domain)
  type  = "DEV1-M"
  image = "docker"

  tags = [ var.env, "linux" ]
}

# Cloudflare

provider "cloudflare" {
  version = "~> 2.0"
  email = var.cloudflare_email
  api_key = var.cloudflare_token
}

resource "cloudflare_record" "vaulttec" {
  zone_id = "xyz"
  name    = format("%s-%s", var.hostname, var.env)
  value   = scaleway_instance_ip.public_ip.address
  type    = "A"
  proxied = false
}

…but then I wanted to use provisioner to download or transfer the Docker configuration for each particular container and start it.

  provisioner "file" {
    source      = "./docker/traefik"
    destination = "/root/docker/"

    connection {
      type     = "ssh"
      host     = scaleway_instance_ip.public_ip.address
      user     = var.username
      private_key = file("~/.ssh/id_rsa")
    }
  }

  provisioner "remote-exec" {
    inline = [
      "cd /root/docker/traefik/",
      "docker-compose up -d"
    ]
    connection {
      type     = "ssh"
      host     = scaleway_instance_ip.public_ip.address
      user     = var.username
      private_key = file("~/.ssh/id_rsa")
    }
  }

There are two issues:

it seems that each and every change in the Docker files leads to the destruction of Scaleway instance and recreation of it
it can’t work with encrypted private ssh key

Error: Failed to read ssh private key: password protected keys are
not supported. Please decrypt the key prior to use.

Other options

Ok, using Terraform for the infrastructure management is ok, but I need to look for some other options for managing my Docker instances:

There is a Docker provider for Terraform, but I would need to have a registry from which I would be downloading the final images, which is not my case
Right now I’m looking at Ansible and what are the options for Docker management… it seems to me, that I would need to replace my docker-compose.yaml files with Ansible playbooks and on top of it manage the transfer of data to to the Docker host using Ansible… Seems to be lot of work a the beginning, but at least I could learn something new…
Create a simple SHELL script, which would me easiest for me, but doesn’t look very DevOps…

At this moment, I’m going to investigate the second option. So stay tuned for next post. Hopefully, it won’t lead you to /dev/null again.

And that’s all folks!