Creating and managing Azure with Terraform

Introduction

What if your Azure infrastructure gets bigger and bigger? How would you make it automatable? How would you document it or what if you want to manage a multi cloud environment? These are some questions which are solved with an “Infrastructure-As-Code” tool. Such a tool automates, reduces human error, is consistent every run, is fast and the “Infrastructure-As-Code” description is a form of documentation. In this blog I explain why I advise Terraform and how you can start to deploy infrastructure on the Azure platform with Terraform.

This blog consists of two parts. In this part I describe Terraform, why you want to use it instead of other Infrastructure-As-Code tools and how you use Terraform. In the second part we will create infrastructure in the Microsoft Azure Cloud with Terraform and the knowledge we gained of Terraform from the first part of the blog.

Infrastructure-As-Code tools

Before I start with a deep dive of Terraform, I will discuss some other Infrastructure-As-Code tools, which differ in a few important aspects. These differences are in the type, the method and the architecture.

Type:                Configuration and Orchestration

Method:            Procedural and Declarative

Architecture:      Client + Server and Client-only

Type

There are two different types of Infrastructure-As-Code tools. These two types of tools are configuration management tools and the orchestration tools. Tools such as Ansible, Puppet, Chef and Saltstack were made for configuring servers in mind. These tools have some capabilities to orchestrate infrastructure, but it is not their main strength.

Orchestration tools such as CloudFormation and Terraform are made to orchestrate the infrastructure layer and are specifically made to do so. Firstly you have a clear distinction between configuring and creating the infrastructure. This helps with creating an overview. Secondly these tools are made with one focus in mind which is creating infrastructure and have some capabilities that configuration management tools don’t have. For example, with Terraform it is possible to validate changes and view the differences before running.

To visualize the different types and layers my go-to explanation is as follows: there are two layers for making and configuring your servers and infrastructure. The bottom layer lays the foundation of your infrastructure/house. It gives the servers the right amount of memory, CPU and storage and makes sure all the virtual networking is done correctly. After that you lay the top layer of your infrastructure/house. Here you configure your software such as the LAMP stack.

Method

Infrastructure-As-Code tools have two distinct ways they work. These are: Procedural and Declarative. The tools which are Procedural work very script like, they work from the top to bottom and make changes based on the input in each step. The Declarative tools work very differently, because you only code the changes you want to be made, not how you will make them. The tool knows what end result needs to be, and makes the changes accordingly.

Architecture

The different Infrastructure-As-Code tools work from either a client + server or a client-only architecture. In a client + server architecture you work from the server and have an agent installed on the host you want to configure. This means more overhead on your servers but has the upside of having your management done from a central location, which helps with logging actions done on your infrastructure. In a client-only architecture the upside is less overhead. It makes your virtual machines smaller, use less network bandwidth and use less resources used. But because there isn’t an agent installed, logging and keeping a single source of truth is harder. Some tools offer an enterprise version which help with logging, such as Ansible Tower.

Differences

In the following table you can find a small overview of the biggest differences between them:

Where does Terraform stand?

Terraform is a tool with the focus of creating, managing and destroying virtual infrastructure. It does so in a declarative way and is client-only. It currently supports the orchestration of most well-known clouds and on-premise solutions, databases, network and monitoring providers/vendors. Some examples include:

Cloud:

  • AWS
  • Azure
  • Google Cloud Platform
  • vCloud Director
  • VMware vSphere

Databases:

  • MySQL
  • PostgreSQL
  • MongoDB

Network:

  • Cloudflare
  • DNS
  • Cisco ASA
  • F5 BIG-IP

Monitoring:

  • Datadog
  • Grafana
  • New Relic

For a full list of providers go to the following link: https://www.terraform.io/docs/providers/index.html

How do you create your Terraform resources?

You create Terraform resources with the HashiCorp language also called HCL or with JSON. HashiCorp is the vendor of Terraform. Their own language is easy to read and create. A tip from me is to use the Visual Studio Code editor with the “Terraform” plugin by Mikael Olenfalk. It helps with autocompletion, available inputs and more. In this blog we are going to use the HashiCorp language which has the extension “.tf”. These are the files that Terraform uses to create the infrastructure. You define the infrastructure by declaring providers and resources. The providers are where we want to interact with, and the resources is what we want to create or manage on that provider.

Provider

In the provider block you name the provider block and some arguments, which are used to authenticate to the provider. For Azure you will need to provide the “azurerm” provider, which needs the subscription_id, client_id, client_secret and the tenant_id. See the following example:

To get this information you will need to log into your account with a running subscription in the Azure portal. Open your Azure CLI and type: “az account list”. From the output note down your “id”. This is what is needed in the “subscription_id” in your azurerm provider information. To manage your terraform you want to create a “Service Principal” which is a created identity in which applications such as Terraform can use to authenticate. To create such a service principle, you need to enter the following command:

az ad sp create-for-rbac –role=”Contributor” –scopes=”/subscriptions/$SUBSCRIPTION_ID”

the output will look something like this:

You want to map these values to the following Terraform variables;

  • appId is the client_id defined in the azurerm provider;
  • password is the client_secret defined in the azurerm provide;
  • tenant is the tenant_id defined in the azurerm provider.

Note: although this is fine for local development. It is recommended to store these values in your environment variables in your machine, so your code can be stored in versions control system like Bitbucket, GitLab or Github without leaking this information. This could lead to misuse by 3rd parties. Another save way to store your login details is the use of Vault. This is another tool made by HashiCorp and allows the ability to store information in a save manner and is integrated very nicely in Terraform with their own provider.

Resources:

In the resources blocks you give the information of what you want to create in your provider(s). Every resource block has its own required parameters, and most have some recommended parameters. It is important to know that the parameters don’t overlap between providers. This means that the code of Azure resources is not the same as code for AWS resources.

Take the following example:

In here you can see we want to make a resource with the type: “azurerm_virtual_network” with the name “main”. In the block we give the information that is needed to create such a resource.

Working with Terraform

Terraform has four main commands with are used to create, manage and destroy resources. These commands are called:

  • Terraform init
  • Terraform plan
  • Terraform apply
  • Terraform destroy

Terraform init

This command is used to initialize the current working directory for use. It downloads any needed provider plugins that is given in the Terraform files (.tf).

Terraform plan

This is the next command used to work with Terraform. In this command the whole syntax is checked for all Terraform files in the current working directory and you will get an overview of all changes that will be made. This is shown in the following way:

Terraform apply

In this command you will apply the changes that were shown in the Terraform plan command. It also shows the changes that are made, just like the Terraform plan, but now you get the option to agree with them. When you agree the changes are made and the output shows progress and feedback when it’s completed.

Terraform destroy

After you make your changes and create your infrastructure you can also destroy it with Terraform with the terraform destroy command. Just like the apply command you will get an overview on the items that will be destroyed and askes you if you agree with it. If you do so, you get feedback on how Terraform is progressing.

 

Some extra features in Terraform

In Terraform there are some extra features that can be used manage or visualize your environment. One of this is the Terraform graph command. This will return a DOT format file which describes your whole Terraform environment. When combined with the “dot” command which is provided by a Linux package called GraphViz you can visualize the environment in an image format. These commands are combined in a pipe as documented below:

terraform graph | dot -Tsvg > graph.svg

You will get an overview in an image like this:

As you can see it is very good for seeing dependencies and viewing the size of your infrastructure.

Another command is called terraform taint. This makes sure to “taint” a resource of your infrastructure and will be remade after an apply even if there will be no changes to that resource. This has an use case where you can “reset” your machines by recreating them and thus restoring them to base settings or deploying updates.

 

Wrapping up:

With Terraform is it possible to create, manage and destroy infrastructure on different platforms with only one Infrastructure-as-Code tool. Terraform also has a clear focus on the infrastructure layer. This means that it has capabilities such as showing changes, managing the creation at the providers and having one language to manage your infrastructure in your on-premises and clouds. Next week we will start building infrastructure on the Microsoft Azure Platform, so stay tuned for that! 😁

15 oktober 2019
Ramon de Goede