Developing a CI/CD pipeline to provision an AWS infrastructure using Terraform, GitHub, and Ubuntu 20.04
Continuous integration (CI) and continuous deployment (CD) have evolved alongside the DevOps culture. It has introduced powerful tools allowing more efficient collaboration, faster time-to-market, and effective management of downtime. With just a few steps, you can commission your infrastructure in multiple cloud environments and decommission them when they are not required.
This post is meant as an “exercise” for introductory exposure to AWS, Terraform, GitHub & Linux — it does not represent best practices
This blog uses an Ubuntu OS to demonstrate how to create Terraform templates on Microsoft Visual Code studio and push these to GitHub for source control. With just a few steps, it illustrates how Terraform can be used to provision a pre-baked Ec2 instance, running Apache2 webserver on AWS, and the ease at which it can be decommissioned. The post also upholds aspects of reverting to a previous stable state when there is an accidental/unintentional bug in the code.
1. Setting up Terraform and Visual Code Studio
a. Navigate to https://www.terraform.io/
b. Download, unzip, and install the version that is suitable for your operating system (I have prepared an Ubuntu virtual machine using a hosted hypervisor.)
c. Download Visual Studio Code from https://code.visualstudio.com/. It is an extremely handy tool for writing code in any programming language, YAML/JSONscripts, or in our case, terraform scripts.
I. Switch to the Visual Studio Code window and download the desired extension to enjoy a seamless experience.
2. Set up a Terraform (TF) Cloud account. After setting up the account, it should prompt you to:
a. Create an organization for the project.
I. The organization name must be unique
b. TF will take you through the process of creating a Version Control Workflow
I. Since we are beginner users, we will select the first option.
II. This will enable Terraform configuration files, shared variables, current and previous TF states to be stored in Source Control Management (SCM) system like GitHub
III. If you do not have a GitHub or GitLab account, you must set one up. It is free and one of the key tools that you need to learn in your DevOps career. I used online videos and training documents that I found on the Internet to learn about git.
IV. When creating the repository, I like to initially set it as Private. This is because, during learning and testing, we might accidentally upload credentials which could lead to a massive security breach if the repository is set to Public.
V. Take a note of the default branch and change the settings to modify the name.
VII. Now, as the repository is private and we will “SSH” into it using the guest virtual machine in our local system, we must generate a key pair and enter the public key into our GitHub settings
i. Navigate to terminal
ii. RUN ssh-keygen : This will create a key pair
iii. Keep pressing Enter as we will accept the defaults since this is a demonstration. In a production setting, we would need to change these to fit our requirements.
iv. RUN cat .ssh/id_rsa.pub : This will print the contents of the public key on the screen
v. Copy this and navigate to Github
vi. Click the dropdown menu next to your picture/account on the top right- hand side of the window and navigate to Settings > SSH and GPG Keys
vii. Select New SSH Key. Paste the key and give it a meaningful name
viii. Switch to Terraform > Choose Github.com from the dropdown > Select Authorize Terraform Cloud
ix. If the web browser prompts you to enable popups, go ahead and do that since we trust this website.
x. Select the repositories that you created and press Install
xi. You may be required to select and authorize GitHub.com again. Once you do it, you will see your profile and the repositories that you selected appear. Choose one of them to move forward.
xii. Now in this demonstration, we will not go into the discussion about branches. We will keep it simple and handle everything using the master branch.
xiii. At a high-level though, GitHub branches allow you to create a hierarchical structure. This opens up working on numerous branches to prevent the master branch from becoming cluttered.
xiv. Provide a meaningful Workspace Name and proceed to create a workspace. For consistency, I have kept it the same as the repository name.
3. The workspace will prompt us to configure variables.
a. This is a crucial step because it allows the templates that we upload to our SCM to call the AWS API and provision the infrastructure. It also keeps the access keys secure and allows us to avoid hard-coding them into our templates (which is always a bad practice)
b. Log into your AWS account and set up a user who will be in charge of provisioning infrastructure.
I. Always follow the principle of least privilege and adjust if required later. Since we will be deploying VPC and Ec2 components, the following permissions should do just fine.
II. Proceed to review and create a user.
III. Download the .csv file to a secure location. Check that you are alone and open the file.
IV. Switch to your Terraform Cloud account and choose Configure variables.
V. Scroll down to Environment variables > Add variable
vi. Enter the key from below but choose the value from the .csv file that you downloaded. We will add two variables and always check the Sensitive box on the right side.
1. Key: AWS_ACCESS_KEY_ID
Value: <iam_user_access_id>
2. Key: AWS_SECRET_ACCESS_KEY
Value: <iam_user_secret_key>
vii. Navigate to the folder where you saved the .csv file and permanently delete it. It is always bad practice to store credentials on the localhost.
4. Switch to your guest VM and set up Git
a. An extremely useful and powerful tool when using any Linux-based distributions are the man pages. If you are confused about the commands or the options, look at the man pages. An example command to run is:
I. RUN man apt
b. We are using an Ubuntu machine, so we need to install git first
I. RUN sudo apt -y install git
i. This command utilizes the root’s privilege to install the git package and -y instructs it to accept all prompts.
c. Set global variables for the git configuration file. These are used purely for identifying and communicating with the user(s) that are modifying the code.
I. RUN git config — global user.name <name-of-user>
II. RUN git config — global user.email <email-of-user>
i. The commands above set the global values in the git .config file
III. RUN git config --list
i. This command will print the keys and values on the screen
d. First, navigate to the production repository and copy the SSH key
e. Clone the git repository and initiate it in a folder
I. RUN git clone <repository-ssh-value> <name-of-the-directory-that-you-want-to-clone-into>
i. Type yes to any prompts that appear
ii. This command clones the git repository and initiates git on to a directory that you specify
iii. RUN cd <name-of-the-directory-that-repo-was-cloned-into> and navigate into the directory that you created. RUN ls -la to check. You can see the .git directory meaning that git has been initialized
iv. RUN git config --list : You should now see the value of remote.origin.url as the SSH URL for your repository
5. Preparing Visual Studio Code
a. Now, switch to Visual Code Studio which should be prepared with the Terraform extension.
I. Create a folder to store the projects.
II. Create a new file
III. Name it with a .tf extension
i. The .tfis the file extension for Terraform
b. The default syntax for Terraform templates are:
resource “<provider-name>_<resource-type>” “terraform-name” {
Key1 = value1
Key2 = value2
}I. Example:
resource “aws_vpc” “<name-of-vpc-here>” {
cidr_block = “10.0.0.0/16”
}i. Having this in mind will help you understand the .tf template better
ii. If you are confused and/or not sure how to proceed, a great place to start is the Terraform documentation. It does an excellent job of explaining the syntax.
6. Configuring Terraform template for production
a. Terraform versions 0.13 and later require this part
b. Configure the provider as AWS and specify the region. For this demo, I have used us-east-1
c. Create a custom VPC that will contain our resources. Also, set a CIDR block and assign tags for ease of tracking or reporting.
d. Deploy an internet gateway, which will allow traffic to traverse the Internet. Since this will be attached to our VPC, we need to reference the custom VPC that we created earlier using its ID viaaws_vpc.Project-VPC.id
c. Following this, we need to create a custom route table that will give our VPC knowledge on how to forward traffic. For this demo, we will set up the routing table to point any IPv4 or IPv6 traffic to the internet gateway resource
d. Next, we will deploy our production subnet in the custom VPC. It will contain any resource that we deploy for our production environment. We have specified it to deploy in Availability Zone us-east-1a.
e. If configurations of the AWS infrastructure are not changed, custom subnets are associated with the main route table. Since we created a custom route table and want our subnet to incorporate that, we will associate the custom subnet with the newly created route table.
f. Because we want people to visit our webpages, we will create a security group and later attach it to the network interface that we deploy. This will allow both HTTPS and HTTP traffic on ports, 443 and 80, respectively. The egress portion of the template dictates any traffic in the outbound direction. We have specified it with -1 that means all protocols and the CIDR block as 0.0.0.0/0 meaning any traffic.
g. After that we move on to provisioning a network interface card (NIC) which we will attach to our Ec2 instance. We specify the subnet and private IP. Since Ec2 instances come with a primary elastic network interface (ENI), it dynamically allocates an IP address meaning that if we stop and start the instance a new one will be allocated. Creating a secondary ENI allows us to attach/detach as we please. The following attach a private IP address to our instance.
h. We also need to configure a public IP address to resolve web traffic and attach it to our custom network interface. Because the syntax for Terraform is declarative and since we want to allocate our public IP to a VPC with an Internet Gateway, we specify the depends_on argument to dictate what is deployed first.
i. Finally, we deploy our Ec2 instance. The network_interface block is a parameter that we pass inside the resource object specifying that this is the first interface of the Ec2 instance device_index = 0. We control what happens when the Ec2 instance boots using the user_data argument. It starts with <<-EOF and ends with EOF allowing Terraform to understand where to initiate and finish.
I. The user_data for this script begins by dictating that we want to use the bash shell. We perform updates and install the apache2 webserver. Next, enable and start it up so that every time the Ec2 boots, it will start the apache2 server. Afterward, we define a simple html block and output it to the /var/www/html/index.html file which apache2 initially loads when resolving traffic.
II. Save the project
7. Combining it together to run a test
a. Switch to the guest VM running on your local machine. Navigate to the folder that contains the .tf files you created.
I. RUN pwd
i. This will output the directory that you are on. Copy this onto your clipboard.
b. Now, navigate to the production directory that you initiated git on.
I. RUN cp <paste-copied-directory-here>/<enter-file-name.tf> .
i. This will copy the file that you created on to your current directory which is denoted by .
II. RUN git status
i. The file that we just copied is untracked.
III. RUN git add <name-of-file.tf>
i. Now our file is being tracked. RUN git status to check.
IV. RUN git commit -a -m "A meaningful message here"
i. The file has been committed to the branch on your local guest VM. It is compulsory to specify a message using -m when committing.
V. Finally, RUN git push origin to save our template onto our remote repository on GitHub.
c. Switch to your Terraform Cloud profile. Navigate to the organization that you created and select the production workspace.
I. Select Queue plan
II. Your plan will run and after a while should see that Terraform will deploy some resources onto your AWS. This is just a plan and a great way to check if everything will be properly deployed. Then, we must go ahead and click Confirm & apply. Provide a meaningful message and wait for the Apply finished output.
III. Switch to your AWS console and wait for the Ec2 Status check to show 2/2.
IV. Navigate to your public IP address and you should see the simple webpage that we deployed.
V. To appreciate the usefulness of continuous integration using Terraform and GitHub, let’s say that we need to change the code for our production script from what it was before to:
VI. Switch back to the terminal and run the commands:
i. RUN cp <paste-copied-directory-here>/<enter-file-name.tf> .
ii. RUNgit status : you should see the file has been modified
iii. RUN git commit -a -m "A meaningful message here"
iv. Finally, RUN git push origin
v. Terraform will detect the changes and only provides a new Ec2 instance incorporating the changes. It will see that the rest of the infrastructure is unchanged and so will not bother with it.
vi. Navigate to the public IP address of your Ec2 instance and you should see the modified page.
8. Always be prepared for deployment
a. We will copy the .tf file that we created for production to a safe place as a backup
I. RUN: cp your-prod-file-path/your-prod-file-name.tf> ~/Documents/Backup-of-prod-tf.tf
i. This will ensure that we have a backup copy of our production .tf file, should anything happen to the one that’s being used. It is always a good practice to ensure backups are taken.
b. Now let’s say that there has been an issue with the production landing page and instead of the usual, it is displaying this:
c. Since we have a backup, we can just navigate to our production directory and:
I. RUN cp ~/Documents/Backup-of-prod-tf.tf .
II. RUN git commit -a -m "Your message here"
III. RUN git push origin
IV. Terraform will detect the changes and provision a new Ec2 instance with the previous configurations which will get your web page up and running again.
9. Destroy your environment
a. This part is a destructive process and should be well understood before running it.
I. A great aspect of the environment that we just created is that we can delete it all with just a few simple steps
i. Navigate to Settings > Destruction and Deletion from your Terraform Cloud account.
ii. Then select Queue destroy plan. It will prompt you to enter the name of the Workspace. Do that and proceed.
iii. It will again ask for a confirmation. As you can see, there are a lot of safety checks in place to prevent accidental deletion. Once you understand the consequences of your action, click Confirm and Apply. This will delete all the infrastructure that was provisioned using the Terraform template.
10. Summary
In this demonstration, we have successfully:
i. Set up Terraform
ii. Install Visual Code Studio
iii. Set up a GitHub account
iv. Created a .tf template and uploaded it on GitHub
v. Used Ubuntu terminal to push the code to GitHub and automatically deploy our AWS infrastructure.
vi. Illustrated how continuous integration tools are powerful enough to detect changes in our code and provision the infrastructure accordingly.
vii. Recovered from an error in the code
viii. Deleted our infrastructure
I hope this post was able to demonstrate the power of a CI/CD pipeline coupled with a source control management tool such as GitHub. Imagine having to create numerous instances, in multiple cloud environments (such as Azure, AWS, GCP, etc.). You can automate the process, which drastically reduces the time it takes and reduces errors.
Here is the repository link which I have made public after the demonstration.
I find cloud technology fascinating and will be posting more in-depth exercises as I continue to grow my knowledge.
Thank you for reading! Be sure to take care of yourself and especially each other.
