Automation using Ansible

One of the important aspects of web development is availability. The platform must be available to the authorized user whenever requested. However, even when all best practices are followed, there are unanticipated events that lead to downtime. During such events you have few common choices:

  1. You find the cause of the downtime and till then let the platform be down. This may take from a few minutes if the problem is easy to solve, to hours if it is complicated, which incur a loss by the business.
  2. You can have redundant resources. This choice would cost more as you need to have extra resources than actually required.
  3. If you don’t want to spend your budget on extra resources, you would manually spawn resources and set up the environment when such a mishap event occurs. This choice would help to save the budget but may increase the risk of misconfiguration.
  4. You can use Kubernetes to automate the deployment process. However, the issue with this is that it will add an overhead (setup/maintenance/expertise) and would not be suitable for an application with small/medium traffic.
  5. You can have an automated script that would spawn the resources and set up the required environment whenever needed. This choice saves not only the expense of having extra resources but also helps to avoid the chances of misconfiguration lead by hasty environment setup.

In this article, we will explain how to automate the general deployment process in AWS using Ansible. Ansible is an open-source configuration management tool. Here, we will follow source-replica architecture where the source will be a Linux EC2 instance (spawned using AWS console) on which we will have Ansible installed. In this source instance, we will write a playbook that will have plays for the following:

  1. Spawn a Linux EC2 instance (which will be a replica instance), create SSH to the replica, and Application Load Balancer (ALB) target group
  2. Install requirements on the replica- AWS CLI, Git, Docker, Docker-compose, etc.
  3. Clone the private git repository by establishing an SSH.
  4. Run an application’s docker-compose.yml.
  5. Create ALB and finally updating an existing Route53 record so it points to the new setup.

In order to install Ansible check out this link https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html. This playbook comprises of different plays to achieve the setup. In brief, a playbook is usually written in YAML and can consist of multiple plays, which in turn can define multiple tasks. The YAML file starts with [ — -]. In a play, we use:

  • “name” — To name the play/task which is shown in the output during playbook execution.
  • “hosts” — To specify names of the hosts/groups from the inventory on which the tasks are to be executed.
  • “become” — A privilege escalation module or execute play as a different user.
  • “var_files” — The path of the files containing values for variables used in play.
  • “tasks” — They consist of the list of commands or instructions to be executed on the specified host/group.

To know more about how to work with Ansible visit https://docs.ansible.com/ansible/latest/user_guide/index.html.

PLAY — 1

In the first play of the playbook, the first task is to launch an EC2 instance using the “ec2” ansible module. Note that it is good practice to assign IAM roles to the resources, so as to have control over what it can access. Here we use the “register” ansible keyword to record the output of the task.

---
# FIRST PLAY
- name: Launch EC2 instance, add new host, create SSH and ALB target group
hosts: localhost
vars_files:
- group_vars/credential.yml #File path where AWS credentials are stored
tasks:
# FIRST PLAY - Task 1
- name: Launch EC2 Instance
ec2:
aws_access_key: "{{ aws_access_key_id }}"
aws_secret_key: "{{ aws_secret_access_key }}"
group: ansible-sg #Security group
count: 1 #Number of instances to be launched
instance_type: t2.micro #Type of instance to be launched
image: ami-00b243003d0e9e258 #Linux instance
wait: true #Wait for instance to be in required state
region: us-east-1
zone: us-east-1a
key_name: source-key #Key name replica instance will be using
vpc_subnet_id: subnet-12345678 #Subnet id
assign_public_ip: yes
instance_profile_name: EC2Role #IAM Role
instance_tags:
Name: ansible-replica #Name for the instance
register: ec2 #Save the returned values

The next task is to create a record of new instances in the inventory and provide necessary details that are required to establish an SSH connection. We will use the “add_host” ansible module to achieve this and use the “loop” ansible keyword to iterate through all the instances.

# FIRST PLAY - Task 2
- name: Add new instance to host group
add_host:
hostname: "{{ item.public_ip }}"
ansible_host: "{{ item.public_ip }}"
groups: ansible_hosts #Group name where instance will belong
ansible_ssh_private_key_file: '/path/to/ansible/replica/key.pem'
ansible_ssh_user: ec2-user #User name that will be used while connecting
loop: "{{ ec2.instances }}" #Loop to add all the newly created instances

In the next task, we will use the “wait_for” ansible module. We wait for 10 seconds before checking for instance port 22 to be opened and keep checking for 320 seconds until its available.

# FIRST PLAY - Task 3   
- name: Wait for SSH
wait_for:
host: "{{ item.public_dns_name }}"
port: 22
delay: 10 #Wait for 10 seconds before checking port availability
timeout: 320 #Keep checking for 320 seconds
state: started #Until port is open
loop: "{{ ec2.instances }}"

The last task is to create a target group for our ALB using the “elb_target_group” ansible module, so it can route traffic to our newly created instance.

# FIRST PLAY - Task 4
- name: Create ALB target group
elb_target_group:
aws_access_key: "{{ aws_access_key_id }}"
aws_secret_key: "{{ aws_secret_access_key }}"
name: "ansible-target-group" #Name for the target group
target_type: instance
protocol: "http"
region: "us-east-1"
port: 80
vpc_id: vpc-12345678 #VPC ID
targets:
- Id: "{{ item.id }}"
Port: 80
state: "present"
loop: "{{ ec2.instances }}" #Include all newly created instances

Finally, if we put all the tasks together we get our first play. A common error that can be faced while running this play is “msg”: “boto required for this module”

One of the solutions is to add the following in the “hosts” file, this solution was found at this forum https://github.com/ansible/ansible/issues/15019

[localhost]
localhost ansible_connection=local ansible_python_interpreter=python

PLAY — 2

In the second play, we install the required configuration on the new instance. Note that now the value of “hosts” is “ansible_hosts” (group name specified while recording new instance in inventory). For installing the requirements, we can either use ansible module like yumapt, etc. or use terminal command and pass them as value to the “command” ansible module. One of the differences is that the “yum/apt” module is more efficient for installing packages than using the “command” module to do so.

# SECOND PLAY 
- name: Install requirements on EC2 instance
hosts: ansible_hosts
gather_facts: True
become: True #To allow privilege escalation
tasks:
- name: Install awscli
command: pip install --upgrade awscli
- name: Install docker #Install docker using the usual command
command: yum install docker -y
- name: Install git #Install git using yum ansible module
yum:
name: git #Install git using yum
state: latest #Latest version

We can also use the “shell” ansible module to run multiple commands through the shell as follows

- name: Install docker-compose
shell: |
curl -L "https://github.com/docker/compose/releases/download/1.20.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
group add docker
usermod -aG docker ec2-user
export DOCKER_HOST=127.0.0.1:2375
service docker start

Now, by combining both the snippets we get a complete second play. An error that was faced while running this play was

ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

The error was solved by adding the following, found at the forum https://github.com/docker/compose/issues/4181

usermod -aG docker ec2-user    #Add docker group to current user
export DOCKER_HOST=127.0.0.1:2375 #Explicitly set docker host

PLAY — 3

The third play will clone the private git repository as follows,

# THIRD PLAY 
- name: Clone git repository
hosts: ansible_hosts
tasks:
- name: Copy git SSH keys
copy: src=/home/ec2-user/.ssh/ dest=/home/ec2-user/.ssh/ mode=0700
sudo: true
- name: Change directory permission
command: sudo chmod -R 700 /home/ec2-user/.ssh/
- name: Enable SSH agent
shell: |
eval "$(ssh-agent)" #Start SSH agent
export SSH_AUTH_SOCK
export SSH_AGENT_PID
ssh-add
- name: Clone a private repository
shell: git clone git@gitlab.com:path/to/application.git
- name: shred git keys
shell: |
shred -zuv -n 5 /home/ec2-user/.ssh

The pre-requisite for this play is to have git SSH keys in the “~/.ssh” folder. First, we copy SSH keys to replica using the “copy” ansible module and then set up permission. Note that if you are copying individual files (from source-ansible) then the destination (on replica) must also be a file. Here we are coping folder (from source-ansible) so even the destination (on replica) is a folder. Then, we enable and create an SSH connection to clone the git repository. Finally, we permanently remove SSH keys from the volume store using “shred”. Note that just by using the “rm” command would not be sufficient as the keys are still present on the storage volume until that space is overwritten by some other data. So by using “shred” we delete and overwrite the space utilized to store keys with zeros.

PLAY — 4

Next play will traverse to the directory containing the docker-compose.yml file and do docker-compose up.

# FOURTH PLAY
- name: Run the application
hosts: ansible_hosts
gather_facts: True
tasks:
- name: Do docker-compose up
shell: |
cd /home/ec2-user/git-repository/ #Path where docker-compose.yml is present
docker-compose up -d

PLAY — 5

The final play first creates an ALB with HTTP and HTTPS listeners using the “elb_application_lb“ ansible module as follows

# FIFTH PLAY
- name: Create ALB and modify the existing Route53
hosts: localhost
vars_files:
- group_vars/credential.yml
tasks:
# FIFTH PLAY - Task 1
- name: Create ALB
elb_application_lb:
aws_access_key: "{{ aws_access_key_id }}"
aws_secret_key: "{{ aws_secret_access_key }}"
name: ansible-alb
region: us-east-1
security_groups:
- ansible-sg
subnets:
- subnet-12345678
- subnet-91011121
listeners:
- Protocol: HTTP
Port: 80
DefaultActions:
- Type: forward
TargetGroupName: ansible-target-group #Same name given while creating it
- Protocol: HTTPS
Port: 443
SslPolicy: ELBSecurityPolicy-2016-08
Certificates:
- CertificateArn: arn:aws:acm:us-east-1:123:certificate/456
DefaultActions:
- Type: forward
TargetGroupName: ansible-target-group
purge_listeners: no
state: present
register: elb

The next task then updates an existing Route53 record. Note that the value of the “value” parameter in the “route53” ansible module is not just the DNS name of ALB, rather it is a concatenation of the word “dualstack” and the DNS name of ALB.

# FIFTH PLAY - Task 2
- name: Modify A record in route53
route53:
aws_access_key: "{{ aws_access_key_id }}"
aws_secret_key: "{{ aws_secret_access_key }}"
state: present #Existing record
overwrite: yes #Overwrites the existing record if found
zone: grad4-demo.com
record: grad4-demo.com
type: A
value: "dualstack.{{ elb.dns_name }}" #concatenates dualstack and ALB DNS name
alias: True
alias_hosted_zone_id: "{{ elb.canonical_hosted_zone_id }}"

Now combine both tasks to get the last play. Finally, to execute the playbook we can use the command

ansible-playbook <playbook-name.yml> -vvvv

Here, the number of v’s depend on how detailed the output you need. You can also dry run (show the expected output without actually running it) the playbook to check for errors using either of the following commands

ansible-playbook <playbook-name.yml> --syntax-check 
ansible-playbook <playbook-name.yml> --check
ansible-playbook <playbook-name.yml> -C

Notes:

  • After installation, if you don’t get any ansible files/folders, create a folder named “ansible” at the “/etc” location. After creating this folder, create an “ansible.cfg” file (where configuration setting can be defined), a “hosts” file (it is inventory). If you want to setup ansible vault you can visit the link https://serversforhackers.com/c/how-ansible-vault-works (If you don’t want to pass the vault password file path while executing playbook, you can set “vault_password_file = /path/to/vault_password_file“ in ansible.cfg)
  • In order to completely erase the git keys from the store volume, we have used the “shred” library. However, “shred” might not be the best option for other configuration systems as it depends on the way the file system overwrites data.

GRAD4

At GRAD4, we use the above-described technology to avoid any unanticipated platform downtime so that it is always accessible to the authorized user. We also use it to manage and patch our systems on a regular basis, as well as to have a secure environment configuration during any unfortunate event by avoiding any system misconfiguration. We aim to provide the best customer service. To achieve it, we follow best-recommended practices and also incorporate new technologies to avoid creating any inconvenience to our users and also enhance user satisfaction.

An Article by Apurva Shah

Leave a Reply

Your email address will not be published. Required fields are marked *

Utilisez le formulaire
ci-dessous pour nous écrire

Submit my application

Automation using Ansible