ChatOps Journey with Ansible, Hubot, AWS and Windows - Part 1
Comparing to DevOps, ChatOps, a word coined by GitHub, is trying to leverage Chatbots to make developers' life much easier. Comparing to CLI or web pages, ChatBots are more user-friendly to interact with, and bots are COOL!!!
This series of posts takes you through the process of setting up a Chatbot for deploying artifacts to AWS EC2 Windows instances. Tools and services used in this post include:
This series assumes you have basic knowledge of Ansible and Docker. If not, you should start from tutorials for these two.
Update 2018-02-08: Use 7zip to extract
tar.gz
files.
Launch and configure instances
Why Windows?
Comparing to Linux, Windows automation is generally harder and lacks documentation. Once you can get Windows done, Linux automation is just a piece of cake. The general concept of ChatOps applies to both Windows and Linux.
Configure Ansible with EC2
You can choose to install Ansible on your local machine, but a better choice is to use Docker. I created my own Dockerfile
for the whole project, which is also prepared for deployment to AWS ECS. The directory ansible
contains all the contents for Ansible. All the files are copied to /etc/ansible
. The file ec2.py
is the dynamic inventory file for EC2. An ec2.ini
file should be placed next to ec2.py
to configure EC2 access. You can put AWS access key and secret key in the ec2.ini
file, but it's not recommended. A better choice is to use environment variables and pass them to the Docker container.
FROM williamyeh/ansible:ubuntu16.04
RUN pip install --upgrade pip && \
pip install boto3 && \
pip install boto
ADD ansible /etc/ansible
RUN chmod +x /etc/ansible/ec2.py
Windows version
You should use at least Windows Server 2012 R2 version for better compatibility. Windows Server 2008 has some issues when executing PowerShell scripts using win_shell
. You may encounter "Out of memory" error or StackOverflowException
, and we cannot configure the max memory using MaxMemoryPerShellMB
. This is a known issue. If Windows Server 2008 is required, make sure the patch is applied, see here for more details. This guide is tested on Windows Server 2012 with AMI ami-2013f142
.
Configure Windows
Windows instances need to be configured to be used with Ansible. This is done by using user data that configures the WinRM.
The user data script listed below sets the administrator's password and invokes the PowerShell script to configure WinRM supports for Ansible.
<powershell>
$admin = [adsi]("WinNT://./administrator, user")
$admin.PSBase.Invoke("SetPassword", "{{ win_initial_password }}")
Invoke-Expression ((New-Object System.Net.Webclient).DownloadString('https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1'))
</powershell>
Windows instances launched by EC2 have generated random passwords, so we cannot use that passwords for Ansible. win_initial_password
is a fixed password used by Ansible. Make sure that you use a strong password that satisfies Windows' requirement.
Please note, storing the Windows password in user data has some security issues. If this does become a problem, you can explore other options. For the use case described in this guide, I use user data because it's the simplest solution and good enough.
The user data can be safely removed after the instance is booted. However, Ansible doesn't support modifying instance user data after it's launched, see this issue. A possible approach is to use AWS Lambda to clear the user data using scheduled events.
Launch instances
Now I can launch EC2 instances using an Ansible playbook. The playbook below creates the security group for Window servers and launch an instance. All variables are stored in the file group_vars/all.yml
. The security group opens port 3389
and 5986
. 3389
is for Windows Remote Desktop, while 5986
is for WinRM. I use the tag instance_uid
to identify the server. After waiting for the port 5986
to open, I added the new instance to Ansible's hosts list, see the section below. The group win
is for all Windows servers, while the group dev
is for this instance only. The file secret.yml
contains the password for Windows servers.
build_num
is an extra variable passed to the playbook as the build number of deployable artifact. build_num
is used to generate the URL for the build.
---
- hosts: localhost
vars_files:
- secret.yml
gather_facts: no
tasks:
- name: create ec2 windows server security group
ec2_group:
name: "{{ aws.windows_security_group }}"
description: Windows server
region: "{{ aws.region }}"
rules:
- proto: tcp
from_port: 3389
to_port: 3389
cidr_ip: 0.0.0.0/0
- proto: tcp
from_port: 5986
to_port: 5986
cidr_ip: 0.0.0.0/0
rules_egress:
- proto: -1
cidr_ip: 0.0.0.0/0
register: sg_out
- name: launch windows ec2 instance
ec2:
region: "{{ aws.region }}"
key_name: "{{ aws.key }}"
instance_type: "{{ aws.instance_type }}"
spot_price: "{{ aws.windows_spot_price }}"
image: "{{ aws.windows_image }}"
group_id: "{{ sg_out.group_id }}"
wait: yes
instance_tags:
Name: My windows server
instance_uid: my_windows_server
build_num: "{{ build_num }}"
role: dev
exact_count: 1
count_tag:
instance_uid: my_windows_server
user_data: "{{ lookup('template', 'templates/userdata.txt.j2') }}"
register: ec2
- name: wait for windows server to answer on all hosts
wait_for:
port: 5986
host: "{{ item.public_ip }}"
timeout: 300
with_items: "{{ ec2.tagged_instances }}"
- name: add windows hosts to groups
add_host:
name: "win-{{ item.id }}"
ansible_ssh_host: "{{ item.public_ip }}"
groups: win, dev
changed_when: false
with_items: "{{ ec2.tagged_instances }}"
- name: add cname
route53:
state: present
zone: mycompany.com
record: "app-{{ build_num }}.mycompany.com"
type: CNAME
value: "{{ item.public_dns_name }}"
ttl: 30
overwrite: yes
with_items: "{{ ec2.tagged_instances }}"
Ansible hosts
In the Ansible hosts
file below, I defined a group win
for all Windows servers. Ansible is configured to use winrm
for connections and use the password win_initial_password
set in the user data.
localhost ansible_connection=local ansible_python_interpreter=python
[win]
[win:vars]
ansible_connection=winrm
ansible_ssh_port=5986
ansible_ssh_user=Administrator
ansible_ssh_pass={{ win_initial_password }}
ansible_winrm_server_cert_validation=ignore
Install artifacts
After the server is launched, I start installing the artifact. For Windows servers, it's an .exe
file or a .msi
file. I add a new role app
for the artifact.
Download S3 files
The artifact files are stored in AWS S3, so I need to download them first.
AWS SDK configuration
To download installation files from S3, I need to use the AWS SDK for .NET. The SDK is already installed for Windows instances launched on AWS EC2. If you are using your own servers, use win_package
to install it first, see below.
Note: Not required for AWS instances.
The AWS SDK installer can be downloaded using win_get_url
with the url http://sdk-for-net.amazonwebservices.com/latest/AWSToolsAndSDKForNet.msi and installed using win_package
. MSI installers support silent installations using /quiet /qn
. It's a good practice to use creates_path
to check the existence first.
- name: install aws tools
win_package:
path: http://sdk-for-net.amazonwebservices.com/latest/AWSToolsAndSDKForNet.msi
arguments: '/quiet /qn /le Z:\aws-tools-installlog.txt'
product_id: '{186A6440-3AD4-4E0E-ACC2-C098CA589290}'
state: present
creates_path: 'C:\Program Files\Amazon\Ec2ConfigService'
After AWS SDK is installed, it needs to be configured with access key and secret key. Those two keys are looked up from environment variables using lookup
. Below is the template to create the PowerShell script for configuration.
Set-AWSCredential -AccessKey {{lookup('env', 'AWS_ACCESS_KEY_ID')}} -SecretKey {{lookup('env', 'AWS_SECRET_ACCESS_KEY')}} -StoreAs default
Set-DefaultAWSRegion -Region {{aws.region}}
The script is copied to remote server using win_template
and executed using win_shell
. These tasks are configured in the role aws
.
- name: copy aws config
win_template:
src: aws_config.ps1.j2
dest: 'Z:\aws_config.ps1'
- name: config aws sdk tools
win_shell: Z:\aws_config.ps1
Download from S3
The actual download from S3 is done using the Read-S3Object
command. Key
is the path to the file to download, starts with /
. File
is the local path to save the file.
- name: download artifact
win_command: "PowerShell.exe -Command Read-S3Object -Region {{aws.region}} -BucketName <bucket_name> -Key <key> -File {{ target_path }} "
Unzip tar.gz file
Update: Due to an issue with Codeplex, currently pscx cannot be installed using Chocolatey. So I updated to use 7zip instead.
Use pscx
The deployable artifact file is in tar.gz
file. To unzip tar.gz
file using the win_unzip
module, PowerShell Community Extensions, Pscx is required. I used Chocolatey to install pscx. I added following line in user data template userdata.txt.j2
to install Chocolatey.
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
The task install pscx
installs pscx
using win_chocolatey
.
- name: install pscx
win_chocolatey:
name: pscx
state: present
Now I can extract tar.gz
files using win_unzip
.
Use 7zip
To use 7zip to extract tar.gz
files, 7zip needs to be installed using Chocolatey first.
- name: install 7zip
win_chocolatey:
name: 7zip
state: present
We cannot use win_unzip
to extract tar.gz
files now, win_command
is required. 7zip extracts the tar.gz
file to tar
file first, so we need two commands to extract the actual contents.
- name: unzip tar.gz
win_command: "7z x {{ tar_gz_path }} -o{{ tar_path }} -y"
- name: unzip tar
win_command: "7z x {{ tar_path }} -o{{ extracted_path } -y"
Find the installer file
Sometimes the installer file may not have a fixed name. It's typical to have a dynamic version number suffix. I use win_find
to find the exe
files in the extracted directory.
- name: find exe path
win_find:
paths: "{{ extracted_path }}"
patterns: "*.exe"
register: exe_find
Here I only expect one file, so I can use exe_find.files[0].path
to reference path of the found file. The path can be used in win_package
to run the installer.
The installer is created using install4j, I use -varfile
to provide a response file for all installation options. The generated response file is found in the .install4j
directory inside the installation directory and is named response.varfile
. You can run the installer manually and use the generated response.varfile
as the start point to modify those options.
- name: install app
win_package:
path: "{{ exe_find.files[0].path }}"
arguments: "-q -overwrite -varfile {{ app.install_varfile_path }}"
product_id: {{ app.product_id }}
state: present
creates_path: Z:\app
To install the artifact, I add following to the playbook. hosts
points to the dev
host added before. roles
has aws
for configuring AWS SDK tools, app
for installing the artifact.
- name: install app
hosts: dev
gather_facts: no
vars_files:
- secret.yml
roles:
- aws
- app
Configure Firewall
Usually we need to configure firewall rules to open certain ports. This is also done using user data.
In the code below, I open the port 8080
. This is also included in the template file userdata.txt.j2
.
netsh advfirewall firewall add rule name="App Port 8080" dir=in action=allow protocol=TCP localport=8080
Run the playbook
To run the playbook, I use ansible-playbook -i hosts app.yml
for local testing. To run playbook inside of the Docker container, I used docker run -rm -it --name=ops -e "AWS_ACCESS_KEY_ID=<access key>" -e "AWS_SECRET_KEY_ID=<secret_key>" ansible-playbook -i hosts app.yml
. AWS access key and secret key are passed as environment variables using -e
.
The extra variable build_num
can be passed using --extra-vars
, e.g. --extra-vars "build_num=100"
.
Pass variables between hosts
In the playbook, instances are launched in the host localhost
, while artifacts are installed in the host dev
. It's possible that the installation of artifacts requires some information about the instances. For example, we may need to use the private IP address of the instance to configure the installation. Sharing variables between different hosts can be done using hostvars
.
In the code below, I passed the variable ip
with the value from registered variable ec2
in the host localhost
.
- name: install app
hosts: dev
gather_facts: no
vars:
ip: "{{ hostvars.localhost.ec2.tagged_instances[0].private_ip }}"
vars_files:
- secret.yml
roles:
- aws
- app
That's all for Part 1. In the next Part 2, I'll discuss Hubot and AWS ECS.