Comparing to DevOps, ChatOps, a word coined by GitHub, is trying to leverage Chatbots to make developers' life much easier. Comparing to CLI or web pages, ChatBots are more user-friendly to interact with, and bots are COOL!!!

This series of posts takes you through the process of setting up a Chatbot for deploying artifacts to AWS EC2 Windows instances. Tools and services used in this post include:

• Ansible
• Hubot
• HipChat
• Docker
• Amazon EC2 & ECS

This series assumes you have basic knowledge of Ansible and Docker. If not, you should start from tutorials for these two.

Update 2018-02-08: Use 7zip to extract tar.gz files.

Launch and configure instances

Why Windows?

Comparing to Linux, Windows automation is generally harder and lacks documentation. Once you can get Windows done, Linux automation is just a piece of cake. The general concept of ChatOps applies to both Windows and Linux.

Configure Ansible with EC2

You can choose to install Ansible on your local machine, but a better choice is to use Docker. I created my own Dockerfile for the whole project, which is also prepared for deployment to AWS ECS. The directory ansible contains all the contents for Ansible. All the files are copied to /etc/ansible. The file ec2.py is the dynamic inventory file for EC2. An ec2.ini file should be placed next to ec2.py to configure EC2 access. You can put AWS access key and secret key in the ec2.ini file, but it's not recommended. A better choice is to use environment variables and pass them to the Docker container.

FROM williamyeh/ansible:ubuntu16.04

RUN pip install --upgrade pip && \
  pip install boto3 && \
  pip install boto

ADD ansible /etc/ansible
RUN chmod +x /etc/ansible/ec2.py

Windows version

You should use at least Windows Server 2012 R2 version for better compatibility. Windows Server 2008 has some issues when executing PowerShell scripts using win_shell. You may encounter "Out of memory" error or StackOverflowException, and we cannot configure the max memory using MaxMemoryPerShellMB. This is a known issue. If Windows Server 2008 is required, make sure the patch is applied, see here for more details. This guide is tested on Windows Server 2012 with AMI ami-2013f142.

Configure Windows

Windows instances need to be configured to be used with Ansible. This is done by using user data that configures the WinRM.

The user data script listed below sets the administrator's password and invokes the PowerShell script to configure WinRM supports for Ansible.

<powershell>
$admin = [adsi]("WinNT://./administrator, user")
$admin.PSBase.Invoke("SetPassword", "{{ win_initial_password }}")
Invoke-Expression ((New-Object System.Net.Webclient).DownloadString('https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1'))
</powershell>

Windows instances launched by EC2 have generated random passwords, so we cannot use that passwords for Ansible. win_initial_password is a fixed password used by Ansible. Make sure that you use a strong password that satisfies Windows' requirement.

Please note, storing the Windows password in user data has some security issues. If this does become a problem, you can explore other options. For the use case described in this guide, I use user data because it's the simplest solution and good enough.

The user data can be safely removed after the instance is booted. However, Ansible doesn't support modifying instance user data after it's launched, see this issue. A possible approach is to use AWS Lambda to clear the user data using scheduled events.

Launch instances

Now I can launch EC2 instances using an Ansible playbook. The playbook below creates the security group for Window servers and launch an instance. All variables are stored in the file group_vars/all.yml. The security group opens port 3389 and 5986. 3389 is for Windows Remote Desktop, while 5986 is for WinRM. I use the tag instance_uid to identify the server. After waiting for the port 5986 to open, I added the new instance to Ansible's hosts list, see the section below. The group win is for all Windows servers, while the group dev is for this instance only. The file secret.yml contains the password for Windows servers.

build_num is an extra variable passed to the playbook as the build number of deployable artifact. build_num is used to generate the URL for the build.

---
- hosts: localhost
  vars_files:
    - secret.yml  
  gather_facts: no
  tasks:
    - name: create ec2 windows server security group
      ec2_group:
        name: "{{ aws.windows_security_group }}"
        description: Windows server
        region: "{{ aws.region }}"
        rules:
          - proto: tcp
            from_port: 3389
            to_port: 3389
            cidr_ip: 0.0.0.0/0
          - proto: tcp
            from_port: 5986
            to_port: 5986
            cidr_ip: 0.0.0.0/0
        rules_egress:
          - proto: -1
            cidr_ip: 0.0.0.0/0
      register: sg_out
    - name: launch windows ec2 instance
      ec2:
        region: "{{ aws.region }}"
        key_name: "{{ aws.key }}"
        instance_type: "{{ aws.instance_type }}"
        spot_price: "{{ aws.windows_spot_price }}"
        image: "{{ aws.windows_image }}"
        group_id: "{{ sg_out.group_id }}"
        wait: yes
        instance_tags:
          Name: My windows server
          instance_uid: my_windows_server
          build_num: "{{ build_num }}"
          role: dev
        exact_count: 1
        count_tag: 
          instance_uid: my_windows_server
        user_data: "{{ lookup('template', 'templates/userdata.txt.j2') }}"  
      register: ec2
    - name: wait for windows server to answer on all hosts
      wait_for:
        port: 5986
        host: "{{ item.public_ip }}"
        timeout: 300
      with_items: "{{ ec2.tagged_instances }}"
    - name: add windows hosts to groups
      add_host:
        name: "win-{{ item.id }}"
        ansible_ssh_host: "{{ item.public_ip }}"
        groups: win, dev
      changed_when: false
      with_items: "{{ ec2.tagged_instances }}"   
    - name: add cname
      route53:
        state: present
        zone: mycompany.com
        record: "app-{{ build_num }}.mycompany.com"
        type: CNAME
        value: "{{ item.public_dns_name }}"
        ttl: 30
        overwrite: yes
      with_items: "{{ ec2.tagged_instances }}"      

Ansible hosts

In the Ansible hosts file below, I defined a group win for all Windows servers. Ansible is configured to use winrm for connections and use the password win_initial_password set in the user data.

localhost ansible_connection=local ansible_python_interpreter=python

[win]

[win:vars]
ansible_connection=winrm
ansible_ssh_port=5986
ansible_ssh_user=Administrator
ansible_ssh_pass={{ win_initial_password }}
ansible_winrm_server_cert_validation=ignore

Install artifacts

After the server is launched, I start installing the artifact. For Windows servers, it's an .exe file or a .msi file. I add a new role app for the artifact.

Download S3 files

The artifact files are stored in AWS S3, so I need to download them first.

AWS SDK configuration

To download installation files from S3, I need to use the AWS SDK for .NET. The SDK is already installed for Windows instances launched on AWS EC2. If you are using your own servers, use win_package to install it first, see below.

Note: Not required for AWS instances. The AWS SDK installer can be downloaded using win_get_url with the url http://sdk-for-net.amazonwebservices.com/latest/AWSToolsAndSDKForNet.msi and installed using win_package. MSI installers support silent installations using /quiet /qn. It's a good practice to use creates_path to check the existence first.

- name: install aws tools
  win_package:
    path: http://sdk-for-net.amazonwebservices.com/latest/AWSToolsAndSDKForNet.msi
    arguments: '/quiet /qn /le Z:\aws-tools-installlog.txt'
    product_id: '{186A6440-3AD4-4E0E-ACC2-C098CA589290}'
    state: present
    creates_path: 'C:\Program Files\Amazon\Ec2ConfigService'

After AWS SDK is installed, it needs to be configured with access key and secret key. Those two keys are looked up from environment variables using lookup. Below is the template to create the PowerShell script for configuration.

Set-AWSCredential -AccessKey {{lookup('env', 'AWS_ACCESS_KEY_ID')}} -SecretKey {{lookup('env', 'AWS_SECRET_ACCESS_KEY')}} -StoreAs default

Set-DefaultAWSRegion -Region {{aws.region}}

The script is copied to remote server using win_template and executed using win_shell. These tasks are configured in the role aws.

- name: copy aws config
  win_template: 
    src: aws_config.ps1.j2
    dest: 'Z:\aws_config.ps1'
- name: config aws sdk tools
  win_shell: Z:\aws_config.ps1

Download from S3

The actual download from S3 is done using the Read-S3Object command. Key is the path to the file to download, starts with /. File is the local path to save the file.

- name: download artifact
  win_command: "PowerShell.exe -Command Read-S3Object -Region {{aws.region}} -BucketName <bucket_name> -Key <key> -File {{ target_path }} "

Unzip tar.gz file

Update: Due to an issue with Codeplex, currently pscx cannot be installed using Chocolatey. So I updated to use 7zip instead.

Use pscx

The deployable artifact file is in tar.gz file. To unzip tar.gz file using the win_unzip module, PowerShell Community Extensions, Pscx is required. I used Chocolatey to install pscx. I added following line in user data template userdata.txt.j2 to install Chocolatey.

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

The task install pscx installs pscx using win_chocolatey.

- name: install pscx
  win_chocolatey:
    name: pscx
    state: present

Now I can extract tar.gz files using win_unzip.

Use 7zip

To use 7zip to extract tar.gz files, 7zip needs to be installed using Chocolatey first.

- name: install 7zip
  win_chocolatey:
    name: 7zip
    state: present

We cannot use win_unzip to extract tar.gz files now, win_command is required. 7zip extracts the tar.gz file to tar file first, so we need two commands to extract the actual contents.

- name: unzip tar.gz
  win_command: "7z x {{ tar_gz_path }} -o{{ tar_path }} -y"
- name: unzip tar
  win_command: "7z x {{ tar_path }} -o{{ extracted_path } -y" 

Find the installer file

Sometimes the installer file may not have a fixed name. It's typical to have a dynamic version number suffix. I use win_find to find the exe files in the extracted directory.

- name: find exe path
  win_find:
    paths: "{{ extracted_path }}"
    patterns: "*.exe"
  register: exe_find 

Here I only expect one file, so I can use exe_find.files[0].path to reference path of the found file. The path can be used in win_package to run the installer.

The installer is created using install4j, I use -varfile to provide a response file for all installation options. The generated response file is found in the .install4j directory inside the installation directory and is named response.varfile. You can run the installer manually and use the generated response.varfile as the start point to modify those options.

- name: install app
  win_package:
    path: "{{ exe_find.files[0].path }}"
    arguments: "-q -overwrite -varfile {{ app.install_varfile_path }}"
    product_id: {{ app.product_id }}
    state: present
    creates_path: Z:\app

To install the artifact, I add following to the playbook. hosts points to the dev host added before. roles has aws for configuring AWS SDK tools, app for installing the artifact.

- name: install app
  hosts: dev
  gather_facts: no
  vars_files:
    - secret.yml
  roles:
    - aws
    - app

Configure Firewall

Usually we need to configure firewall rules to open certain ports. This is also done using user data.

In the code below, I open the port 8080. This is also included in the template file userdata.txt.j2.

netsh advfirewall firewall add rule name="App Port 8080" dir=in action=allow protocol=TCP localport=8080

Run the playbook

To run the playbook, I use ansible-playbook -i hosts app.yml for local testing. To run playbook inside of the Docker container, I used docker run -rm -it --name=ops -e "AWS_ACCESS_KEY_ID=<access key>" -e "AWS_SECRET_KEY_ID=<secret_key>" ansible-playbook -i hosts app.yml. AWS access key and secret key are passed as environment variables using -e.

The extra variable build_num can be passed using --extra-vars, e.g. --extra-vars "build_num=100".

Pass variables between hosts

In the playbook, instances are launched in the host localhost, while artifacts are installed in the host dev. It's possible that the installation of artifacts requires some information about the instances. For example, we may need to use the private IP address of the instance to configure the installation. Sharing variables between different hosts can be done using hostvars.

In the code below, I passed the variable ip with the value from registered variable ec2 in the host localhost.

- name: install app
  hosts: dev
  gather_facts: no
  vars:
    ip: "{{ hostvars.localhost.ec2.tagged_instances[0].private_ip }}"
  vars_files:
    - secret.yml
  roles:
    - aws
    - app

That's all for Part 1. In the next Part 2, I'll discuss Hubot and AWS ECS.