ChatOps Journey with Ansible, Hubot, AWS and Windows

This is Part 3 of the series of setting up a Chatbot for deploying artifacts to AWS EC2 Windows instances. In this post, I'll discuss using AWS Lambda to make sure instances are not left running for too long to save money.

The Chatbot makes deployment much easier and encourages developers to run more tests. However, those launched instances are more likely to be left running for a long time. The situation is worse for Windows instances, which are way more expensive that Linux instances ($0.15 comparing to $0.03). So I need a way to terminate those instances when not used.

Here is my solution. I asked developers to specify the time-to-live (TTL) for each deployment with a default ttl of 60 minutes. I added an extra argument ttl and passed it to Ansible playbook. The ttl value is added as a tag to the EC2 instance. I used an AWS Lambda scheduled event to trigger the update of this tag value. When the tag value reaches 0, the instance is terminated. If the developer wants to prolong the deployment, the tag can be updated manually using AWS console, or a new Chatbot command can be developed to do that.

Below is the AWS Lambda JavaScript code to run the check and update tags. All instances created by Ansible have the tag managed_by with value ansible. This function uses AWS JavaScript SDK to check instances. describeInstances lists all running instances managed by Ansible. For each found instance, it checks the value of tag ttl and determines whether the instance should be terminated. Instances are terminated using terminateInstances, while tags are updated using createTags.

function findTag(tags, key) {
  for (var i = tags.length - 1; i >= 0; i--) {
    if (tags[i].Key === key) {
      return tags[i].Value;
    }
  }
}

exports.handler = (event, context, callback) => {
    var AWS = require('aws-sdk');
    var ec2 = new AWS.EC2();
    var params = {
      DryRun: false,
      Filters: [
        {
          Name: 'tag:managed_by',
          Values: [
              'ansible'
          ],
        },
        {
          Name: 'instance-state-code',
          Values: [
            '16',  
          ]
        }
      ]
    };
    
    ec2.describeInstances(params, function (err, data) {
      if (err) {
        context.fail(err);
      } else {
        var instancesToTerminate = [];
        var instancesToUpdate = [];
        var response = [];
        if (data.Reservations && data.Reservations.length > 0) {
          for (var i = data.Reservations.length - 1; i >= 0; i--) {
            var instances = data.Reservations[i].Instances;
            if (instances && instances.length > 0) {
              for (var j = instances.length - 1; j >= 0; j--) {
                var instance = instances[j];
                var ttl = parseInt(findTag(instance.Tags, 'ttl'), 10);
                if (ttl <= 0) {
                  instancesToTerminate.push(instance.InstanceId);
                } else {
                  instancesToUpdate.push({
                    id: instance.InstanceId,
                    ttl: ttl
                  });
                }
              } 
            }
          }
        }
        if (instancesToTerminate.length > 0) {
          var params = {
            InstanceIds: instancesToTerminate,
            DryRun: false,
          };
          ec2.terminateInstances(params, function(err, data) {
            if (err) {
              console.log(err, err.stack);
              context.fail(err);
            } else {
              context.succeed("Terminated instances: " + instancesToTerminate);
            }
          });
        }
        else {
          var tasksNum = instancesToUpdate.length;
          response.push('No instances to terminate');
          for (var i = instancesToUpdate.length - 1; i >= 0; i--) {
            var toUpdate = instancesToUpdate[i];
            function doUpdate(toUpdate) {
              ec2.createTags({
                Resources: [
                  toUpdate.id 
                ],
                Tags: [
                  {
                    Key: 'ttl',
                    Value: (toUpdate.ttl - 1) + ''
                  }  
                ]
              }, function(err, data) {
                tasksNum--;
                if (err) {
                  console.log(err, err.stack);
                  context.fail(err);
                } else {
                  response.push('Updated ttl of ' + toUpdate.id + ' to ' + (ttl - 5));
                }
                if (tasksNum <= 0) {
                  context.succeed(response.join(', '));
                }
              });
            }
            doUpdate(toUpdate);
          }
        }
      }
    });
};

Note: This Lambda function can only terminate instances or update tags in one run, it cannot do both. This is because async programming with AWS Lambda is not that easy. Of course, you can use the library async and upload the packaged code to AWS. But that's too much for a simple task like this. You can improve the script yourself for more advanced usage.

ChatOps Journey with Ansible, Hubot, AWS and Windows - Part 3

Support Me

Ads