Automating stopping and starting of AWS EC2 instances using Amazon Data Pipeline

AWS provides a service called Data Pipeline that can be used to schedule compute resources such as EC2 machines among other things.

This tutorial will demonstrate how to keep a VM running within a specific timeframe - for example the VM should be available from 9am in the morning and then shutdown at 5.30pm and the next day come back online at 9am.

This methodology can in some scenerios save some money - although bare in mind that (according to Amazon) when a command is executed a micro VM instance is launched to process the command - and this is equivilent to 100 mins of runtime compared to just leaving the other VM turned on (assuming it was a micro vm instance.) So ensure that if you are doing it for cost saving that there is a considerable time between starting and stoping the VM.

We should start by by creating a new role:

Roles >> 'Create New Role'

Role Name: DataPipelineDefaultResourceRole'
Role Type: Amzon EC2 Role for Data Pipeline

and then hit 'Create Role'.

We should then create a new policy:

Policy >> Create Policy >> 'Create Your Own Policy'

Policy Name: DataPipelineDefaultResourceRole_EC2_Policy

Policy Document:

{
     "Version": "2012-10-17",
     "Statement": [
          {
               "Effect": "Allow",
               "Action": [
                    "s3:*",
                    "ec2:Describe*",
                    "ec2:Start*",
                    "ec2:RunInstances",
                    "ec2:Stop*",
                    "datapipeline:*",
                    "cloudwatch:*"
               ],
               "Resource": [
                    "*"
               ]
          }
     ]
}

Then click 'Validate Policy' and finally 'Create Policy'.

Then go back to the policies list and find you newly created policy >> select it and click 'Policy Actions' >> Attach >> select the role we create above: 'DataPipelineDefaultResourceRole'.

Proceed by going back to 'Roles' >> 'Create New Role':

Role Name: DataPipelineDefaultRole
AWS Service Role: AWS Data Pipeline

and hit 'Create.'

We should now go to the Amazon Data Pipeline console >> Create Pipeline >>

Name: Start <VM Name>
Source: 'Build using a template' >> 'Run AWS CLI command'
AWS CLI Command: aws ec2 start-instances --instance-ids i-04868gk78 --region eu-west-1
Logging: (optional)
IAM Roles >> Custom: Pipeline Role = DataPipelineDefaultRole and EC2 Instance Role: DataPipelineDefaultResourceRole.

And hit 'Activate' !

Repeat the process for a 'Shutdown' Event.