Tuesday, January 5, 2016

AWS - using SQS to cleanup Active Directory of terminated instances

If your Amazon EC2 instances are part of an Auto Scaling Group and are required to be joined to a Windows Domain, then maintaining a clean Active Directory environment may be an after thought. EC2 instances can be terminated for a variety of reasons, and since they may terminate abruptly their Active Directory objects may not be removed from Active Directory.  The following steps will help you create a process utilizing Amazon's Simple Queue Service (SQS) to remove terminated instances from Active Directory and to help keep a cleaner Active Directory structure.

The following assumes:
  • Your servers are launched using an Auto Scaling Group and are auto joined to an Active Directory domain using the AWS Instance Id as its hostname.  Please see this post for details on how to accomplish this -  Auto Join EC2 instances to domain
  • An EC2 windows instance that has
    • An IAM role assigned to the instance
    • The AWS CLI installed on the instance
    • Access to the Active Directory domain

1. Create the SQS queue

Within AWS, create a new SQS queue.  Be sure to set the message retention period to a value greater than how often you plan to run the scheduled powershell script. We will set the permissions in a later step, after we've created the SNS topic.

2. Create the SNS topic

Create a new SNS topic in AWS and add a subscription to the SNS topic selecting 'Amazon SQS' as the endpoint, ie: arn:aws:sqs:us-east-1:123456789012:SQS-InstanceTerminations

3. Configure the SQS queue permissions

Return to the SQS created in the prior step and select the Permissions tab.  Add/Modify the permissions to allow SQS:SendMessage from the SNS topic you just created.  Modify the below policy to use your SNS ARN and the SQS ARN resource.

4. Configure the notification for the Auto Scaling Group

Select your Auto Scaling Group and choose the 'Notifications' tab and then 'Create notification'.
For the notification choose the option 'terminate' and select the SNS topic created earlier.

5. Configure the IAM role

The EC2 instance that will be running our Powershell cleanup script  requires permissions to access the SQS queue.  To allow this, configure a security policy for the IAM role that is attached to the instance.  Modify the policy below for the Resource ARN to match your SQS ARN.

6. Create the Powershell script to retrieve the SQS messages

Powershell is used to obtain the SQS messages of the terminated instances and then removes the terminated servers from Active Directory.  Save the script on the server that will run the scheduled task.

Here is the script for the complete process. Modify this script to use your SQS queue name.

7. Create a scheduled task to run the Powershell script

To schedule the script,  configure a scheduled task on the Windows EC2 instance to run "Powershell" with an optional argument. 
The program path for PowerShell is: 'C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe'
The optional arguments is the path to your script:  'C:\Scripts\ActiveDirectory-CleanUp.ps1'

NOTE: You must run the scheduled task using a Windows User account that has the appropriate user permissions to remove objects from Active Directory.


Anonymous said...

Thank you for posting this, this was very helpful! I was trying to find out the best way to rip out the Receipt Handle but found parsing to get it difficult. I couldnt find easy steps on ripping it out until I found this page. Out of curiosity, did you find that in documentation somewhere? I felt like I scoured every documentation they had and couldnt find anything about ripping it out the way you did.

Ryan Lawyer said...

I agree, it was not easily found within the PowerShell documentation. But if you look at the Amazon.SQS.Model.Message you will find all the available properties.
SQS Model