Azure Usage monitoring with Azure Automation

When you purchase an Azure subscription it comes with usage caps for various resources. As an example the usage cap for number of cores is 20. You can call use Azure Support and open a free billing support case to increase this core limit.

In the past few years I have had many clients ask for basic alerting capability when they are about to exceed their resource limits. They have Azure subscriptions that are being used by various teams and they want to know if they are reaching their Azure usage limit. They can install Azure PowerShell cmdlet and easily find the answer to this question. However they are looking for automated alerting service. I heard this request last week so I thought I will use Azure Automation to implement this solution.

There are two use case scenarios for this script:

1. It can be used by Azure  Subscription Owner to understand if they are about to exceed the resource(compute cores) quota for an Azure subscription.

2. There have been times when you keep Azure services running longer than you need them. This script will run on a schedule and inform you about the compute cores you are currently using. This could have helped me last year when I left  HD Insight cluster with 32 cores running for a month.

Azure Automation recently became generally available and it can be used to automate error prone, time consume, cloud management tasks. It leverages PowerShell based workflow scripts to automate tasks. You can learn more about it here:

http://azure.microsoft.com/en-us/services/automation/

I also highly recommend this course in virtual academy.

http://www.microsoftvirtualacademy.com/training-courses/automating-the-cloud-with-azure-automation

Here are the high level steps to implement this script.

1. Create Azure automation account
2. Create Credential Asset for Azure Administration
3. Create Credential Asset for Office 365 user that will be used to send emails
4. Create the runbook
5. Test the runbook
6. Publish the runbook
7. Link it to a schedule
8. View Job history

Create Azure Automation Account using Azure Portal

You can do so by selecting Automation and “+ Create” button.

Right now you can create Azure Automation account in “East US”, “Southeast Asia” and “West Europe” only.

When you create an account in a region it stores its assets in that region. However this account can automate tasks in any other region.

Creating Credentials

Azure Active Directory for Azure Credentials

Create a new user in Azure Active Directory

Use Azure Portal and select “Active Directory”

Select your Active Directory instance and navigate to “User” section and use “Add User” button in the bottom toolbar.

Select “new user in your organization”

Enter the user name.

Enter user information in the User Profile section.

Press “Create” button and it will show you the temporary password.

Sign into the Azure Active Directory as this newly created user and change the temporary password.

Sign in to Windows Azure Active Directory

CoAdmin Access

Make this new user a Co-administrator for the Azure subscription you want to monitor.

You do this by select “Settings” –>Administrators and press “Add” button in the bottom toolbar

On the “Add A CO-Administrator” screen specify the Azure AD user you just created and select the appropriate subscription from the list below.

Create an asset of type Credentials in your automation account

Automation accounts has assets that can be used by runbooks. These are convenient place to securely store user names, passwords and connection strings.

We need to create Credentials to get access to the Azure subscription. Select your newly created Azure automation account and select “Assets”. Press in the “Add Setting” button

Select “add credential”

There are two options for credentials:

1. Windows PowerShell Credential

2. Certificate

You need to select “Windows Azure PowerShell”

Enter the name and password of the Azure AD user that is also a Co-Administrator to the Azure subscription you are monitoring.

Create Office 365 Credentials to send out emails

I have Office 365 small business account. I have a separate Azure subscription. Until now I never had a need to use Active Directory associated with my Office 365 account. Here are the steps to setup credentials for Office 365 as assets in Azure Automation.

Use Azure Management Portal   New->App Services ->Active Directory-Directory-Custom Create

On the Add Directory popup you need to select “Use Existing directory”

You will be asked to sign in as administrator for your Office 365 account.

Once Office 365 Directory has been added to the Portal you can see the list of existing users or add a new user that will be used to send out emails about Azure  resource usage.

You need to create an asset of type Credentials in your Azure Automation account next.

The steps the create the Credentials are identical to steps to create Azure administration account. I name the credential object O365Cred.

Create Runbook

Select your Azure Automation Account and select “New->Automation->RunBook->QuickCreate to create you new RunBook.

You can use the Author Tab to create the run book. Authoring in the portal worked OK for me but I had trouble navigating through the script as it grew longer. I tried IE and Chrome and got the same results. In future I may first create the runbook in PowerShell ISC first and unit test it in the Azure portal.

Here is the script for the runbook. It looks like a normal PowerShell script with a few differences

You declare Parameters for the runbook in lines 3 through 9.

You retrieve the credentials for the Azure administration account in line 12.

You determine the current resources consumed in line 16

I want you to look at line 21 carefully as this is where I get list of services that are not in “StoppedDeallocated” status. These are the services that are incurring compute charges. Automation runbooks do not support positional parameters. I had to add –FilterScript after the Where-Object to make this expression work. Without the –FilerScript I was getting the following error:

azure automation parameter set cannot be resolved using the specified named parameters.

You retrieve the office 365 credentials in line 30

You send email with Send-MailMessage cmdlet in line 36

Testing

You can test the runbook in the portal by pressing the “Test” button in the bottom toolbar. When you run your tests you will see a window to enter the parameters. If the script runs successfully you will see the output.

Publishing

Once your testing is complete  you can press the “Publish” button to publish this run.

Here is an email received from the runbook.

You are using 2 of 20 cores.The following services are still incurring compute charges:@{ServiceName=sansoroprovtest; PSComputerName=localhost; PSShowComputerName=True; PSSourceJobInstanceId=5d402195-f0a1-4a72-8b72-c27f0633ab58}

You can schedule this run book to run on daily or hourly basis.

You can create a new schedule by selecting “Schedule” and “Link to New Schedule”

Adding a Schedule

You can view the Job History by looking at the Job section of the runbook.

You can drill down and view the details of the last run.

Summary section of the history shows job summary, input parameters and script output.

There is also a history section that shows information about previous executions of the runbook.

With this simple example I hoped to demonstrate how you can automate cloud management tasks using Azure automation runbooks. Here are a few things about Azure automation worth mentioning:

• Runbooks can call other runbooks inline or invoke them asynchronously.
• You can leverage integration modules as well. As an example I wanted to use Azure Resource Manager with Azure Automation but it is currently not supported. All I had to do was zip the Azure Resource Manager directory upload it and start using it. It is still not officially supported.
• I was surprised to learn that we can call Runbooks from on premise PowerShell cmdlets.
• You can run parallel activities in these runbooks
• Since they are based on workflow you can save the state of a running runbook and rollback if needed.
• Runbooks don’t support positional parameters
• Certain cmdlets like Write-Host are not supported. I replaced Write-Host with Write-Output

Azure Automation is an easy, secure, flexible, extensible and scalable way to automate cloud management tasks. Most of your existing PowerShell scripts can be easily converted into runbooks. There is already a gallery of runbooks available in the Azure portal. You can import these runbooks and use them to automate tasks. You can find many sample runbook’s here:

https://social.technet.microsoft.com/Search/en-US/scriptcenter?query=azure%20automation&beta=0&ac=5#refinementChanges=&pageNumber=2&showMore=false

I highly recommend reading “Authoring Automation Runbooks” guide

http://technet.microsoft.com/en-us/library/dn469262.aspx

As a future enhancement you can pass in subscriptions via a JSON/XML file stored in blob storage. You can send push notifications in addition to emails.

We have barely scratched the surface of Azure Automation. It can and will play pivotal role in implementing continuous deployment and other tasks related to DevOps. I have started using Azure automation in a few of my projects. I will share my learning in future blog posts. How are you using Azure automation?

The post Azure Usage monitoring with Azure Automation appeared first on Raj's Cloud Musings.