Overview
I recently had to install Splunk Universal Forwarder in Azure worker roles. Azure web/worker roles are stateless so the only way to install any software is to do so via Azure startup tasks. Azure startup tasks have been around for many years. MSDN documentation about startup tasks can be reviewed here:
Run startup tasks in Azure
https://msdn.microsoft.com/en-us/library/azure/hh180155.aspx?f=255&MSPPError=-2147217396
Best practices for startup tasks
https://msdn.microsoft.com/en-us/library/azure/jj129545.aspx
One of the best resources about Azure startup tasks is these two blog posts by my friend Chris Clayton.
http://blogs.msdn.com/b/cclayton/archive/2012/05/17/windows-azure-start-up-tasks-part-1.aspx
http://blogs.msdn.com/b/cclayton/archive/2012/05/17/windows-azure-start-up-tasks-part-2.aspx
Even though I will show you how to install Splunk Universal forwarder this approach can be used to install any third party software on a web/worker role.
Details
Requirements
- We only need to install Splunk forwarder once.
- We need to install the forwarder using command line
- We need to leave detailed logs to help us debug any issues that may arise
- We need to use a reliable location to download the setup files.
A few more decisions need to be made upfront.
How to make the installer available in the web/worker role?
Your choices are:
- Include the installer with the source code
- Download the installer from an external locations.
Having large deployment package can affect the speed of deployment so I tend to prefer downloading the installer from either Azure blob storage or external location like the vendor website that created the installer file.
Which scripting language to use for the startup task?
You choices are:
1. Combination of a command batch file and a PowerShell script.
2. Do all the installation via a command batch file.
In my case I chose to do use just a command batch file to install Splunk. Batch files have been around for 25 years and I wanted to keep things as simple as possible.
Step 1
Splunk Universal Forward MSI file can be downloaded from splunk.com. However we cannot be certain that download location of the installer will not change in future. So I downloaded the MSI file for Splunk installer and uploaded into blob container in a storage account. Create a Azure storage account or use an existing storage account. I used Azure management portal to create a new storage account. This account should be created in the same region where you will be deploying your cloud service. This is not a requirement but having your storage account with Splunk installer in the same region as your cloud service will increase the download speed.
Create a storage container where you will upload the Splunk installer. I created a public container called Splunk in Azure management portal.
Download Splunk Installer and upload it into a storage container. I used Azure management studio to upload the msi file to the newly created container. You can use any tool of your choice to upload the MSI.
Step 2
Create Startup.cmd file and add it to the worker/web role implementation project. This is a standard cmd file however it needs to be saved in “Unicode (UTF-8 without signature) – Codepage 65001” or it will not execute as a startup task.
Select the newly add startup.cmd file in the solution explorer and set the property “copy to output directory” to “Copy if newer” as shown below.
Startup.cmd needs to download the Splunk installer. I chose to use Azcopy command line utility. My other option was to use PowerShell azure storage cmdlets to download the file. You can learn more about azcopy here:
https://azure.microsoft.com/en-us/documentation/articles/storage-use-azcopy/
I downloaded azcopy and its dependencies and included them in my worker role implementation project.
Set the property “copy to output directory” to “copy if newer” to make sure azcopy and its dependencies get copied to the role instance during deployment.
You need to define your startup task in the role ServiceDefinition.csdef file. Here are a few things worth mentioning:
- You define startup task by adding <Startup> section in the WorkerRole element.
- You can define as many tasks as you want.
- Task element mentions that command that will be executed during the execution of the startup tasks.
- elevationContext defines the level of access the startup task will have
- taskType can be simple, foreground or background.
- You can define variables which can access Role Environment during the startup task execution.
1: <?xml version="1.0" encoding="utf-8"?>
2: <ServiceDefinition name="AzureStatupTask" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2015-04.2.6">
3: <WorkerRole name="MyWorkerRole" vmsize="Small">
4: <Startup>
5: <Task commandLine="Startup.cmd" executionContext="elevated" taskType="simple">
6: <Environment>
7: <Variable name="LogFileName" value="Startup.log" />
8: <Variable name="LogFileDirectory">
9: <RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/LocalResources/LocalResource[@name='LogsPath']/@path" />
10: </Variable>
11: <Variable name="InstanceId">
12: <RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/@id" />
13: </Variable>
14: <Variable name="RoleName">
15: <RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/@roleName" />
16: </Variable>
17: </Environment>
18: </Task>
19: </Startup>
20: <ConfigurationSettings>
21: <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" />
22: </ConfigurationSettings>
23: <LocalResources>
24: <LocalStorage name="LogsPath" cleanOnRoleRecycle="false" sizeInMB="1024" />
25: </LocalResources>
26: <Imports>
27: <Import moduleName="RemoteAccess" />
28: <Import moduleName="RemoteForwarder" />
29: </Imports>
30: </WorkerRole>
31: </ServiceDefinition>
Here is the startup.cmd file I used to deploy Splunk Universal forwarder. Here is a summary of the steps:
- I check to see if StartupComplete.txt exists. If it exists I exit from the script.
- I used azcopy to download the Splunk installer from blob storage to local storage
- I execute the installer to install Splunk forwarder
- I update inputs.conf to setup monitoring for the application log files
- I start Splunk service
- If there are no errors I create StartupComplete.txt
- If the entire script is successful you need to exit it with return code of 0
- If there is a failure you return an errorlevel
1: SET LogPath=%LogFileDirectory%%LogFileName%
2:
3: ECHO Current Role: %RoleName% >> "%LogPath%" 2>&1
4: ECHO Current Role Instance: %InstanceId% >> "%LogPath%" 2>&1
5: ECHO Current Directory: %CD% >> "%LogPath%" 2>&1
6:
7: ECHO We will first verify if startup has been executed before by checking %RoleRoot%\StartupComplete.txt. >> "%LogPath%" 2>&1
8:
9: IF EXIST "%RoleRoot%\StartupComplete.txt" (
10: ECHO Startup has already run, skipping. >> "%LogPath%" 2>&1
11: EXIT /B 0
12: )
13:
14: AzCopy\AzCopy.exe /Source:https://rajpublic.blob.core.windows.net/splunk/ /Dest:%TEMP% /Pattern:splunkforwarder-6.2.3-264376-x64-release.msi /Y >> "%LogPath%" 2>&1
15:
16: IF ERRORLEVEL EQU 0 (
17:
18: Echo Installing Splunk Forwarder >> "%LogPath%" 2>&1
19:
20: msiexec.exe /i %TEMP%\splunkforwarder-6.2.3-264376-x64-release.msi AGREETOLICENSE=Yes RECEIVING_INDEXER="10.0.0.68:9997" LAUNCHSPLUNK=0 SERVICESTARTTYPE=auto WINEVENTLOG_APP_ENABLE=1 SET_ADMIN_USER=1 PERFMON=cpu,memory,network,diskspace /quiet >> "%LogPath%" 2>&1
21:
22: IF ERRORLEVEL EQU 0 (
23:
24:
25: Echo [monitor://D:\logs] >> "D:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf"
26: Echo disabled = false >> "D:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf"
27: Echo followTail = true >> "D:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf"
28: Echo index = main >> "D:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf"
29: Echo sourcetype = general >> "D:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf"
30:
31: "D:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" start >> "%LogPath%" 2>&1
32:
33: IF ERRORLEVEL EQU 0 (
34: ECHO Splunk installed. Startup completed. >> "%LogPath%" 2>&1
35: ECHO Startup completed. >> "%RoleRoot%\StartupComplete.txt" 2>&1
36: EXIT /B 0
37: ) ELSE
38: (
39: ECHO An error occurred while starting Splunk. The ERRORLEVEL = %ERRORLEVEL%. >> "%LogPath%" 2>&1
40: EXIT %ERRORLEVEL%
41: )
42: ) ELSE (
43: ECHO An error occurred. The ERRORLEVEL = %ERRORLEVEL%. >> "%LogPath%" 2>&1
44: EXIT %ERRORLEVEL%
45: )
46: ) ELSE (
47: ECHO An error occurred while downloading Splunk forwarder The ERRORLEVEL = %ERRORLEVEL%. >> "%LogPath%" 2>&1
48: EXIT %ERRORLEVEL%
49: )
50:
Verification and Troubleshooting
If startup task completed successfully you will see a file called e:\startupcomplete.txt as shown below.
This file is used to determine if Splunk installer was installed successfully.
Verify that Splunk forwarder was installed successfully.
Verify that post install configuration was complete successfully.
Troubleshooting
If your deployment fails or if Splunk in not installed successfully you will follow these steps to troubleshoot issues:
Verify that AzCopy and Startup.cmd were copied to the e:\approot directory.
Look in C:\Resources\temp\xxxxxxxxxxxxxxxxxxxxxxxxx.MyWorkerRole\RoleTemp
You should see the splunk installer in this directory.
Look for the full log file created by the startup.cmd in C:\Resources\Directory\a1be486474e348728ec129e4109d4e13.MyWorkerRole.LogsPath
Here is a sample startuplog.txt file that was created when there were no errors.
1: Current Role: MyWorkerRole
2: Current Role Instance: MyWorkerRole_IN_0
3: Current Directory: E:\approot
4: We will first verify if startup has been executed before by checking E:\StartupComplete.txt.
5: [2015/07/03 22:42:35] Transfer summary:
6: -----------------
7: Total files transferred: 1
8: Transfer successfully: 1
9: Transfer skipped: 0
10: Transfer failed: 0
11: Elapsed time: 00.00:00:03
12: Installing Splunk Forwarder
13:
14: Splunk> CSI: Logfiles.
15:
16: Checking prerequisites...
17: Checking mgmt port [8089]: Loading 'screen' into random state - done
18: Generating a 1024 bit RSA private key
19: ....++++++
20: ..................................................................++++++
21: writing new private key to 'privKeySecure.pem'
22: -----
23: Loading 'screen' into random state - done
24: Signature ok
25: subject=/CN=RD000D3A909BA6/O=SplunkUser
26: Getting CA Private Key
27: writing RSA key
28: open
29: Checking conf files for problems...
30: Done
31: All preliminary checks passed.
32:
33: Starting splunk server daemon (splunkd)...
34:
35: SplunkForwarder: Starting (pid 2436)
36: Done
37:
38: Splunk installed. Startup completed.
39:
Code Sample
You can get the source of this sample from my GitHub Repository here: https://github.com/rajinders/azure-startup-task
Summary
Azure startup tasks are the only way to install third party software on your Azure web/worker roles. Failures in startup tasks can lead to role startup issues so you need to log extensively to help you troubleshoot errors. Startup tasks do have access to your role environment.
The post Installing Splunk Forwarder in Azure Web/Worker Roles with a Startup Task appeared first on Raj's Cloud Musings.
