Debugging Tips: cfn-init and configsets
Here are some debugging tips that will help when working with cfn-init.
How It Works
The cfn-init script is usually called by the UserData script to kick off a bootstrap process. The cfn-init call is what actually applies the configset.
UserData cfn-init
You can make sure configsets are applied when instances are launched by calling the cfn-init script in UserData. Here’s an example UserData script.
#!/bin/bash
yum install -y aws-cfn-bootstrap # install cfn-init
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource Instance --region ${AWS::Region}
The ${AWS::StackName} and ${AWS::Region} will be substituted for their actual values at CloudFormation runtime. The --resource Instance specifies which resource logical id in the CloudFormation template that has the metadata with configset instructions to apply.
Note: On AmazonLinux2 cfn-init is already installed.
Run Manually
The UserData script runs at launch time automatically. But you can also invoke the cfn-init script manually. Just log into the instance and call it. This can be especially useful for debugging. Here’s an example of running the script.
/opt/aws/bin/cfn-init -v --stack my-stack --resource Instance --region us-west-2
The command above downloads the Configset from CloudFormation template Instance resource and applies it.
Where is the metadata.json downloaded?
The cfn-init downloads the configset metadata to:
/var/lib/cfn-init/data/metadata.json
Note, this file gets overwritten each time cfn-init runs.
Run With a File
One quick way to debug configsets is to copy the generated /var/lib/cfn-init/data/metadata.json, modify it, and provide it directly to the cfn-init command. Example:
cp /var/lib/cfn-init/data/metadata.json /root/configset.json
/opt/aws/bin/cfn-init -v /root/configset.json
The cfn-init script itself only supports JSON when providing a file.
Here’s a YAML to JSON One Liner to help you convert it.
Running repeatedly with lono
Though knowing how to run the configset on the local machine is very helpful in understanding how configsets work, have found that is is quickest to just use lono to repeatedly apply the configsets in one terminal and while watching the logs in another terminal. In one terminal:
lono up BLUEPRINT -y
In another terminal, ssh into the machine and tail the logs with:
tail -f /var/log/cfn-init.log
Where are the Logs?
Knowing where the cfn-init related logs are can also save a ton of time.
| Path | Description |
|---|---|
| /var/log/cfn-init-cmd.log | cfn-init and command output with timestamps. |
| /var/log/cfn-init.log | cfn-init and command output. Useful for when cfn-init is manually ran. |
| /var/log/cloud-init.log | cloud-init logs from UserData launch script. |
| /var/log/cloud-init-output.log | Output from the cloud-init UserData commands themselves. Useful for when the instance is launched for the first time. |
cfn-hup reloader
The UserData script runs only at launch time. This means if you make changes to the CloudFormation Metadata.AWS::CloudFormation::Init configset, they do not magically get applied. The cfn-hup daemon must be set up. The cfn-hup daemon listens for changes in the metadata and applies them automatically.
It is recommended to set up cfn-hup reloader, so you do not have to log into the instance and kick off the cfn-init script manually.
Here’s the cfn-hup configset that sets up the cfn-hup reloader. The BoltOps Pro cfn-hup configset is available for free.
cfn-hup retries
Notice that cfn-hup will repeatedly retry running the configsets until it has successfully completed. So if you have an instruction in the the configset that is failing, you’ll see in the /var/log/cfn-init.log repeatedly running every minute or so. Unsure if there’s a way to control this behavior.