Skip to content

AWS Updated 2026-07-04

Debugging EC2 user-data scripts that never ran: reading the cloud-init logs

When an EC2 user-data script never ran, the answer is in the cloud-init logs. Here is where to look, the once-per-instance rule that catches everyone, and the shebang and syntax mistakes that fail silently.

When an EC2 user-data script “never ran,” one of three things is true: it ran and failed (the most common), it ran on a previous boot and will not run again because user data executes once per instance by default, or cloud-init never recognized it as a script because the shebang or MIME formatting is wrong. All three leave evidence in the same two files: /var/log/cloud-init.log and /var/log/cloud-init-output.log. Read those before changing anything — guessing at user data by relaunching instances is the slowest debugging loop on AWS.

What’s happening

On Amazon Linux, Ubuntu, Debian, and most modern distros, cloud-init is the agent that fetches user data from the instance metadata service during boot and decides what to do with it. A shell script (first line #!/bin/bash or similar) is written to /var/lib/cloud/instance/scripts/ and executed late in boot, as root, by the scripts-user module. A #cloud-config YAML document is interpreted instead. Anything that starts with neither is quietly ignored — which reads as “my script never ran.”

The rule that catches everyone: the scripts-user module runs once per instance, keyed on instance ID, tracked under /var/lib/cloud/instances/. Reboots do not re-run it. Stopping an instance, editing the user data, and starting it again does not re-run it either, because the instance ID has not changed. That surprise accounts for a large share of “user data is broken” reports.

The execution environment is also less forgiving than your terminal: the script runs as root with a minimal, non-interactive environment — no meaningful $HOME, a short PATH, no TTY. Commands that prompt (apt-get install without -y), rely on a user profile, or assume the network stack of a fully booted machine will hang or die. And on Windows, none of this applies — EC2Launch v2 handles user data with its own log at C:\ProgramData\Amazon\EC2Launch\log\agent.log.

What to check first

  • Is there user data at all, and does it start with a shebang or #cloud-config? EC2 > Instances > Actions > Instance settings > Edit user data shows what the instance actually received.
  • Is this the first boot of this instance? If the instance ran before the user data was added or edited, the once-per-instance rule already explains everything.
  • Did it run but fail? cloud-init-output.log holds the script’s stdout/stderr — a package error at line 3 looks exactly like “it never ran” if you never look.
  • Is the AMI cloud-init based? Minimal or custom AMIs sometimes ship without cloud-init, and Windows uses EC2Launch v2.
  • Does the script need the network or IMDS? Calls to the metadata service must use IMDSv2 tokens — see the IMDSv2 401 problem — and external downloads need working egress at boot time.

How to verify

On the instance, in order:

# Did cloud-init finish, and with what result?
cloud-init status --long

# The script's own output and errors live here
sudo less /var/log/cloud-init-output.log

# cloud-init's internal decisions (modules run, user data parsing)
sudo grep -iE 'scripts-user|userdata|Traceback|WARN|FAIL' /var/log/cloud-init.log

# Was the script even extracted? If this dir is empty, parsing failed
sudo ls -l /var/lib/cloud/instance/scripts/

Confirm what user data the instance actually holds (from the instance itself, via IMDSv2):

TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/user-data

Or from outside: aws ec2 describe-instance-attribute --instance-id i-0123456789abcdef0 --attribute userData (base64-encoded). Timing analysis, when boot ordering is the suspect, comes from cloud-init analyze show. The cloud-init behaviour reference is part of the EC2 documentation at https://docs.aws.amazon.com/ec2/.

The fix

  1. Script ran and failed: fix the actual error. cloud-init-output.log shows the failing line. The classics: apt-get install without -y, a package repo not reachable yet, a command not in the minimal PATH (use absolute paths), or a Windows-edited file with CRLF line endings that breaks the shebang. Add set -euxo pipefail near the top so future failures are loud and located.
  2. Once-per-instance rule bit you: force a re-run or relaunch. For a one-off, sudo cloud-init clean --logs && sudo reboot resets cloud-init’s state so user data runs again on the next boot. For repeatable behaviour, either bake the logic into the AMI, or use a MIME multi-part user data with a #cloud-config part setting cloud_final_modules: [[scripts-user, always]] so the script runs on every boot — and make the script idempotent before you do.
  3. No shebang or wrong format: make the first line count. A bare list of commands is not a script to cloud-init. Start shell scripts with #!/bin/bash as byte one — no leading blank line, no BOM. YAML must start with #cloud-config exactly.
  4. Edited user data on a stopped instance and expected it to run: it will not, by default. Same fix as item 2 — cloud-init clean before starting, or accept that user data is a first-boot mechanism and put day-2 changes in SSM Run Command or your configuration management instead.
  5. Script needs to be debugged interactively: run it by hand. The extracted script sits in /var/lib/cloud/instance/scripts/part-001. Execute it as root in a clean environment (sudo env -i /bin/bash /var/lib/cloud/instance/scripts/part-001) and you will reproduce boot behaviour far faster than relaunching instances.

When this is the wrong move

If you are iterating on a 300-line user-data script, stop — user data is the wrong place for real provisioning logic. Bake a golden AMI (Packer or EC2 Image Builder) and keep user data to a few lines of instance-specific configuration; your Auto Scaling groups will boot faster and fail less, which matters when scale-out speed is the point. Also skip the user-data rabbit hole when the instance never became reachable at all — a boot that dies before cloud-init is a different problem with different logs.

Common mistakes

  • Rebooting to “re-trigger” user data and concluding it is broken when nothing happens — reboot never re-runs it by default.
  • No -y on package installs, so the script waits forever on a prompt no one can answer.
  • Assuming the script runs as the default user — it runs as root, so ~ and user-owned paths point somewhere else.
  • Testing the script in an interactive shell where PATH and environment differ from boot, then shipping it untested against the real environment.

Prevention

  • Start every user-data script with #!/bin/bash and set -euxo pipefail, and end it with an explicit marker line you can grep for.
  • Keep user data short and idempotent; put heavy provisioning in the AMI.
  • Log to the console too (exec > >(tee /var/log/user-data.log | logger -t user-data) 2>&1) so the output shows up in Get system log even when SSH is broken.
  • Version user data in your launch templates so “what changed” is a diff, not a memory.
  • Test new user data on a throwaway instance before rolling it into the template your fleet boots from.

Ten minutes reading cloud-init logs beats an afternoon of relaunching instances on a hunch. If nobody on your team truly owns the AWS environment end to end, that is the root of most of these problems — Stack Harbor runs AWS accounts daily under AWS Operations, and you can book an AWS review.