AWS EC2 Compute: Hands-on Runbook
AWS EC2 Compute: Hands-on Runbook
The EC2 lab as commands: a launch template with user data, one test instance reached without SSH keys, an Application Load Balancer with a target group, an Auto Scaling group across two AZs with a target tracking policy, the three proving tests (traffic, CPU stress, forced termination), then Spot and commitments, placement groups, monitoring and clean-up. Tick as you go. Concepts and the eight diagrams live in bubim-awscompute.
Prepare
0 / 0The default VPC is enough for the lab; use two of its public subnets in different AZs.
aws --version session-manager-plugin --version
export AWS_REGION=eu-central-1 export ACCOUNT=$(aws sts get-caller-identity --query Account --output text) export VPC=$(aws ec2 describe-vpcs --filters Name=is-default,Values=true --query 'Vpcs[0].VpcId' --output text) read SUBNET1 SUBNET2 <<< "$(aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC Name=default-for-az,Values=true \ --query 'Subnets[0:2].SubnetId' --output text)" export SUBNET1 SUBNET2; echo $VPC $SUBNET1 $SUBNET2
export SNS=$(aws sns create-topic --name compute-lab --query TopicArn --output text) aws sns subscribe --topic-arn $SNS --protocol email --notification-endpoint [email protected]
Security groups and the instance role
0 / 0Instances accept HTTP only from the load balancer; nobody needs port 22.
export ALB_SG=$(aws ec2 create-security-group --group-name lab-alb --description "ALB public" \ --vpc-id $VPC --query GroupId --output text) aws ec2 authorize-security-group-ingress --group-id $ALB_SG --protocol tcp --port 80 --cidr 0.0.0.0/0 aws ec2 authorize-security-group-ingress --group-id $ALB_SG --protocol tcp --port 443 --cidr 0.0.0.0/0
export WEB_SG=$(aws ec2 create-security-group --group-name lab-web --description "Web instances" \ --vpc-id $VPC --query GroupId --output text) aws ec2 authorize-security-group-ingress --group-id $WEB_SG --protocol tcp --port 80 --source-group $ALB_SG
aws iam create-role --role-name lab-web-ec2 --assume-role-policy-document \
'{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy --role-name lab-web-ec2 \
--policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam create-instance-profile --instance-profile-name lab-web-ec2
aws iam add-role-to-instance-profile --instance-profile-name lab-web-ec2 --role-name lab-web-ec2Launch template
0 / 0The model for every instance: AMI, type, security group, role, IMDSv2 and the user data that installs the web server.
export AMI=$(aws ssm get-parameter --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \ --query Parameter.Value --output text)
cat > user-data.sh <<'EOF' #!/bin/bash dnf install -y httpd TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 300") MD=http://169.254.169.254/latest/meta-data ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" $MD/instance-id) AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" $MD/placement/availability-zone) echo "<h1>$ID in $AZ</h1>" > /var/www/html/index.html systemctl enable --now httpd EOF
jq -n --arg ami "$AMI" --arg sg "$WEB_SG" --arg ud "$(base64 -w0 user-data.sh)" '{
ImageId: $ami, InstanceType: "t3.micro",
IamInstanceProfile: {Name: "lab-web-ec2"}, SecurityGroupIds: [$sg],
MetadataOptions: {HttpTokens: "required", HttpPutResponseHopLimit: 2, HttpEndpoint: "enabled"},
Monitoring: {Enabled: true}, UserData: $ud,
TagSpecifications: [{ResourceType: "instance", Tags: [{Key: "Name", Value: "lab-web"}]}]}' > lt.json
aws ec2 create-launch-template --launch-template-name lab-web --version-description v1 \
--launch-template-data file://lt.jsonaws ec2 create-launch-template-version --launch-template-name lab-web --source-version 1 \
--version-description "t3.small" --launch-template-data '{"InstanceType":"t3.small"}'
aws ec2 describe-launch-template-versions --launch-template-name lab-web \
--query 'LaunchTemplateVersions[].[VersionNumber,VersionDescription,LaunchTemplateData.InstanceType]' --output tableOne instance, inspected
0 / 0Launch a single instance from the template to check user data, metadata and status checks before scaling anything.
export ID=$(aws ec2 run-instances --launch-template LaunchTemplateName=lab-web,Version='$Latest' \ --subnet-id $SUBNET1 --query 'Instances[0].InstanceId' --output text) aws ec2 wait instance-status-ok --instance-ids $ID aws ec2 describe-instance-status --instance-ids $ID \ --query 'InstanceStatuses[0].[SystemStatus.Status,InstanceStatus.Status,AttachedEbsStatus.Status]' --output text
aws ssm start-session --target $ID
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-ipv4; echo
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/user-data | head -3
curl -s -o /dev/null -w '%{http_code}\n' http://169.254.169.254/latest/meta-data/ # 401: IMDSv1 is refused
ec2-metadata --instance-type --availability-zone
curl -s localhostaws ec2 create-key-pair --key-name lab --key-type ed25519 --query KeyMaterial --output text > lab.pem chmod 400 lab.pem ssh -i lab.pem ec2-user@PUBLIC_IP
aws ec2 terminate-instances --instance-ids $ID
Load balancer
0 / 0Target group first, then the ALB, then the listener.
export TG=$(aws elbv2 create-target-group --name lab-web --protocol HTTP --port 80 --vpc-id $VPC \ --target-type instance --health-check-path / --health-check-interval-seconds 15 \ --healthy-threshold-count 2 --unhealthy-threshold-count 2 \ --query 'TargetGroups[0].TargetGroupArn' --output text) aws elbv2 modify-target-group-attributes --target-group-arn $TG \ --attributes Key=deregistration_delay.timeout_seconds,Value=30
read ALB ALB_DNS <<< "$(aws elbv2 create-load-balancer --name lab-alb --type application --scheme internet-facing \ --subnets $SUBNET1 $SUBNET2 --security-groups $ALB_SG \ --query 'LoadBalancers[0].[LoadBalancerArn,DNSName]' --output text)" export ALB ALB_DNS; echo $ALB_DNS
export HTTP_L=$(aws elbv2 create-listener --load-balancer-arn $ALB --protocol HTTP --port 80 \ --default-actions Type=forward,TargetGroupArn=$TG --query 'Listeners[0].ListenerArn' --output text)
export CERT=$(aws acm request-certificate --domain-name www.example.com --validation-method DNS \
--query CertificateArn --output text)
aws acm wait certificate-validated --certificate-arn $CERT
aws elbv2 create-listener --load-balancer-arn $ALB --protocol HTTPS --port 443 --certificates CertificateArn=$CERT \
--ssl-policy ELBSecurityPolicy-TLS13-1-2-2021-06 --default-actions Type=forward,TargetGroupArn=$TG
aws elbv2 modify-listener --listener-arn $HTTP_L \
--default-actions 'Type=redirect,RedirectConfig={Protocol=HTTPS,Port=443,StatusCode=HTTP_301}'aws elbv2 create-rule --listener-arn $HTTP_L --priority 10 \
--conditions 'Field=source-ip,SourceIpConfig={Values=[203.0.113.0/24]}' \
--actions Type=forward,TargetGroupArn=$TG2Auto Scaling group behind the load balancer
0 / 0Fig 7 of the reference. The group launches from the template, registers its instances in the target group and replaces any that fail the ELB health check.
aws autoscaling create-auto-scaling-group --auto-scaling-group-name lab-web \ --launch-template LaunchTemplateName=lab-web,Version='$Latest' \ --min-size 2 --max-size 6 --desired-capacity 2 \ --vpc-zone-identifier "$SUBNET1,$SUBNET2" --target-group-arns $TG \ --health-check-type ELB --health-check-grace-period 120 --default-instance-warmup 120
aws elbv2 wait target-in-service --target-group-arn $TG aws elbv2 describe-target-health --target-group-arn $TG \ --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State]' --output table
aws autoscaling put-notification-configuration --auto-scaling-group-name lab-web --topic-arn $SNS \ --notification-types autoscaling:EC2_INSTANCE_LAUNCH autoscaling:EC2_INSTANCE_TERMINATE \ autoscaling:EC2_INSTANCE_LAUNCH_ERROR autoscaling:EC2_INSTANCE_TERMINATE_ERROR
aws autoscaling enable-metrics-collection --auto-scaling-group-name lab-web --granularity 1Minute
Scaling policies
0 / 0Target tracking first; the step policy uses a classic alarm (CPU above 75% for 5 minutes) for comparison. Run one of the two, not both.
aws autoscaling put-scaling-policy --auto-scaling-group-name lab-web --policy-name cpu-50 \
--policy-type TargetTrackingScaling \
--target-tracking-configuration '{"PredefinedMetricSpecification":{"PredefinedMetricType":"ASGAverageCPUUtilization"},"TargetValue":50.0}'POLICY=$(aws autoscaling put-scaling-policy --auto-scaling-group-name lab-web --policy-name cpu-75-step \ --policy-type StepScaling --adjustment-type ChangeInCapacity \ --step-adjustments MetricIntervalLowerBound=0,ScalingAdjustment=2 --query PolicyARN --output text) aws cloudwatch put-metric-alarm --alarm-name lab-web-cpu-75 --namespace AWS/EC2 --metric-name CPUUtilization \ --dimensions Name=AutoScalingGroupName,Value=lab-web --statistic Average --period 300 --evaluation-periods 1 \ --threshold 75 --comparison-operator GreaterThanThreshold --alarm-actions $POLICY $SNS
aws autoscaling put-scheduled-update-group-action --auto-scaling-group-name lab-web \ --scheduled-action-name office-open --recurrence "45 8 * * 1-5" --time-zone Europe/Sofia --min-size 4 aws autoscaling put-scheduled-update-group-action --auto-scaling-group-name lab-web \ --scheduled-action-name office-closed --recurrence "0 19 * * 1-5" --time-zone Europe/Sofia --min-size 2
LABEL="$(echo $ALB | cut -d/ -f2-)/$(echo $TG | cut -d: -f6)"
aws autoscaling put-scaling-policy --auto-scaling-group-name lab-web --policy-name req-1000 \
--policy-type TargetTrackingScaling --target-tracking-configuration \
"{\"PredefinedMetricSpecification\":{\"PredefinedMetricType\":\"ALBRequestCountPerTarget\",\"ResourceLabel\":\"$LABEL\"},\"TargetValue\":1000}"Prove it: three tests
0 / 0Traffic is spread, load makes the group grow, a killed instance comes back.
for i in $(seq 1 10); do curl -s http://$ALB_DNS/; echo; done | sort | uniq -c
aws ssm send-command --document-name AWS-RunShellScript \ --targets Key=tag:aws:autoscaling:groupName,Values=lab-web \ --parameters 'commands=["dnf install -y stress-ng","stress-ng --cpu 0 --timeout 600s"]' \ --query Command.CommandId --output text
aws autoscaling describe-scaling-activities --auto-scaling-group-name lab-web --max-items 6 \ --query 'Activities[].[StartTime,StatusCode,Description]' --output table aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names lab-web \ --query 'AutoScalingGroups[0].[DesiredCapacity,length(Instances)]'
ONE=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names lab-web \ --query 'AutoScalingGroups[0].Instances[0].InstanceId' --output text) aws ec2 terminate-instances --instance-ids $ONE sleep 90 aws autoscaling describe-scaling-activities --auto-scaling-group-name lab-web --max-items 2 \ --query 'Activities[].Description' --output text
ONE_MORE=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names lab-web \ --query 'AutoScalingGroups[0].Instances[1].InstanceId' --output text) aws ssm send-command --document-name AWS-RunShellScript --instance-ids $ONE_MORE \ --parameters 'commands=["systemctl stop httpd"]'
Cost: Spot, right-sizing, commitments
0 / 0Keep two On-Demand instances as the base, run the rest on Spot across several types, then size the commitment to the base.
aws autoscaling update-auto-scaling-group --auto-scaling-group-name lab-web --capacity-rebalance \
--mixed-instances-policy '{
"LaunchTemplate": {"LaunchTemplateSpecification": {"LaunchTemplateName": "lab-web", "Version": "$Latest"},
"Overrides": [{"InstanceType": "t3.micro"}, {"InstanceType": "t3a.micro"}, {"InstanceType": "t3.small"}]},
"InstancesDistribution": {"OnDemandBaseCapacity": 2, "OnDemandPercentageAboveBaseCapacity": 0,
"SpotAllocationStrategy": "price-capacity-optimized"}}'aws compute-optimizer update-enrollment-status --status Active aws compute-optimizer get-auto-scaling-group-recommendations \ --query 'autoScalingGroupRecommendations[].[autoScalingGroupName,finding,recommendationOptions[0].configuration.instanceType]' --output table
aws ce get-savings-plans-purchase-recommendation --savings-plans-type COMPUTE_SP \ --term-in-years ONE_YEAR --payment-option NO_UPFRONT --lookback-period-in-days THIRTY_DAYS \ --query 'SavingsPlansPurchaseRecommendation.SavingsPlansPurchaseRecommendationSummary' aws ce get-savings-plans-coverage --time-period Start=2026-09-01,End=2026-10-01 \ --query 'SavingsPlansCoverages[].Coverage.CoveragePercentage'
aws ce get-reservation-utilization --time-period Start=2026-09-01,End=2026-10-01 \ --query 'Total.UtilizationPercentage' --output text
Placement groups
0 / 0Fig 8 of the reference. Burstable T types cannot go into a cluster group, so this phase uses c7i.
aws ec2 create-placement-group --group-name lab-cluster --strategy cluster aws ec2 create-placement-group --group-name lab-partition --strategy partition --partition-count 3 aws ec2 create-placement-group --group-name lab-spread --strategy spread aws ec2 describe-placement-groups --query 'PlacementGroups[].[GroupName,Strategy,PartitionCount]' --output table
aws ec2 run-instances --launch-template LaunchTemplateName=lab-web,Version='$Latest' \ --instance-type c7i.large --count 2 --subnet-id $SUBNET1 --placement GroupName=lab-cluster \ --query 'Instances[].[InstanceId,Placement.GroupName]' --output table
aws ec2 run-instances --launch-template LaunchTemplateName=lab-web,Version='$Latest' \ --instance-type c7i.large --subnet-id $SUBNET2 --placement GroupName=lab-partition,PartitionNumber=2
Monitoring
0 / 0Status checks with actions, plus the metrics EC2 cannot see from outside.
aws cloudwatch put-metric-alarm --alarm-name recover-$ID --namespace AWS/EC2 \ --metric-name StatusCheckFailed_System --dimensions Name=InstanceId,Value=$ID \ --statistic Maximum --period 60 --evaluation-periods 2 --threshold 1 \ --comparison-operator GreaterThanOrEqualToThreshold \ --alarm-actions arn:aws:automate:$AWS_REGION:ec2:recover $SNS
aws cloudwatch put-metric-alarm --alarm-name lab-unhealthy --namespace AWS/ApplicationELB \ --metric-name UnHealthyHostCount \ --dimensions Name=TargetGroup,Value=$(echo $TG | cut -d: -f6) Name=LoadBalancer,Value=$(echo $ALB | cut -d/ -f2-) \ --statistic Maximum --period 60 --evaluation-periods 3 --threshold 0 \ --comparison-operator GreaterThanThreshold --alarm-actions $SNS
aws iam attach-role-policy --role-name lab-web-ec2 \ --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy # in user data: dnf install -y amazon-cloudwatch-agent, then amazon-cloudwatch-agent-ctl -a fetch-config
Clean up
0 / 0Reverse order. The group goes first, or it relaunches what you delete.
aws autoscaling delete-auto-scaling-group --auto-scaling-group-name lab-web --force-delete
aws elbv2 delete-load-balancer --load-balancer-arn $ALB aws elbv2 wait load-balancers-deleted --load-balancer-arns $ALB aws elbv2 delete-target-group --target-group-arn $TG
aws ec2 delete-launch-template --launch-template-name lab-web for G in lab-cluster lab-partition lab-spread; do aws ec2 delete-placement-group --group-name $G; done aws ec2 delete-security-group --group-id $WEB_SG aws ec2 delete-security-group --group-id $ALB_SG aws iam remove-role-from-instance-profile --instance-profile-name lab-web-ec2 --role-name lab-web-ec2 aws iam delete-instance-profile --instance-profile-name lab-web-ec2 for P in AmazonSSMManagedInstanceCore CloudWatchAgentServerPolicy; do aws iam detach-role-policy --role-name lab-web-ec2 --policy-arn arn:aws:iam::aws:policy/$P; done aws iam delete-role --role-name lab-web-ec2 aws ec2 delete-key-pair --key-name lab
Playbooks
For the day something goes wrong. Ticks here do not count toward the runbook progress.
PB1 Instance impaired
when a status check fails
aws ec2 describe-instance-status --instance-ids $ID --include-all-instances \ --query 'InstanceStatuses[0].[SystemStatus.Details,InstanceStatus.Details,Events]'
aws ec2 stop-instances --instance-ids $ID && aws ec2 wait instance-stopped --instance-ids $ID aws ec2 start-instances --instance-ids $ID
aws ec2 get-console-output --instance-id $ID --latest --output text | tail -n 40
PB2 The group keeps replacing instances
when launch, unhealthy, terminate repeats every few minutes
aws elbv2 describe-target-health --target-group-arn $TG \ --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason,TargetHealth.Description]' --output table
aws autoscaling suspend-processes --auto-scaling-group-name lab-web --scaling-processes ReplaceUnhealthy # resume afterwards aws autoscaling resume-processes --auto-scaling-group-name lab-web --scaling-processes ReplaceUnhealthy
PB3 Lost the private key
when the only .pem is gone and the instance must be reached
PB4 Spot capacity is interrupted
when rebalance recommendations or 2-minute notices arrive often
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 60") curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/spot/instance-action
Troubleshooting
| Symptom | Cause | Fix | Step |
|---|---|---|---|
| Metadata call returns 401 | IMDSv1 call against an instance that requires IMDSv2 | get a token with PUT first | 3.3 |
| Containers on the instance cannot reach metadata | hop limit 1 | HttpPutResponseHopLimit 2 in the template | 2.3 |
| start-session fails, instance not managed | role missing AmazonSSMManagedInstanceCore, or no route to SSM endpoints | fix the instance profile, or add SSM VPC endpoints in private subnets | 1.3 |
| Page not served, health check failing | user data failed | read /var/log/cloud-init-output.log | 2.2 |
| Targets unhealthy, instance fine | instance security group does not allow the ALB group | ingress 80 from ALB_SG | 1.2 |
| ALB returns 503 | no registered or healthy targets | check describe-target-health, the group’s target group link | 5.2 |
| ALB returns 504 | target too slow or blocked | app timeouts, security group, idle timeout | 4.1 |
| Uneven load between AZs | uneven target counts with cross-zone off | balance the group or turn cross-zone on | 5.1 |
| Group does not scale out under load | policy metric not breached, warmup, max size reached | describe-scaling-activities, raise max | 6.1 |
| Group flaps in and out | thresholds too close, no warmup | target tracking, default instance warmup | 6.1 |
| Launch fails in a cluster placement group | burstable type, or insufficient capacity for separate launches | c or m types, one request with the full count | 9.2 |
| Spot instances never launch | single type in one AZ | more types and AZs, price-capacity-optimized | 8.1 |

0 comments