AWS Build Series: Hands-on Runbook
Runbook · AWS · Build and run a web app
AWS Build Series: Hands-on Runbook
The five builds as commands. Eleven phases move a WordPress site to Lightsail behind a CDN, make it highly available, stand up the Rails app on Elastic Beanstalk with RDS, make it scale, take the cost out, move image labelling into a SAM function and route /api to Lambda. Every step is one decision or one command block; tick as you go. Concepts and the fourteen diagrams live in bubim-awsbuild.
sam deploy –guided cpx · bub.im
0 / 0
P0
Prepare
0 / 0One Region for everything except the Lightsail domain, certificate and distribution calls, which go to us-east-1.
0.1 Pick the Region closest to the usersLatency to visitors matters more than the few percent of price difference between Regions.
0.2 Install and check the tools
aws --version # AWS CLI v2 pipx install awsebcli # EB CLI eb --version sam --version # SAM CLI, from the AWS installer for your OS jq --version
0.3 Export the variables used below
export AWS_REGION=eu-central-1 export ACCOUNT=$(aws sts get-caller-identity --query Account --output text) export DOMAIN=example.com
0.4 Put a budget alarm in place before anything runs
aws budgets create-budget --account-id $ACCOUNT \
--budget '{"BudgetName":"build-series","BudgetLimit":{"Amount":"50","Unit":"USD"},"TimeUnit":"MONTHLY","BudgetType":"COST"}' \
--notifications-with-subscribers '[{"Notification":{"NotificationType":"ACTUAL","ComparisonOperator":"GREATER_THAN","Threshold":80},"Subscribers":[{"SubscriptionType":"EMAIL","Address":"[email protected]"}]}]'P1
Lightsail WordPress instance
0 / 0Console path: Lightsail, Region, WordPress, plan, name. The CLI below does the same and shows the IDs first, because blueprint and bundle IDs change over time.
1.1 Look up the WordPress blueprint and a Linux bundlePrefer the blueprint whose provider is Lightsail; pick Micro (1 GB) or Small (2 GB).
aws lightsail get-blueprints --query "blueprints[?contains(name,'WordPress')].[blueprintId,name,version]" --output table aws lightsail get-bundles --query "bundles[?contains(supportedPlatforms,'LINUX_UNIX')].[bundleId,price,ramSizeInGb]" --output table export BP=REPLACE_WITH_BLUEPRINT_ID BUNDLE=REPLACE_WITH_BUNDLE_ID
1.2 Create the instanceIt boots into the default WordPress page within a few minutes.
aws lightsail create-instances --instance-names wp1 --availability-zone ${AWS_REGION}a \
--blueprint-id $BP --bundle-id $BUNDLE
aws lightsail get-instance-state --instance-name wp1 --query state.name --output text1.3 Attach a static IP and open HTTPSThe default public IP changes on stop and start; DNS must point at something stable.
aws lightsail allocate-static-ip --static-ip-name wp1-ip aws lightsail attach-static-ip --static-ip-name wp1-ip --instance-name wp1 aws lightsail open-instance-public-ports --instance-name wp1 --port-info fromPort=443,toPort=443,protocol=TCP aws lightsail get-static-ip --static-ip-name wp1-ip --query staticIp.ipAddress --output text
1.4 Read the admin password over browser SSHConnect from the instance page, then sign in at http://STATIC_IP/wp-admin as user.
cat ~/application_credentials # Lightsail blueprint cat ~/bitnami_application_password # Bitnami blueprint
1.5 Update WordPress core, themes and pluginsThe image is older than the latest release; update before importing anything so the import lands on current code.
1.6 Snapshot the clean instance and turn on daily snapshots
aws lightsail create-instance-snapshot --instance-name wp1 --instance-snapshot-name wp1-fresh
aws lightsail enable-add-on --resource-name wp1 \
--add-on-request 'addOnType=AutoSnapshot,autoSnapshotAddOnRequest={snapshotTimeOfDay=02:00}'P2
Migrate the site
0 / 0All-in-One WP Migration on both ends. The import replaces the users table.
2.1 Install All-in-One WP Migration on the old site and on LightsailPlugins, Add New, search, install, activate. Same plugin version on both sides avoids format surprises.
2.2 Export from the old site to a fileAll-in-One WP Migration, Export to, File. The .wpress file holds database, media, themes, plugins and users.
2.3 Raise the PHP upload limits on Lightsail and restart ApacheSet the limits above the size of the export file.
PHPINI=$(ls /etc/php/*/apache2/php.ini | head -n1) # Bitnami: /opt/bitnami/php/etc/php.ini
sudo cp "$PHPINI" "$PHPINI.bak"
sudo sed -i -e 's/^upload_max_filesize.*/upload_max_filesize = 512M/' \
-e 's/^post_max_size.*/post_max_size = 512M/' \
-e 's/^max_execution_time.*/max_execution_time = 300/' "$PHPINI"
grep -E '^(upload_max_filesize|post_max_size|max_execution_time)' "$PHPINI"
sudo systemctl restart apache2 # Bitnami: sudo /opt/bitnami/ctlscript.sh restart2.4 Import on LightsailAll-in-One WP Migration, Import from, File. Afterwards the old site’s users and passwords apply; the Lightsail admin password no longer works.
2.5 Check the site and resave permalinksSettings, Permalinks, Save rewrites the rules. Click through posts, pages, media, forms and the plugins that call external services.
2.6 Snapshot the migrated site
aws lightsail create-instance-snapshot --instance-name wp1 --instance-snapshot-name wp1-migrated
P3
CDN, custom domain and HTTPS
0 / 0Lightsail DNS zones, distributions and their certificates are managed through us-east-1, whatever Region the instance is in.
3.1 Create the DNS zone, or keep your providerWith a Lightsail zone, set the four name servers it lists at your registrar and wait for them to propagate.
aws lightsail create-domain --region us-east-1 --domain-name $DOMAIN aws lightsail get-domain --region us-east-1 --domain-name $DOMAIN --query "domain.domainEntries[?type=='NS'].target"
3.2 Decide where TLS endsSimple: HTTPS at the distribution, HTTP to the instance. End to end: run the instance HTTPS setup (Lightsail setup wizard, or bncert-tool on Bitnami) first, then set the origin protocol to HTTPS only.
3.3 Create the distribution with the Best for WordPress presetConsole: Networking, Create distribution, origin wp1. The CLI shows the result and its default domain.
aws lightsail get-distributions --region us-east-1 \ --query 'distributions[].[name,domainName,status]' --output table export CDN_NAME=wp-cdn CDN_DOMAIN=REPLACE_WITH_DISTRIBUTION_DOMAIN
3.4 Request the certificate and read the validation records
aws lightsail create-certificate --region us-east-1 --certificate-name wp-cert \ --domain-name $DOMAIN --subject-alternative-names www.$DOMAIN aws lightsail get-certificates --region us-east-1 --certificate-name wp-cert --include-certificate-details \ --query 'certificates[0].certificateDetail.domainValidationRecords[].resourceRecord' --output table
3.5 Add the validation CNAMEs to DNSThis is the proof of ownership. Repeat for each record from the previous step; validation usually completes within minutes.
aws lightsail create-domain-entry --region us-east-1 --domain-name $DOMAIN \ --domain-entry name=_abc123.example.com,type=CNAME,target=_def456.acm-validations.aws
3.6 Attach the certificate to the distribution
aws lightsail attach-certificate-to-distribution --region us-east-1 \ --distribution-name $CDN_NAME --certificate-name wp-cert
3.7 Point the apex and www at the distribution
aws lightsail create-domain-entry --region us-east-1 --domain-name $DOMAIN \ --domain-entry name=$DOMAIN,type=A,target=$CDN_DOMAIN,isAlias=true aws lightsail create-domain-entry --region us-east-1 --domain-name $DOMAIN \ --domain-entry name=www.$DOMAIN,type=CNAME,target=$CDN_DOMAIN
3.8 Set the site URL in wp-config.php and restart ApacheReplace the two default WP_SITEURL and WP_HOME lines with the snippet from the reference sheet (CDN section).
sudo cp /var/www/wp-config.php /var/www/wp-config.php.backup # Bitnami: /opt/bitnami/wordpress/wp-config.php sudo vim /var/www/wp-config.php sudo systemctl restart apache2
3.9 Prove that the CDN cachesRequest a static file twice; the second answer should say Hit from cloudfront.
curl -sI https://$DOMAIN/wp-includes/css/dist/block-library/style.min.css | grep -i -E '^(x-cache|age):'
P4
Highly available WordPress
when one instance is not enough0 / 0Fig 1 of the reference: managed database, media in a bucket, identical instances behind a Lightsail load balancer.
4.1 Create a high-availability managed databasePick an HA bundle and a MySQL blueprint from the lists; the master password is generated and can be read back.
aws lightsail get-relational-database-bundles --query "bundles[?contains(bundleId,'ha')].[bundleId,price,ramSizeInGb]" --output table
aws lightsail get-relational-database-blueprints --query 'blueprints[].[blueprintId,engine,engineVersion]' --output table
aws lightsail create-relational-database --relational-database-name wp-db \
--relational-database-blueprint-id REPLACE_DB_BLUEPRINT --relational-database-bundle-id REPLACE_HA_BUNDLE \
--master-database-name wordpress --master-username wpadmin --availability-zone ${AWS_REGION}a
aws lightsail get-relational-database-master-user-password --relational-database-name wp-db4.2 Move the database and repoint WordPressRead DB_NAME and DB_USER from wp-config.php, dump, load into the managed database, then set DB_HOST, DB_USER and DB_PASSWORD to the new values.
grep -E "DB_(NAME|USER|HOST)" /var/www/wp-config.php mysqldump --single-transaction --no-tablespaces -u WP_DB_USER -p WP_DB_NAME > wp.sql mysql -h WP_DB_ENDPOINT -u wpadmin -p wordpress < wp.sql
4.3 Move media into a bucketCreate the bucket, give the instances access keys scoped to it, and let an offload plugin write new uploads there and rewrite media URLs.
aws lightsail get-bucket-bundles --query 'bundles[].[bundleId,price,storagePerMonthInGb]' --output table aws lightsail create-bucket --bucket-name wp-media-$ACCOUNT --bundle-id REPLACE_BUCKET_BUNDLE aws lightsail create-bucket-access-key --bucket-name wp-media-$ACCOUNT
4.4 Clone the instance into two more AZs
aws lightsail create-instance-snapshot --instance-name wp1 --instance-snapshot-name wp1-ha
aws lightsail create-instances-from-snapshot --instance-snapshot-name wp1-ha \
--instance-names wp2 --availability-zone ${AWS_REGION}b --bundle-id $BUNDLE
aws lightsail create-instances-from-snapshot --instance-snapshot-name wp1-ha \
--instance-names wp3 --availability-zone ${AWS_REGION}c --bundle-id $BUNDLE4.5 Put a load balancer in front and make it the distribution originChange the distribution origin from wp1 to the load balancer in the console once all three instances report healthy.
aws lightsail create-load-balancer --load-balancer-name wp-lb --instance-port 80 --health-check-path / aws lightsail attach-instances-to-load-balancer --load-balancer-name wp-lb --instance-names wp1 wp2 wp3 aws lightsail get-load-balancer --load-balancer-name wp-lb \ --query 'loadBalancer.instanceHealthSummary[].[instanceName,instanceHealth]' --output table
P5
Rails app on Elastic Beanstalk with RDS
0 / 0Fig 3 of the reference. The database is created first and outside Beanstalk, with its own security group that only the app’s security group can reach.
5.1 Security groups: one for the app, one for the database
export VPC=$(aws ec2 describe-vpcs --filters Name=is-default,Values=true --query 'Vpcs[0].VpcId' --output text) export SUBNETS=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC --query 'Subnets[].SubnetId' --output text | tr '\t' ',') export APP_SG=$(aws ec2 create-security-group --group-name detekto-app --description "Detekto web servers" \ --vpc-id $VPC --query GroupId --output text) export DB_SG=$(aws ec2 create-security-group --group-name detekto-db --description "Detekto Postgres" \ --vpc-id $VPC --query GroupId --output text) aws ec2 authorize-security-group-ingress --group-id $DB_SG --protocol tcp --port 5432 --source-group $APP_SG
5.2 Create the PostgreSQL instanceDev settings for the build; RDS keeps the master password in Secrets Manager.
aws rds create-db-instance --db-instance-identifier detekto-db --engine postgres \ --db-instance-class db.t4g.micro --allocated-storage 20 --max-allocated-storage 100 \ --db-name detekto --master-username detekto --manage-master-user-password \ --vpc-security-group-ids $DB_SG --no-publicly-accessible --backup-retention-period 7 aws rds wait db-instance-available --db-instance-identifier detekto-db export DB_HOST=$(aws rds describe-db-instances --db-instance-identifier detekto-db \ --query 'DBInstances[0].Endpoint.Address' --output text) export DB_SECRET=$(aws rds describe-db-instances --db-instance-identifier detekto-db \ --query 'DBInstances[0].MasterUserSecret.SecretArn' --output text)
5.3 Create the image bucketBlock Public Access is on by default; the app reads and writes through its role.
export BUCKET=detekto-images-$ACCOUNT aws s3 mb s3://$BUCKET
5.4 Give the Beanstalk instance role the app’s permissionsaws-elasticbeanstalk-ec2-role is the default instance profile; it exists after the first environment or the first console visit.
cat > detekto-app.json <<EOF
{"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":"arn:aws:s3:::$BUCKET/*"},
{"Effect":"Allow","Action":"rekognition:DetectLabels","Resource":"*"},
{"Effect":"Allow","Action":"secretsmanager:GetSecretValue","Resource":"$DB_SECRET"}]}
EOF
aws iam put-role-policy --role-name aws-elasticbeanstalk-ec2-role --policy-name detekto-app \
--policy-document file://detekto-app.json5.5 Initialise the project for the Ruby platform
cd detekto eb init detekto --platform ruby --region $AWS_REGION eb platform list | grep -i ruby
5.6 Create the production environmentTwo instances behind an ALB, the app security group attached, connection details passed as properties.
eb create detekto-prod --elb-type application --instance_type t3.small --scale 2 \ --vpc.id $VPC --vpc.ec2subnets $SUBNETS --vpc.elbsubnets $SUBNETS \ --vpc.securitygroups $APP_SG --vpc.elbpublic --vpc.publicip \ --envvars DB_HOST=$DB_HOST,DB_NAME=detekto,DB_SECRET_ARN=$DB_SECRET,IMAGES_BUCKET=$BUCKET
5.7 Deploy, open, watch health
eb deploy eb open eb health --refresh
5.8 Check Rekognition on a test image from the CLI
aws s3 cp hotdog.jpg s3://$BUCKET/test/hotdog.jpg
aws rekognition detect-labels --image "{\"S3Object\":{\"Bucket\":\"$BUCKET\",\"Name\":\"test/hotdog.jpg\"}}" \
--max-labels 5 --query 'Labels[].[Name,Confidence]' --output tableP6
Scale under load
0 / 0Triggers, schedules and rolling updates live in .ebextensions so they travel with the code.
6.1 Add the scaling configuration and deploy itUse .ebextensions/scaling.config from the reference sheet: CPU trigger at 50 and 20 percent, 1-minute window, rolling updates.
mkdir -p .ebextensions vim .ebextensions/scaling.config # paste the block from the reference sheet eb deploy
6.2 Add a lunchtime scheduled actionRecurrence is cron in UTC: 08:30 UTC is 11:30 in Sofia and Kyiv in summer, 10:30 in winter.
option_settings:
- namespace: aws:autoscaling:scheduledaction
resource_name: LunchPeak
option_name: MinSize
value: '4'
- namespace: aws:autoscaling:scheduledaction
resource_name: LunchPeak
option_name: Recurrence
value: '30 8 * * 1-5'
- namespace: aws:autoscaling:scheduledaction
resource_name: AfterLunch
option_name: MinSize
value: '2'
- namespace: aws:autoscaling:scheduledaction
resource_name: AfterLunch
option_name: Recurrence
value: '30 11 * * 1-5'6.3 Load test with JMeter and watch the fleetRamp up slowly; the instance count should follow CPU within a couple of minutes and come back down after the test.
jmeter -n -t detekto.jmx -Jhost=$(eb status | awk '/CNAME/ {print $2}') -l results.jtl -e -o report/
eb health --refresh6.4 Check CPU credits on the burstable instancesA falling CPUCreditBalance under steady load says the family is wrong for this app, not that it needs more instances.
ASG=$(aws elasticbeanstalk describe-environment-resources --environment-name detekto-prod \ --query 'EnvironmentResources.AutoScalingGroups[0].Name' --output text) aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUCreditBalance \ --dimensions Name=AutoScalingGroupName,Value=$ASG --statistics Average --period 300 \ --start-time $(date -u -d '-3 hours' +%FT%TZ) --end-time $(date -u +%FT%TZ) --output table
6.5 Change the instance type with rolling updates on
aws elasticbeanstalk update-environment --environment-name detekto-prod \ --option-settings Namespace=aws:ec2:instances,OptionName=InstanceTypes,Value=m7i.large
6.6 Resize the database in the maintenance window, not now
aws rds modify-db-instance --db-instance-identifier detekto-db --db-instance-class db.t4g.medium --no-apply-immediately aws rds describe-db-instances --db-instance-identifier detekto-db --query 'DBInstances[0].PendingModifiedValues'
6.7 Turn on Multi-AZ for productionThe conversion runs online; later class changes then finish with a short failover instead of a full outage.
aws rds modify-db-instance --db-instance-identifier detekto-db --multi-az --apply-immediately
P7
Cost down while scaling
0 / 0Fig 4, 7 and 9 of the reference, as actions.
7.1 Create a small developer cloneSingle instance, no load balancer. Create detekto-dev-db first as in step 5.2 (db.t4g.micro) and export DEV_DB_HOST and DEV_DB_SECRET.
eb create detekto-dev --single --instance_type t3.micro \ --vpc.id $VPC --vpc.ec2subnets $SUBNETS --vpc.securitygroups $APP_SG --vpc.publicip \ --envvars DB_HOST=$DEV_DB_HOST,DB_NAME=detekto,DB_SECRET_ARN=$DEV_DB_SECRET,IMAGES_BUCKET=$BUCKET
7.2 Scale the dev environment to zero outside office hoursScheduled actions inside Beanstalk; Instance Scheduler would stop the instance and Auto Scaling would replace it.
option_settings:
- namespace: aws:autoscaling:scheduledaction
resource_name: Night
option_name: MinSize
value: '0'
- namespace: aws:autoscaling:scheduledaction
resource_name: Night
option_name: MaxSize
value: '0'
- namespace: aws:autoscaling:scheduledaction
resource_name: Night
option_name: Recurrence
value: '0 17 * * 1-5'
- namespace: aws:autoscaling:scheduledaction
resource_name: Morning
option_name: MinSize
value: '1'
- namespace: aws:autoscaling:scheduledaction
resource_name: Morning
option_name: MaxSize
value: '1'
- namespace: aws:autoscaling:scheduledaction
resource_name: Morning
option_name: Recurrence
value: '0 5 * * 1-5'7.3 Stop the dev database with Instance SchedulerDeploy Instance Scheduler on AWS from the Solutions library, install its scheduler-cli, define the schedule, tag the database.
scheduler-cli create-period --stack instance-scheduler --name office-hours \ --begintime 08:00 --endtime 19:00 --weekdays mon-fri scheduler-cli create-schedule --stack instance-scheduler --name dev-office \ --periods office-hours --timezone Europe/Sofia aws rds add-tags-to-resource --resource-name arn:aws:rds:$AWS_REGION:$ACCOUNT:db:detekto-dev-db \ --tags Key=Schedule,Value=dev-office
7.4 Run the surplus web servers on SpotTwo On-Demand instances stay as the base; everything above is Spot across several types.
option_settings:
aws:ec2:instances:
EnableSpot: true
InstanceTypes: t3.small,t3a.small,t3.medium
SpotFleetOnDemandBase: 2
SpotFleetOnDemandAboveBasePercentage: 07.5 Commit the steady baseRead the recommendation, buy for the floor of the curve, not the average. For the database compare an RDS RI with a Database Savings Plan.
aws ce get-savings-plans-purchase-recommendation --savings-plans-type COMPUTE_SP \ --term-in-years ONE_YEAR --payment-option NO_UPFRONT --lookback-period-in-days THIRTY_DAYS aws rds describe-reserved-db-instances-offerings --db-instance-class db.t4g.medium \ --product-description postgresql --duration 1 --offering-type "No Upfront" \ --query 'ReservedDBInstancesOfferings[].[MultiAZ,RecurringCharges[0].RecurringChargeAmount]' --output table
7.6 Let old images tier down by themselves
aws s3api put-bucket-lifecycle-configuration --bucket $BUCKET --lifecycle-configuration \
'{"Rules":[{"ID":"intelligent-tiering","Status":"Enabled","Filter":{},"Transitions":[{"Days":0,"StorageClass":"INTELLIGENT_TIERING"}]}]}'7.7 Release idle public IPv4 addressesUnattached Elastic IPs and Lightsail static IPs are billed by the hour.
aws ec2 describe-addresses --query 'Addresses[?AssociationId==null].[PublicIp,AllocationId]' --output table aws lightsail get-static-ips --query 'staticIps[?isAttached==`false`].[name,ipAddress]' --output table
P8
Serverless image labelling with SAM
0 / 0Fig 13 of the reference: uploads land in S3, an event runs the function, Rekognition labels the image, DynamoDB keeps the result.
8.1 Create the project filestemplate.yaml and src/app.py exactly as in the reference sheet (Serverless section).
mkdir -p detekto-labels/src && cd detekto-labels # write template.yaml and src/app.py from the reference sheet sam validate --lint
8.2 Build and deployThe first deploy is guided and saves its answers in samconfig.toml; use a lowercase stack name, it becomes part of the bucket name.
sam build sam deploy --guided --stack-name detekto-labels --capabilities CAPABILITY_IAM
8.3 Upload a test image and read the labels
LBUCKET=$(aws cloudformation describe-stacks --stack-name detekto-labels \
--query "Stacks[0].Outputs[?OutputKey=='Bucket'].OutputValue" --output text)
TABLE=$(aws cloudformation describe-stack-resource --stack-name detekto-labels \
--logical-resource-id LabelsTable --query StackResourceDetail.PhysicalResourceId --output text)
aws s3 cp hotdog.jpg s3://$LBUCKET/uploads/hotdog.jpg
sam logs --stack-name detekto-labels --tail
aws dynamodb get-item --table-name $TABLE --key '{"id":{"S":"uploads/hotdog.jpg"}}'8.4 Point the app at the new bucket and tableThe web tier now only uploads and later reads the labels; give its role GetItem on the table.
eb setenv IMAGES_BUCKET=$LBUCKET LABELS_TABLE=$TABLE
8.5 Take Rekognition away from the web serversLeast privilege: only the function needs DetectLabels now. Edit detekto-app.json and put the policy again.
aws iam put-role-policy --role-name aws-elasticbeanstalk-ec2-role --policy-name detekto-app \ --policy-document file://detekto-app.json
P9
Path-based routing: /api to Lambda
0 / 0Fig 14 of the reference. Use a shared ALB that you own (eb create –shared-lb), so the rule does not belong to a Beanstalk environment.
9.1 Create a Lambda target group and allow the ALB to invoke the function
FN_ARN=$(aws lambda get-function --function-name detekto-api --query Configuration.FunctionArn --output text) TG=$(aws elbv2 create-target-group --name detekto-api --target-type lambda \ --query 'TargetGroups[0].TargetGroupArn' --output text) aws lambda add-permission --function-name detekto-api --statement-id alb-invoke \ --action lambda:InvokeFunction --principal elasticloadbalancing.amazonaws.com --source-arn $TG aws elbv2 register-targets --target-group-arn $TG --targets Id=$FN_ARN
9.2 Add the path rule on the HTTPS listener
LB=$(aws elbv2 describe-load-balancers --names shared-web --query 'LoadBalancers[0].LoadBalancerArn' --output text) LISTENER=$(aws elbv2 describe-listeners --load-balancer-arn $LB \ --query "Listeners[?Port==\`443\`].ListenerArn" --output text) aws elbv2 create-rule --listener-arn $LISTENER --priority 10 \ --conditions Field=path-pattern,Values='/api/*' --actions Type=forward,TargetGroupArn=$TG
9.3 Test both paths/api answers from the function, every other path from Rails.
curl -s -o /dev/null -w '%{http_code} %{url_effective}\n' https://app.$DOMAIN/api/health
curl -s -o /dev/null -w '%{http_code} %{url_effective}\n' https://app.$DOMAIN/P10
Operate and clean up
0 / 0The checks that keep the build cheap and up after the build is done.
10.1 Alarm on 5xx answers from the app
aws cloudwatch put-metric-alarm --alarm-name detekto-5xx --namespace AWS/ApplicationELB \ --metric-name HTTPCode_Target_5XX_Count --dimensions Name=LoadBalancer,Value=app/shared-web/0123456789abcdef \ --statistic Sum --period 60 --evaluation-periods 5 --threshold 10 \ --comparison-operator GreaterThanThreshold --alarm-actions arn:aws:sns:$AWS_REGION:$ACCOUNT:ops
10.2 Look at the bill by service once a month
aws ce get-cost-and-usage --time-period Start=2026-10-01,End=2026-11-01 --granularity MONTHLY \ --metrics UnblendedCost --group-by Type=DIMENSION,Key=SERVICE \ --query 'ResultsByTime[0].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' --output table
10.3 Keep only the last 50 application versions
aws elasticbeanstalk update-application-resource-lifecycle --application-name detekto \
--resource-lifecycle-config 'VersionLifecycleConfig={MaxCountRule={Enabled=true,MaxCount=50,DeleteSourceFromS3=true}}'10.4 Retire the old hosting after the cutoverKeep the last snapshot of the old site for a month, then delete it together with unused Lightsail snapshots.
aws lightsail get-instance-snapshots --query 'instanceSnapshots[].[name,createdAt,sizeInGb]' --output table
PB
Playbooks
For the day something goes wrong. Ticks here do not count toward the runbook progress.
PB1 WordPress broken after the import
when white screen, endless redirects or a lost admin login after step 2.4
Launch the pre-import snapshot as a new instance and move the static IP to it
aws lightsail create-instances-from-snapshot --instance-snapshot-name wp1-fresh \
--instance-names wp1b --availability-zone ${AWS_REGION}a --bundle-id $BUNDLE
aws lightsail detach-static-ip --static-ip-name wp1-ip
aws lightsail attach-static-ip --static-ip-name wp1-ip --instance-name wp1bFind the cause on the broken instance before retryingUsual suspects: upload limit hit mid-import, a plugin that pins the old domain, WP_HOME still on the old URL, mixed plugin versions.
sudo tail -n 50 /var/log/apache2/error.log
PB2 A Beanstalk deployment broke production
when health turns Severe right after eb deploy
Redeploy the previous version label
aws elasticbeanstalk describe-application-versions --application-name detekto --max-items 5 \ --query 'ApplicationVersions[].[VersionLabel,DateCreated]' --output table eb deploy detekto-prod --version PREVIOUS_LABEL
Read the logs of the failed version
eb logs detekto-prod --all
Next time use immutable or blue/green deploymentsAn immutable deployment never touches the running instances; eb clone plus eb swap gives a one-command rollback.
PB3 Spot capacity disappears
when many Spot interruptions in a row, or Spot requests stay unfulfilled
Widen the type list and raise the On-Demand base for now
aws elasticbeanstalk update-environment --environment-name detekto-prod --option-settings \ Namespace=aws:ec2:instances,OptionName=InstanceTypes,Value=t3.small,t3a.small,t3.medium,m7i.large \ Namespace=aws:ec2:instances,OptionName=SpotFleetOnDemandBase,Value=4
PB4 Database CPU pinned at 100%
when page times climb while web CPU stays low
Find the expensive queries before buying a bigger classPerformance Insights shows the top SQL by load; a missing index usually beats a bigger instance.
If it is volume, scale: bigger class in the window, a read replica for reads, RDS Proxy for connection storms
FIX
Troubleshooting
| Symptom | Cause | Fix | Step |
|---|---|---|---|
| Import stops with a size error | PHP upload limit below the export size | raise upload_max_filesize and post_max_size, restart Apache | 2.3 |
| Cannot log in after the import | users table replaced by the old site’s | use the old site’s credentials | 2.4 |
| Redirect loop behind the CDN | WordPress sees HTTP while the visitor used HTTPS | add the CloudFront-Forwarded-Proto check to wp-config.php | 3.8 |
| Certificate stuck in pending validation | CNAME records missing or added with the domain twice | compare the record names with get-certificates output | 3.4, 3.5 |
| Site reachable by IP, not by name | registrar still on the old name servers | set the Lightsail name servers, wait for propagation | 3.1 |
| CSS changes do not show | distribution cache | reset the distribution cache after theme updates | 3.9 |
| App cannot reach the database | DB security group does not allow the app security group | ingress 5432 from APP_SG | 5.1 |
| AccessDenied on S3 or Rekognition from the app | instance role missing the permission | put-role-policy on aws-elasticbeanstalk-ec2-role | 5.4 |
| All instances replaced at once on a type change | rolling updates off | enable rolling updates by health | 6.1 |
| Fast for an hour, then slow, at steady load | T2 credits exhausted | unlimited mode or a non-burstable family | 6.4 |
| Dev instance keeps coming back after being stopped | Auto Scaling replaces stopped instances | scheduled actions to zero instead of stopping | 7.2 |
| sam deploy fails with a circular dependency | bucket referenced with !Ref in the function policy | fixed bucket name with !Sub, as in the template | 8.1 |
| Function runs but labels never appear | role lacks DynamoDB or Rekognition rights, or the event filter does not match | sam logs, check the policy templates | 8.3 |
| /api returns 502 from the ALB | Lambda response not in ALB format, or invoke permission missing | return statusCode, headers, body; add-permission for elasticloadbalancing | 9.1 |
AWS Build Series Runbook · REV 2026.10
© 2026 · BUB.IM · made in vim
0 comments