Start with an application
A customer opens your website. A computer runs the application that answers their request. On AWS, that virtual computer is called an EC2 instance. What happens if it fails, or if more customers arrive than it can handle?
Build the answer below: first one application, then several copies sharing requests, then an Auto Scaling group to keep the computers running. Each step introduces a term after showing what it does. Later chapters explain how demand, startup time, available supply, and updates change the same system.
Follow the explanation
The page now scrolls through separate visuals. The first five introduce one application, a load balancer, the Auto Scaling group, replacement startup, and the three capacity counts. The later visuals are collected into demand, capacity supply, fleet changes, and diagnosis. You can read them in order or use the section links to revisit one mechanism.
Within an interactive visual, use Next step to inspect its explanation. The “Read every step” section includes the alternate examples as a readable reference. Playback walks through authored examples and comparisons; it does not call AWS or simulate every possible event ordering. Times and workload figures are illustrative.
Build the system
Begin with one application answering one customer. Add shared traffic, automatic replacement, startup states, timers, and health evidence in that order.
You can identify each component and explain what “ready” means at its boundary.One computer runs the application
A customer asks your application for a page. The application runs on an EC2 instance, which is a virtual computer in AWS.
If this computer stops, there is no other application copy to answer the next request.
Several copies can share the work
Run the same application on several EC2 instances. An Application Load Balancer receives requests and chooses a healthy registered destination.
The load balancer routes traffic. It does not create a new computer when one disappears.
Ask an Auto Scaling group to maintain the fleet
An Auto Scaling group, or ASG, manages EC2 instances. You give it a desired capacity: the number of instances you want it to maintain.
The ASG sees five healthy members against a desired count of six, so it uses the launch template to request a replacement.
A replacement exists before it is useful
A launch request begins a sequence. The virtual computer must be created, boot, start the application, and pass the checks used by this workload.
- Five readyExisting copies keep serving
- Launch requestedASG works toward six
- Computer startingEC2 exists but the app is preparing
- Health checkingLoad balancer waits for a good response
- Six readyThe replacement can receive traffic
Each milestone answers a different question. “EC2 is running” does not yet mean “the application can answer customers.”
Now the capacity counts have a meaning
Look at the moment when the replacement computer exists but its application is still starting.
The counts disagree because they describe different milestones. Serving becomes six only after the replacement application is ready.
Keep a web application running
A customer opens your website. An EC2 instance is the virtual computer running the application that answers them. Start with that one computer, then add the pieces needed to share work and replace a failed machine.
Build the system one piece at a time. Follow a customer request before introducing the group that keeps the computers running.
- 01One applicationA customer sends a request
- 02Share the requestsAdd application copies and a load balancer
- 03Keep six runningGive an Auto Scaling group a target
- 04Prepare the replacementBoot → start the app → check readiness
- 05Name the three countsDesired 6 · present 6 · serving 5
Direct sequence separate clock or conceptual relationship
An EC2 instance is a virtual computer. Your application runs on it and sends a response to the customer. If that one computer fails, this example has no other application copy to answer.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceThis worked example assumes five existing healthy application copies and one replacement still starting.
Setting in playDesiredCapacity = 6; the launch template supplies each new instance’s configuration.
Read every step and alternate example
Start here: one application
Build the system one piece at a time. Follow a customer request before introducing the group that keeps the computers running.
- One application
An EC2 instance is a virtual computer. Your application runs on it and sends a response to the customer. If that one computer fails, this example has no other application copy to answer.
- Share the requests
Run the application on more instances. An Application Load Balancer receives customer requests and sends them to healthy registered application destinations, called targets. It distributes traffic; it does not create replacement computers.
- Keep six running
An Auto Scaling group, or ASG, manages a set of EC2 instances. You ask it to maintain six. When one fails, the group can launch a replacement using a launch template: the saved configuration for creating a machine.
- Prepare the replacement
The replacement exists before it can answer requests. EC2 boots, the application starts, and the load balancer checks it. During that preparation, the other five healthy application copies carry the traffic.
- Name the three counts
Now the terms describe something you have seen: desired is the six instances requested; present is the five existing machines plus the replacement; serving is the five application copies ready for requests. When the replacement is ready, serving also becomes six.
Scale from 6 to 8
The request changes immediately. Instances and usable capacity arrive later.
- Desired 8
The policy has requested eight instances, but a request cannot serve a customer. Desired capacity is the count ASG is trying to maintain.
- Present 6
ASG starts two launch attempts. Present capacity counts machines that actually exist, so failed or unfinished attempts cannot fill the gap.
- In service 6
InService is ASG’s active membership state. Here the six existing instances remain active while the new launches are pending.
- Serving 6
Serving counts the instances that can handle this application’s requests. Six are ready; the requested eight will only help once the new application copies are ready.
Launches blocked
A failed launch does not become an instance, and the unmet request remains.
- Desired 8
ASG still wants eight instances after a failed attempt. Failure to fulfill a request does not automatically lower that target.
- Attempts 2
The eligible subnets have no free network addresses. These attempts return no instance IDs, so there are no new computers to count.
- Present 6
Six existing machines remain. Look at the failed scaling activity to explain why present capacity has not reached desired.
- Serving 6
The existing healthy application copies keep answering requests. Restore launch capacity to close the two-instance gap.
Unhealthy member
Replacement restores the target even when no scaling policy asks for more capacity.
- Desired 6
The target stays six. Replacing a failed member does not require asking the group to grow.
- Unhealthy 1
A configured health source reports one instance unhealthy. ASG can use that evidence to request replacement.
- Replacement
ASG starts another machine. A new instance ID means new capacity is being created, not that the failed machine moved between zones.
- Serving 5
Only five application copies are ready while replacement starts. Serving capacity recovers after the new application is prepared and healthy.
Finish preparation before joining
A new computer can boot before your application has downloaded its configuration. A lifecycle hook is a pause in the group’s launch or termination transition. Your preparation program completes the hook when its work is done; receiving a notification alone does not release the pause.
The hook holds the ASG transition while a worker prepares the instance.
- 01PendingEC2 launch accepted
- 02Pending:WaitHook token is active
- 03WorkerDomain join completed
- 04Pending:ProceedCONTINUE received
- 05InServiceEligible for registration
Direct sequence separate clock or conceptual relationship
EC2 has accepted the launch. Pending means the instance has not yet entered ASG’s active InService state.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceDelivery and lifecycle completion are separate events.
Setting in playLifecycleHookSpecificationList[].HeartbeatTimeout
Read every step and alternate example
Launch hook completes
The hook holds the ASG transition while a worker prepares the instance.
- Pending
EC2 has accepted the launch. Pending means the instance has not yet entered ASG’s active InService state.
- Pending:Wait
The launch hook holds the transition in Pending:Wait. Its action token identifies this particular wait so a worker can complete the right action.
- Worker
A preparation program joins the instance to the required domain. Delivering the notification started work; the work itself still had to succeed.
- Pending:Proceed
The worker sends CONTINUE for the current action. ASG can now proceed beyond this hook rather than waiting for its timeout.
- InService
ASG admits the instance into active service. Load-balancer registration and application checks still determine whether it can receive requests.
Duplicate delivery
At-least-once notification delivery must not repeat application work.
- Pending:Wait
One lifecycle action is waiting for preparation. Several messages about it do not create several independent actions.
- Message 1
The worker claims the task and records the action token so it can recognize repeat deliveries.
- Message 2
The same notification arrives again. The worker checks its record and avoids running the domain join twice.
- CONTINUE
The worker completes the original action once. Duplicate delivery has not duplicated the application-side work.
Termination retained
A failed termination action can retain the EC2 instance outside active capacity.
- Deregister
Deregistration takes the target out of new traffic selection while existing requests drain in the ordinary termination path.
- Terminating:Wait
The termination hook holds the transition so the cleanup program can finish its work.
- ABANDON
The worker reports ABANDON because cleanup failed. The configured retention rule now determines the resource’s disposition.
- Retained
The retain setting preserves this EC2 instance outside active capacity. Someone must resolve it; retention is not free replacement capacity.
Give startup time to finish
Your application needs time to settle after startup. A health-check grace period delays covered replacement decisions; warmup delays the use of new capacity in scaling calculations. These clocks have different jobs even when they overlap.
Health checks keep running while grace and warmup gate different consumers.
- 01Boot 0–55sOperating system starts
- 02Hook 10–100sASG transition held
- 03Grace 100–400sFrom entry into InService
- 04Warmup 100–280sScaling metric eligibility waits
- 05Serving at 130sTarget readiness is separate
Direct sequence separate clock or conceptual relationship
The operating system is starting. A running VM alone does not prove the application is ready.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceEach bar begins at its documented trigger, not at page time zero.
Setting in playDefaultInstanceWarmup = 180; HealthCheckGracePeriod = 300
Read every step and alternate example
Startup clocks overlap
Health checks keep running while grace and warmup gate different consumers.
- Boot 0–55s
The operating system is starting. A running VM alone does not prove the application is ready.
- Hook 10–100s
The launch hook pauses admission into active service while preparation continues inside the machine.
- Grace 100–400s
Once the instance enters InService, the grace period gives covered startup health failures time to recover before ASG acts on them.
- Warmup 100–280s
Warmup starts from InService in this example. Its 180 seconds govern scaling eligibility, independently of the application’s traffic checks.
- Serving at 130s
At 130 seconds the target passes readiness. It can receive traffic even though its scaling warmup has not expired.
Stopped during grace
Leaving EC2 running can cause immediate replacement despite the startup grace period.
- EC2 running
This instance has entered InService and is within its configured startup grace period.
- Instance stopped
The VM is stopped. This is an EC2 state change, not simply an application taking longer to answer a health check.
- Health action
ASG can act immediately on an instance that leaves running. Increasing the grace period does not keep a stopped VM usable.
Check whether the application works
A running computer can host a broken application. Health checks are tests of different layers: EC2 checks the machine, while a load balancer checks a registered application destination, called a target. ASG uses configured health evidence to decide whether to replace an instance.
The VM runs, but the target and aggregated application status fail.
- 01EC2 systemPassed
- 02EC2 instancePassed
- 03ApplicationFailed
- 04Target groupUnhealthy
- 05ASG healthReplacement after grace
Direct sequence separate clock or conceptual relationship
EC2 system checks test the infrastructure supporting the VM. Passing them narrows the fault location but says little about application behavior.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceRaw target reason and aggregated application status remain visible.
Setting in playHealthCheckType = EC2,ELB
Read every step and alternate example
Application fails
The VM runs, but the target and aggregated application status fail.
- EC2 system
EC2 system checks test the infrastructure supporting the VM. Passing them narrows the fault location but says little about application behavior.
- EC2 instance
EC2 instance checks also pass. The machine is reachable at that layer even though its application may fail.
- Application
The application check fails. Its raw result is separate from the machine checks and can contribute to aggregated application status.
- Target group
The load balancer’s target check fails too. With healthy alternatives available in this example, routing can avoid this destination.
- ASG health
With the relevant health evidence enabled, ASG can replace the instance after applicable grace handling. Routing and replacement are separate decisions.
Check excluded
An excluded EC2 application check does not change the aggregated application status.
- EC2 system
The machine-level check passes, so this example is about application-check aggregation.
- Application
The application check fails but is excluded from the aggregate. Exclusion changes which evidence contributes, not the raw test result.
- Aggregation
The remaining included checks pass. That produces a passing aggregate despite the excluded failure.
- ASG health
ASG has no failing aggregate from this check to act on. The application problem still needs investigation.
Turn demand into a request
See how measurements, alarms, queue depth, schedules, and forecasts ask the group to change its size.
You can identify who asked for a capacity change and what evidence it used.Add computers when they get busy
A metric is a measured value, such as CPU utilization. A target-tracking policy asks ASG to adjust instance count toward a chosen metric value. Scale out means add capacity; scale in means remove it. Two policies can disagree about which is needed.
A scale-out recommendation can proceed when any target-tracking policy needs capacity.
- 01CPU 78 / 50%+3 proposed
- 02Requests 920 / 800+1 proposed
- 03Final request+3 capacity
- 04WarmupNew capacity excluded
Direct sequence separate clock or conceptual relationship
CPU is above the configured target. This fixture assigns the CPU policy a proposal of three more instances; it does not reproduce AWS’s private controller formula.
Step 1 of 4
Inspect the evidence and effective setting
EvidenceThe panel shows each policy proposal without claiming AWS controller internals.
Setting in playTargetTrackingConfiguration.TargetValue
Read every step and alternate example
Either policy scales out
A scale-out recommendation can proceed when any target-tracking policy needs capacity.
- CPU 78 / 50%
CPU is above the configured target. This fixture assigns the CPU policy a proposal of three more instances; it does not reproduce AWS’s private controller formula.
- Requests 920 / 800
Requests per target also exceed their target. That policy proposes one extra instance, a different request based on different evidence.
- Final request
The larger scale-out proposal determines this example’s increase. Any target-tracking policy can support scale-out.
- Warmup
The newly launched capacity must warm up before it contributes normally to the scaling calculation. The request has changed before its full effect is measurable.
Scale-in disagreement
Target-tracking policies with scale-in enabled must agree before capacity falls.
- CPU 22 / 50%
Low CPU supports removing two instances in this authored example.
- Requests 870 / 800
The requests policy still needs capacity. Low CPU does not establish that the application has spare request-handling capacity.
- Final request
The scale-in-enabled target-tracking policies do not agree, so the CPU proposal cannot shrink the fleet.
Choose how much capacity to add
An alarm reports that a measurement crossed a threshold. Step scaling chooses an adjustment based on how far it crossed; simple scaling uses an adjustment followed by a cooldown before another simple-policy action. The intervals below are alternatives, not successive launches.
The metric is 23 points above the alarm threshold.
- 010 to 10+1
- 0210 to 20+2
- 0320 or more+30%
- 0410 instancesRaw result +3
- 05Desired 13Request accepted
Direct sequence separate clock or conceptual relationship
For a breach less than ten points above the threshold, this policy would add one instance. That interval does not match the current 23-point breach.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceMetric − threshold = 23.
Setting in playStepAdjustments[].MetricIntervalLowerBound = 20
Read every step and alternate example
Step boundary
The metric is 23 points above the alarm threshold.
- 0 to 10
For a breach less than ten points above the threshold, this policy would add one instance. That interval does not match the current 23-point breach.
- 10 to 20
The next interval covers breaches from ten up to twenty points. It also does not match 23; these are alternative rules.
- 20 or more
The 23-point breach matches the interval starting at twenty, which requests a thirty-percent increase.
- 10 instances
Thirty percent of the ten-instance starting fleet is three. This example needs no fractional rounding.
- Desired 13
The resulting target is thirteen, subject to the group’s limits. It is a requested count, not thirteen immediately usable machines.
Simple cooldown
A repeated alarm breach waits for the policy cooldown; replacement does not use that gate.
- Alarm breach
The alarm triggers a simple scaling action that requests two additional instances.
- Cooldown
The applicable cooldown is still active. It spaces simple-policy actions so the previous change has time to affect demand.
- Health failure
An instance fails health checks during that wait. Replacement is a separate path and can proceed.
- Next breach
Another breach occurs, but this simple-policy action is gated by cooldown. The group itself has not been frozen.
Give queued jobs enough workers
Some applications receive jobs through a queue instead of a load balancer. A worker takes a job, processes it, and acknowledges completion. Backlog is work still waiting; dividing it by usable workers helps reveal whether each worker has too much to do.
At two seconds per job and 60 seconds acceptable delay, the illustrative target is 30 messages per usable worker.
- 01Arrivals18 messages/s
- 02Visible backlog240
- 03Usable workers4
- 04Backlog / worker60
- 05Target30 → scale out
Direct sequence separate clock or conceptual relationship
Jobs arrive at the queue at the assumed rate. A queue lets work wait until a worker can take it.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceService time and queue-delay assumptions are explicit.
Setting in playCustomizedMetricSpecification · backlog / usable workers
Read every step and alternate example
Backlog target
At two seconds per job and 60 seconds acceptable delay, the illustrative target is 30 messages per usable worker.
- Arrivals
Jobs arrive at the queue at the assumed rate. A queue lets work wait until a worker can take it.
- Visible backlog
There are 240 visible waiting messages. Work already being processed belongs in a separate in-flight count.
- Usable workers
Four workers can take jobs. Use comparable, usable workers as the denominator for this teaching model.
- Backlog / worker
Dividing 240 by four gives sixty waiting messages per worker, twice the illustrative target.
- Target
At two seconds per job, a sixty-second acceptable delay suggests thirty messages per worker. More usable workers can reduce this ratio, subject to downstream limits.
Zero workers
Division by zero and absent CPU samples need an independent wake-up path.
- Visible backlog
Twelve messages are waiting, so there is work even though the fleet is empty.
- Usable workers
There are no usable workers. Backlog per worker cannot be calculated by dividing by zero.
- CPU metric
There are also no workers publishing CPU samples. An absent sample does not mean idle machines exist.
- Wake-up alarm
An independent queue-depth alarm requests the first worker. Once workers exist, the per-worker metric can become useful.
Prepare for demand you expect
If traffic arrives every morning, waiting for busy computers may be too late. A schedule changes capacity at a chosen time. Predictive scaling uses historical measurements to forecast future demand; dynamic policies can still react when actual demand differs.
A recurring schedule raises the floor before the morning load.
- 0108:45 localSchedule executes
- 02Min 6Desired clamps upward
- 0309:00 localTraffic begins
- 04Serving 6Prepared fleet
Direct sequence separate clock or conceptual relationship
The scheduled action runs at the configured local time. The time zone matters when converting recurring occurrences to UTC.
Step 1 of 4
Inspect the evidence and effective setting
EvidenceInspect the scheduled action’s recurrence and time zone to establish when it will run.
Setting in playPutScheduledUpdateGroupAction.TimeZone
Read every step and alternate example
Business hours
A recurring schedule raises the floor before the morning load.
- 08:45 local
The scheduled action runs at the configured local time. The time zone matters when converting recurring occurrences to UTC.
- Min 6
Raising minimum to six also raises a desired count that was below six. This starts preparation before the traffic arrives.
- 09:00 local
Customers arrive at nine in this example. The benefit depends on how long startup actually takes.
- Serving 6
Here six application copies finished preparation in time. A schedule requests capacity early; it cannot guarantee launches will succeed.
Forecast miss
Predictive capacity arrives early, but dynamic scaling handles demand above the fixture forecast.
- History
Predictive scaling needs historical measurements before it can forecast. The fixture supplies sufficient history.
- Forecast
The authored forecast calls for eight instances. Forecast demand is a prediction, separate from actual arrivals.
- Prelaunch
Prelaunch requests that capacity before it is expected to be needed, allowing time for startup.
- Actual demand
Actual demand instead needs eleven instances under the workload assumptions. The forecast was too low.
- Dynamic scale-out
Dynamic scaling requests the additional three. The application still waits for those new instances to become usable.
Find capacity to launch
Follow a request through prepared instances, zones, instance types, purchase choices, Spot supply, and reservations.
You can tell a policy decision from a launch destination or capacity shortage.Prepare spare computers in advance
When setup takes minutes, you can prepare instances before you need them. A warm pool holds those instances outside the active fleet, often stopped. Starting one still requires restart and application-readiness work.
The instance restarts from the pool, then finishes hooks and readiness.
- 01Warm poolStopped · prepared
- 02RestartEC2 returns to running
- 03Launch hookConfiguration check
- 04InServiceActive ASG member
- 05Target readyRegistered and healthy
Direct sequence separate clock or conceptual relationship
The pool contains an instance that has already performed preparation and is now stopped. It contributes no running application service yet.
Step 1 of 5
Inspect the evidence and effective setting
EvidencePool population and active desired capacity use separate ledgers.
Setting in playInstanceReusePolicy.ReuseOnScaleIn = true
Read every step and alternate example
Reuse prepared instance
The instance restarts from the pool, then finishes hooks and readiness.
- Warm pool
The pool contains an instance that has already performed preparation and is now stopped. It contributes no running application service yet.
- Restart
EC2 restarts it. This avoids some cold initialization but still takes time.
- Launch hook
A launch hook can verify current configuration before returning the prepared machine to active service.
- InService
The instance enters InService after the hook completes. Target registration and health checks must still finish.
- Target ready
Once the registered target is healthy, the application can use it. Pool preparation reduced work without removing every startup gate.
Pool depleted
Prepared capacity is consumed first; remaining demand falls back to cold launch.
- Demand +4
The group requests four more active instances. This is separate from the pool’s desired population.
- Pool 2
Only two prepared instances are actually available to restart, even if the pool target was larger.
- Cold launch 2
The remaining two must take the cold launch path, including full application setup.
- Serving gap
The prepared pair can become useful earlier. The remaining serving gap persists until cold launches finish.
Choose where new computers can run
An Availability Zone is a separate location within an AWS Region. A subnet supplies network addresses within one zone. Spreading instances across zones reduces dependence on one location, but a launch still needs both an address and available EC2 capacity.
One zone is eligible for the group but its selected subnet has no free addresses.
- 01us-east-1a2 instances · 18 IPs
- 02us-east-1b2 instances · 0 IPs
- 03us-east-1c2 instances · 11 IPs
- 04Retry 1cLaunch accepted
Direct sequence separate clock or conceptual relationship
The first zone has room in its subnet. Free IP addresses are one launch prerequisite.
Step 1 of 4
Inspect the evidence and effective setting
EvidencePlacement failure is distinct from instance-type capacity.
Setting in playCapacityDistributionStrategy = balanced-best-effort
Read every step and alternate example
Subnet exhausted
One zone is eligible for the group but its selected subnet has no free addresses.
- us-east-1a
The first zone has room in its subnet. Free IP addresses are one launch prerequisite.
- us-east-1b
The second zone’s subnet has no free addresses. Even an available EC2 type cannot launch there without networking.
- us-east-1c
The third zone also has addresses available. The placement strategy determines whether it can be used instead.
- Retry 1c
Balanced best effort can try another eligible zone. The successful launch changes distribution; later rebalancing can restore it.
Balanced only
Strict distribution leaves demand unmet when the required zone cannot launch.
- Zone 1a
Two units already run in the first zone. Adding there would not satisfy the intended placement balance.
- Zone 1b
The second zone also has two units. The extra unit is needed in the constrained zone.
- Zone 1c
The needed zone cannot accept the launch. Balanced-only does not use another zone merely to fill the requested count.
- Capacity gap
One unit remains unfulfilled. Changing demand thresholds would not repair this placement restriction.
Use more than one kind of computer
Instance types offer different CPU and memory sizes. A mixed-instance group can choose among types and purchase options. A weight says how many configured capacity units one instance contributes; it does not guarantee equivalent application performance. On-Demand is ordinary purchased capacity; Spot uses spare capacity that AWS can reclaim.
Only weight-4 types are available for a desired capacity of 10 units.
- 01Requirements3 matching types
- 02AvailableOnly c7i.2xlarge
- 033 instances4 units each
- 04Delivered 122-unit overshoot
Direct sequence separate clock or conceptual relationship
The requirement filter finds three compatible types. Eligibility is a configuration question before it is a supply question.
Step 1 of 4
Inspect the evidence and effective setting
EvidenceCandidate filtering happens before allocation preference.
Setting in playOverrides[].WeightedCapacity = "4"
Read every step and alternate example
Weighted overshoot
Only weight-4 types are available for a desired capacity of 10 units.
- Requirements
The requirement filter finds three compatible types. Eligibility is a configuration question before it is a supply question.
- Available
Only the type with weight four is available in this fixture. Other eligible types need not be launchable now.
- 3 instances
Two such instances supply eight units, below desired ten, so a third is needed.
- Delivered 12
Three instances supply twelve configured units. Whole instances explain the surplus; twelve units are not twelve machines.
Purchase split
An unweighted fleet has desired 10, On-Demand base 2, then 25% On-Demand above base.
- Base
The first two units of the requested ten are assigned to On-Demand by the base setting.
- Above base
Eight units remain above that base. Apply the percentage to these eight rather than to all ten.
- 25%
Twenty-five percent of eight is two more On-Demand units. Together with the base, that makes four.
- Remainder
The remaining six are assigned to Spot. This arithmetic describes the requested purchase mix before actual supply constraints.
Replace interruptible capacity early
Spot instances can be reclaimed. A rebalance recommendation warns that an instance is at elevated interruption risk. Capacity Rebalancing lets ASG try a replacement early, but the warning does not guarantee enough time or available supply.
A rebalance recommendation starts a proactive replacement.
- 0100:00Recommendation
- 0200:08Replacement launch
- 0301:02Target ready
- 0401:10Drain old target
- 0502:00Forced deadline
Direct sequence separate clock or conceptual relationship
A recommendation warns of elevated interruption risk. This fixture places the interruption notice at the same starting time; recommendations do not universally provide this lead time.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceThe forced deadline does not pause for hooks or protection.
Setting in playCapacityRebalance = true
Read every step and alternate example
Replacement ready
A rebalance recommendation starts a proactive replacement.
- 00:00
A recommendation warns of elevated interruption risk. This fixture places the interruption notice at the same starting time; recommendations do not universally provide this lead time.
- 00:08
ASG tries to create replacement capacity while the old instance still exists.
- 01:02
The replacement passes target checks in time in this example. Serving work can move to it.
- 01:10
The old target starts draining. Cleanup has its own timing and does not control AWS’s reclamation.
- 02:00
The forced interruption arrives on its own schedule. The earlier replacement is what preserved serving capacity.
No capacity
The replacement attempt fails and the interruption still arrives.
- 00:00
The risk notification starts a proactive attempt. It is not a promise that spare capacity is available.
- 00:06
The replacement launch fails because eligible Spot supply is unavailable.
- 00:40
This example has no applicable On-Demand fallback to satisfy the missing replacement. Rebalancing alone does not create one.
- 02:00
AWS reclaims the instance despite the failed replacement. Hooks cannot postpone this deadline.
Use capacity reserved for the workload
A reservation holds matching EC2 capacity; a Capacity Block supplies capacity for a specified time window. Distribution segments express an order of capacity sources. A reservation must both have room and match the instance you want to launch.
An empty preferred reservation segment falls through to On-Demand.
- 01Segment 1Capacity Block expired
- 02Segment 2Targeted reservation full
- 03On-DemandFallback eligible
- 04LaunchFuture instance only
Direct sequence separate clock or conceptual relationship
The first preferred source was time-bounded. Its expired capacity is no longer usable for this launch.
Step 1 of 4
Inspect the evidence and effective setting
EvidenceAvailability and reservation eligibility are shown separately.
Setting in playCapacityReservationPreference = open
Read every step and alternate example
Fallback allowed
An empty preferred reservation segment falls through to On-Demand.
- Segment 1
The first preferred source was time-bounded. Its expired capacity is no longer usable for this launch.
- Segment 2
The next reservation has no remaining matching room. Naming a reservation does not make it unlimited.
- On-Demand
This configuration allows a later On-Demand source. The ordered search can continue rather than stop at the empty reservation.
- Launch
A new launch can use that source. Existing instances do not move just because future launch preferences changed.
No fallback
The ordered segments contain no eligible available capacity.
- Segment 1
The preferred source has no available room. There is nothing to consume there.
- Segment 2
Another reservation has capacity but does not match the requested type. Availability and eligibility are distinct.
- Fallback
The configuration does not allow a further source. ASG cannot silently ignore that restriction.
- Unmet demand
Two units remain requested but unavailable. Supply or allowed sources must change to fulfill the request.
Change the fleet safely
Replace, refresh, protect, park, detach, or rebalance instances while the application continues to need capacity.
You can explain the constraint that lets a planned change proceed or makes it wait.Roll out a new application image
A launch template is the configuration used to create an instance, including its machine image. Changing the template does not update existing machines. Instance refresh replaces the old fleet in batches; minimum healthy capacity and maximum overlap constrain each batch.
Desired 6, a 100% floor, and 150% ceiling permit three temporary replacements.
- 01Old cohort6 serving
- 02New cohort3 warming
- 03Healthy floor6 required
- 04Overlap ceiling9 allowed
- 05Next actionWait for warmup
Direct sequence separate clock or conceptual relationship
Six old application copies still serve traffic. They provide availability while replacements start.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceBatch selection and wait reason are recorded separately.
Setting in playPreferences.MinHealthyPercentage = 100; MaxHealthyPercentage = 150
Read every step and alternate example
Safe rolling batch
Desired 6, a 100% floor, and 150% ceiling permit three temporary replacements.
- Old cohort
Six old application copies still serve traffic. They provide availability while replacements start.
- New cohort
Three new instances exist but are warming. Presence alone does not make them ready to replace the old cohort.
- Healthy floor
The hundred-percent minimum requires six healthy units for the planned replacement calculation.
- Overlap ceiling
A hundred-fifty-percent maximum allows nine total units against desired six. That supplies room for three temporary replacements.
- Next action
Refresh waits for the replacements’ required readiness and warmup before retiring the next old instances.
One instance: conflicting bounds
A single-instance group requires 100% healthy but allows no overlap.
- Desired
The group wants one instance. Percentage rules become whole-machine constraints at this size.
- Healthy floor
A hundred-percent healthy minimum asks the operation to preserve the one healthy machine.
- Ceiling
An explicit hundred-percent maximum permits no overlap. These bounds cannot both be maintained while replacing one machine.
- Terminate first
Terminating first sacrifices the requested healthy minimum. This is one possible tradeoff to inspect.
- Launch first
With explicit maximum 100%, AWS launches a new instance first when both bounds cannot be honored. It temporarily exceeds desired capacity; omitting maximum instead gives different replacement ordering.
Update the startup disk
An AMI is a machine image used to initialize a computer. The root volume is its startup disk. A root-volume refresh replaces that disk while preserving supported instance resources, but the application must pause and restart. Check compatibility before choosing this strategy.
The instance identity and attached non-root resources persist across the reboot.
- 01i-0abcIdentity + ENI persist
- 02DeregisterTraffic pauses
- 03Root volumeOld data discarded
- 04Reboot + hookReadiness work
- 05InServiceSame instance ID
Direct sequence separate clock or conceptual relationship
The instance ID identifies the machine; its network interface is its connection to the subnet. Supported resources persist through this strategy.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceOld and new root volume IDs are both recorded.
Setting in playStrategy = ReplaceRootVolume
Read every step and alternate example
Compatible replacement
The instance identity and attached non-root resources persist across the reboot.
- i-0abc
The instance ID identifies the machine; its network interface is its connection to the subnet. Supported resources persist through this strategy.
- Deregister
The application is taken out of traffic before the disk work. Identity preservation does not mean uninterrupted service.
- Root volume
The startup disk is replaced. Data on the old root volume is not automatically carried into the new image.
- Reboot + hook
The instance reboots and performs the required preparation. Hooks can coordinate that work before service resumes.
- InService
The same instance ID returns to InService. Application checks still establish whether requests can succeed.
Incompatible AMI
A failed root-volume replacement can branch to full instance replacement.
- Template check
A numeric template version pins the configuration being evaluated, instead of a moving default or latest version.
- AMI
This selected AMI describes multiple volumes and does not meet this strategy’s prerequisites.
- Root replace
Validation rejects this configuration. No root-volume replacement has succeeded.
- Choose a supported strategy
Choose a compatible image or use ordinary rolling replacement. Runtime replacement failure can also lead to full instance replacement; validation rejection does not itself guarantee automatic fallback.
Choose which computer to remove
Reducing desired capacity starts scale-in selection. Scale-in protection excludes an instance from ordinary scale-in, for example while it finishes a job. The termination policy chooses among eligible candidates; it does not override every earlier exclusion.
Placement needs narrow the candidates before the configured policy breaks the tie.
- 016 activeInitial candidates
- 023 in zone 1bImbalanced zone
- 032 unprotectedScale-in eligible
- 04NewestInstancei-0f7 selected
- 05Drain + hookTermination path
Direct sequence separate clock or conceptual relationship
Start with six active instances. This is the initial pool, not a promise that each can be terminated.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceEvery exclusion reason stays inspectable.
Setting in playTerminationPolicies[] = [NewestInstance]
Read every step and alternate example
Candidate funnel
Placement needs narrow the candidates before the configured policy breaks the tie.
- 6 active
Start with six active instances. This is the initial pool, not a promise that each can be terminated.
- 3 in zone 1b
The overrepresented zone narrows the ordinary scale-in candidates according to placement needs.
- 2 unprotected
Protection excludes a candidate from ordinary scale-in. Two eligible machines remain in this example.
- NewestInstance
NewestInstance selects among that narrowed set. An older machine elsewhere can survive without the policy being ignored.
- Drain + hook
The chosen target drains and completes the applicable termination work. Selection and actual removal are separate steps.
All protected
Desired capacity falls, but ordinary scale-in has no eligible instance.
- Desired 4
The requested size is reduced from six to four. Updating the target can succeed before any machine is removed.
- Present 6
All six existing instances have scale-in protection. The old count therefore remains present.
- Candidates 0
No instance is eligible for this ordinary scale-in action. Choosing a different age policy cannot overcome that exclusion.
- Serving 6
All six continue serving. Releasing appropriate protection allows the group to work toward four.
Take an instance out for maintenance
Standby keeps an instance associated with the group while it leaves normal service. Detach removes it from group management. Both can leave EC2 running; your choice about reducing desired capacity determines whether ASG needs a replacement.
The same instance remains a group member but leaves ordinary service.
- 01InServiceDesired 6 · present 6
- 02EnterStandbyDecrement desired: no
- 03DeregisterTarget drains
- 04StandbyOutside active capacity
- 05ReplacementGroup launches one
Direct sequence separate clock or conceptual relationship
The group has six active instances. One is selected for maintenance.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceDirect target deregistration alone would not create Standby state.
Setting in playShouldDecrementDesiredCapacity = false
Read every step and alternate example
Enter standby
The same instance remains a group member but leaves ordinary service.
- InService
The group has six active instances. One is selected for maintenance.
- EnterStandby
EnterStandby keeps membership but requests removal from normal service. Here desired capacity is not decremented.
- Deregister
ASG initiates load-balancer deregistration so existing traffic can drain.
- Standby
The same EC2 instance reaches Standby. It remains associated with the group but is outside its active serving fleet.
- Replacement
Because desired still asks for six active instances, the group launches replacement capacity, subject to normal constraints.
Detach and decrement
The instance leaves ASG management while EC2 continues running.
- InService
This instance begins as an active group member under ASG management.
- Detach
Detach removes management, and the decrement option reduces desired with it, provided bounds allow the operation.
- Detached EC2
The EC2 machine continues running independently. Detaching is not terminating it.
- Desired −1
The reduced target avoids asking ASG to replace the detached member. The independently running resource still needs an owner.
Pause an automatic action
ASG has separate processes for launching, terminating, health replacement, and other work. Suspending one blocks that process. A zonal shift addresses an impaired Availability Zone and has separate placement, health, and traffic-routing effects.
The desired request survives while the process that fulfills it is suspended.
- 01Desired +2Request accepted
- 02LaunchSuspended
- 03AttemptsNone started
- 04ResumeReconciliation may proceed
Direct sequence separate clock or conceptual relationship
The group accepts a request for two more instances. Its desired target changes.
Step 1 of 4
Inspect the evidence and effective setting
EvidenceSuspended process and outstanding capacity gap are shown together.
Setting in playSuspendProcesses.ScalingProcesses[] = Launch
Read every step and alternate example
Launch suspended
The desired request survives while the process that fulfills it is suspended.
- Desired +2
The group accepts a request for two more instances. Its desired target changes.
- Launch
The Launch process is suspended, so ASG cannot perform the launch work required by that request.
- Attempts
There are no new attempts to count. Investigate the suspended process before hunting for an EC2 launch error.
- Resume
Resuming Launch permits reconciliation with the outstanding target. Other processes have their own resume behavior.
Active zonal shift
Placement avoids the impaired zone while traffic routing remains a separate surface.
- Zone 1a
A shift is active away from an impaired zone. Enabling the integration alone would not start this shift.
- Health behavior
The configured IgnoreUnhealthy behavior changes how affected unhealthy instances are handled during the shift.
- Zones 1b/1c
New placement uses the remaining zones, whose capacity and subnet constraints still apply.
- Headroom
Only one additional unit is available there in this fixture. Avoiding an impaired zone does not create surviving-zone headroom.
Read the incident
Combine the same mechanisms when launches fail or several changes happen at once.
You can locate the first stalled stage and choose evidence that belongs to it.Find where a launch stopped
A capacity request passes through configuration, permissions, EC2 launch, application setup, and readiness. Activity history records ASG actions; application logs explain work inside the machine. Match the failed stage to its evidence before changing the scaling policy.
The group and template validate, but encrypted-volume creation is denied.
- 01ASG requestAccepted
- 02TemplateVersion 12 resolved
- 03AuthorizationKMS grant denied
- 04EC2 instanceNot created
- 05Capacity gapRequest remains
Direct sequence separate clock or conceptual relationship
ASG accepts the desired-capacity request. That validates neither the entire launch path nor application readiness.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceActivity history names the denied actor and action.
Setting in playLaunch template EBS KmsKeyId + ServiceLinkedRoleARN
Read every step and alternate example
KMS denial
The group and template validate, but encrypted-volume creation is denied.
- ASG request
ASG accepts the desired-capacity request. That validates neither the entire launch path nor application readiness.
- Template
The launch resolves a specific template version. Check that version when investigating encrypted disk settings.
- Authorization
KMS, the key-management service, denies the access needed for encrypted storage. Inspect the launch service identity and key policy.
- EC2 instance
This fixture has no usable created instance. Real failures can also create an ID briefly before termination; activity and EC2 evidence distinguish them.
- Capacity gap
The original request remains unmet. Changing the scaling target will not grant the missing encryption permission.
Bootstrap egress
EC2 launches, but the application never becomes ready.
- ASG request
The requested capacity is accepted, so the investigation proceeds into the launch path.
- EC2 boot
EC2 reaches running. Placement and basic boot have succeeded far enough to inspect the application.
- User data
The startup script, supplied as user data, cannot download an artifact. Its outbound network path or destination must be checked.
- Target health
The application never answers the configured target health check. A running VM cannot serve this workload yet.
- Replacement
Replacement repeats the same broken setup. Fixing the shared launch configuration prevents the next instance from failing identically.
Put the mechanisms together
Real incidents overlap. A refresh may be running when traffic rises, or a warm pool may hide a broken image until it empties. Follow the requested capacity, usable application capacity, and the reason each instance is changing.
Traffic rises while three old instances are already leaving the fleet.
- 01Needed serving10
- 02Serving6
- 03New cohort3 warming
- 04Scale-out+2 requested
- 05Below need4m 20s fixture
Direct sequence separate clock or conceptual relationship
The assumed workload now needs ten usable application copies. That is a demand estimate, separate from ASG’s current desired target.
Step 1 of 5
Inspect the evidence and effective setting
EvidenceThe replay attributes each instance to refresh, scale-out, or replacement.
Setting in playCompare: MaxHealthyPercentage 100 → 150
Read every step and alternate example
Spike during refresh
Traffic rises while three old instances are already leaving the fleet.
- Needed serving
The assumed workload now needs ten usable application copies. That is a demand estimate, separate from ASG’s current desired target.
- Serving
Only six copies can handle requests. The immediate service shortfall is four under this workload model.
- New cohort
Three replacements are warming. Counting them as already usable would hide part of the incident.
- Scale-out
A scaling action requests two more. Those launches compete for the same supply and still need startup time.
- Below need
The authored incident assigns four minutes twenty seconds below need. It illustrates exposure during overlap, not a computed AWS latency prediction.
Protected workers
A queue grows while every busy worker is protected from scale in.
- Backlog
The queue has 1,840 waiting jobs and is growing. Work arrives faster than usable workers finish it.
- Workers
Six workers are occupied. Check throughput and downstream constraints as well as instance count.
- Protection
All six are protected from ordinary scale-in while busy. Protection itself does not block adding workers.
- Scale-out
Maximum size prevents further growth in this configuration. That explains the backlog’s scaling constraint.
- Completion
After jobs finish, the application must release protection when safe. Otherwise a later scale-in request cannot remove these workers.
Warm pool + bad AMI
The pool empties, then cold launches repeat a broken bootstrap.
- Pool
Two instances were prepared using a working configuration. Their presence can make the service look ready for growth.
- Demand
Demand asks for five more instances, exceeding the prepared population.
- Warm starts
The prepared pair becomes useful after restart and checks, consuming the available pool.
- Cold launches
Three cold launches use the bad image. An AMI is the machine image, so a broken shared image can repeat the same setup failure.
- Serving gap
The three-instance gap persists. Compare image and template versions between the working pool members and the failing cold launches.
Keep the capacity counts separate
Once you have watched a replacement start, the counts have something concrete to refer to.
| Count | Meaning | Example while a replacement starts |
|---|---|---|
| Desired | How many instances the group is asked to maintain | 6 requested |
| Present | How many instances actually exist | 5 existing + 1 starting = 6 |
| InService | How many instances ASG has admitted to active membership | A starting replacement may not be admitted yet |
| Serving | How many application copies can handle requests in this example | 5 ready; the replacement is still preparing |
| Metric eligible | Which instances contribute to the selected scaling calculation | A new instance may still be warming even after it begins serving |
Serving is a teaching measure of application usefulness. An ASG instance count does not itself establish that a load balancer can send requests successfully.
The timers have different owners
Auto Scaling timing settings sit close together in the console, but they do not pause the same action. An ELB target keeps receiving health checks during an ASG health-check grace period. Grace changes when the group acts on a startup health failure. It does not suppress the raw checks, and leaving EC2 running can still lead to immediate replacement.
Warmup controls eligibility for scaling calculations and refresh progress. It does not erase CloudWatch samples or decide when a target begins receiving traffic. A simple-scaling cooldown gates another simple-scaling action; it does not lock health replacement, schedules, manual changes, target tracking, or step scaling.
| Clock | Starts when | Gates |
|---|---|---|
| Lifecycle heartbeat | The instance enters the hook wait state | Timeout of the current lifecycle action; a heartbeat restarts this timer |
| Lifecycle absolute limit | The lifecycle action begins | Total hook wait, limited to the smaller of 48 hours or 100 heartbeat periods |
| Health-check grace | A startup, attach, or standby-return case enters its applicable path | When ASG acts on covered startup health failures |
| Instance warmup | The policy or refresh-specific trigger occurs | Scaling calculations or refresh readiness for that consumer |
| Simple cooldown | The simple scaling activity starts | Another action from that simple policy |
| Deregistration delay | The target begins deregistering | How long in-flight connections may drain |
| Refresh checkpoint delay | The refresh reaches a configured checkpoint | When the next replacement batch may begin |
| Refresh bake time | Replacement work has otherwise finished | Final observation before refresh success |
A termination hook and target deregistration delay are separate clocks. In the ordinary flow, draining begins before the termination hook. Neither clock extends a forced Spot interruption deadline.
Read the configuration by owner
The same word can name several settings. MinSize on the group and MinSize on a warm pool are different parameters. Refresh warmup can override group warmup, while a policy can have another fallback chain. Inspect the exact path, owner, units, explicit or omitted state, effective source, and change timing before comparing values.
| Family | Representative settings and operations | Labs |
|---|---|---|
| Group and placement | MinSize, DesiredCapacity, MaxSize, VPCZoneIdentifier, AvailabilityZoneDistribution | V01, V10, V13 |
| Health and integrations | HealthCheckType, HealthCheckGracePeriod, target groups, traffic sources | V03, V04, V19 |
| Launch and mixed instances | LaunchTemplate, overrides, InstanceRequirements, weights, InstancesDistribution | V11, V12, V19 |
| Scaling policies | TargetTrackingConfiguration, StepAdjustments, Cooldown, metric specifications | V05–V07 |
| Schedules and forecasts | PutScheduledUpdateGroupAction, PredictiveScalingConfiguration | V08 |
| Hooks and warm pools | HeartbeatTimeout, DefaultResult, CompleteLifecycleAction, PutWarmPool | V02, V03, V09 |
| Maintenance and refresh | InstanceMaintenancePolicy, RefreshPreferences, checkpoints, alarms, rollback | V14, V15 |
| Instance operations | Standby, attach, detach, explicit launch, protection, termination | V16, V17 |
| Control and evidence | SuspendProcesses, metrics collection, activities, notifications, tags | V18–V20 |
Changing a launch template reference affects future launches. It does not repaint existing instances as updated. A rolling refresh, a root-volume replacement, and a health replacement follow different eligibility and lifecycle rules even when they converge on the same desired template.
Choose settings from the workload
Start with what makes capacity useful. A web service may care about registered targets, request latency, and safe connection draining. A queue worker may care about visible backlog per usable worker, message visibility timeout, long-job protection, and acknowledgment. A GPU workload may be constrained by a reservation or an instance type before any scaling policy gets a chance to help.
The queue fixture uses two seconds of average service time and 60 seconds of acceptable queue delay, which gives a starting target of 30 messages per worker. That relationship assumes comparable workers and service times; it is not a latency guarantee. At zero workers, the ratio is undefined and CPU samples may be absent, so an independent queue-depth signal must wake the fleet.
| Workload question | Configuration surface to inspect | Failure to replay |
|---|---|---|
| How early must capacity be usable? | Warm pools, schedules, predictive scaling, startup hooks | Traffic arrives while instances are present but not serving |
| What signal changes with capacity? | Target tracking metric and missing-data behavior | Average looks healthy while load is skewed |
| Can work stop safely? | Scale-in protection, deregistration, termination hook, queue visibility | A long job is interrupted before acknowledgment |
| Which supply is actually available? | Zones, subnets, instance requirements, Spot and reservations | Desired rises while every launch destination fails |
| How much rollout overlap is affordable? | Maintenance minimum and maximum healthy percentages | A refresh cannot satisfy both its floor and ceiling |
Diagnose the first missing condition
Begin with the requested-versus-serving gap, then find the stage where progress stopped. Activity history can show that a launch request failed; EC2 state can show that an instance exists; target health can show why it is not routed; refresh status can show which readiness gate is still waiting. Those observations describe different moments in the path.
| Observation | Check next | Why |
|---|---|---|
| Desired exceeds present capacity | Scaling activities, placement, subnet addresses, quotas, launch authorization | The request exists, but EC2 capacity has not been created |
| EC2 is running but the target is unhealthy | Bootstrap logs, listener port, target path, security rules, application readiness | Launch succeeded; service readiness did not |
| Target is healthy but refresh is waiting | Refresh warmup, minimum healthy floor, checkpoints, alarms, protected instances | Routing health is only one refresh gate |
| Desired fell but the fleet did not shrink | Scale-in protection, standby, suspended Terminate, retention | The capacity request and termination eligibility diverged |
| Warm starts succeed, then replacements loop | Pool depletion and the template used by cold launches | Prepared instances can hide a broken current launch path |
| A Spot instance disappears during cleanup | Recommendation time, replacement readiness, drain and hook clocks | Protection and hooks cannot freeze forced reclamation |
Repeated launch failures can lead to administrative process suspension. Treat that as evidence about the group’s recent failures, not as proof that the scaling policy is wrong. Keep observation timestamps visible: a missing metric sample is not zero, a pending instance is not a successful launch, and a healthy replacement does not automatically mean a rollout has passed bake time.
Operations that look similar are not interchangeable
EnterStandby changes ASG membership state and begins deregistration. Directly deregistering a target does not put the instance in standby. Detaching removes ASG management while the EC2 instance continues; entering standby preserves group membership. Whether either operation decrements desired capacity determines whether the group asks for replacement capacity.
Scale-in protection applies to ordinary scale-in selection. EC2 termination protection, group deletion protection, refresh treatment of protected instances, and a forced Spot interruption are separate safeguards. When every active instance is protected, desired capacity can fall while actual capacity remains above it.
The nine suspendable processes also have narrow effects: Launch, Terminate, AddToLoadBalancer, AlarmNotification, AZRebalance, HealthCheck, InstanceRefresh, ReplaceUnhealthy, and ScheduledActions. Resuming a process does not universally replay everything missed while it was suspended. In particular, past scheduled actions are not replayed, and instances missed while AddToLoadBalancer was suspended need repair.
Sources and documentation
- EC2 Auto Scaling API actions
- Auto Scaling group lifecycle
- Lifecycle hooks
- Health checks for Auto Scaling instances
- Dynamic scaling
- Scheduled scaling
- Predictive scaling
- Warm pools
- Mixed instances groups
- Capacity Rebalancing
- Instance refresh
- Termination policies
- Suspended processes
- Auto Scaling group metrics
- Health-check grace period and its starting point
- Refresh preferences and fallback ordering
- Root-volume replacement prerequisites
- Warm-pool lifecycle and hooks