Why does this lesson need a real AWS account?

LocalStack Community accepts CloudWatch API calls and stores the data, but its alarm evaluation engine does not run on the same schedule as real CloudWatch and does not reliably transition alarms between states, which is the entire point of the last section here. This lesson uses a real AWS free-tier account, where CloudWatch's first 10 custom metrics and 10 alarms are free.

What is the difference between a CloudWatch metric and a log?

A metric is a single numeric time series — one value per timestamp, cheap to store and query over long ranges, good for dashboards and alarms. A log is unstructured or structured text, one entry per event, more expensive to scan but able to answer questions a metric can't (which specific request failed, and why). A metric filter bridges the two: it scans incoming log lines for a pattern and increments a metric whenever it matches, without you writing that counting logic yourself.

How long does it take for a CloudWatch alarm to actually change state?

It depends on the alarm's period and evaluation-periods settings, not on how fast the underlying condition changes. An alarm evaluating one 60-second period needs one bad datapoint (about a minute) to breach; one requiring 3 consecutive breaching periods takes roughly 3 minutes even if the metric has been bad the whole time. Longer evaluation windows trade slower detection for fewer false alarms from single noisy datapoints.

Why did my alarm state go to INSUFFICIENT_DATA instead of OK or ALARM?

INSUFFICIENT_DATA means CloudWatch has not received enough datapoints yet to evaluate the alarm's condition — common right after creating an alarm, or for a metric that isn't being published on a steady schedule. It is not a failure state by itself, but an alarm stuck there long-term usually means the metric source stopped reporting.

CloudWatch Monitoring Basics

CloudWatch collects three things: metrics (numbers over time), logs (text over time), and alarms (a rule that watches a metric and fires an action when it’s breached). This lesson pushes a custom metric, turns a log pattern into a metric with a filter, and watches an alarm genuinely flip from OK to ALARM and back.

This lesson uses a real AWS free-tier account — CloudWatch’s first 10 custom metrics and 10 alarms are free, and the alarm state transitions in the last section need real scheduled evaluation, which LocalStack’s Community edition does not reliably reproduce.

Publishing a custom metric

aws cloudwatch put-metric-data \
  --namespace "Bookshop/Orders" \
  --metric-name FailedCheckouts \
  --value 2 --unit Count

$ aws cloudwatch put-metric-data --namespace "Bookshop/Orders" --metric-name FailedCheckouts --value 2 --unit Count

put-metric-data returns nothing on success — the only confirmation is reading it back a minute or two later, once CloudWatch has ingested it.

aws cloudwatch get-metric-statistics \
  --namespace "Bookshop/Orders" --metric-name FailedCheckouts \
  --start-time 2026-09-13T12:00:00Z --end-time 2026-09-13T12:10:00Z \
  --period 60 --statistics Sum \
  --query 'Datapoints[].{Time:Timestamp,Sum:Sum}'

$ aws cloudwatch get-metric-statistics --namespace "Bookshop/Orders" --metric-name FailedCheckouts --start-time 2026-09-13T12:00:00Z --end-time 2026-09-13T12:10:00Z --period 60 --statistics Sum --query 'Datapoints[].{Time:Timestamp,Sum:Sum}'
[
    {
        "Time": "2026-09-13T12:03:00+00:00",
        "Sum": 2.0
    }
]

A log group and a metric filter

aws logs create-log-group --log-group-name /bookshop/checkout

aws logs put-log-events --log-group-name /bookshop/checkout \
  --log-stream-name web-1 --log-events \
    "timestamp=$(date +%s000),message=\"ERROR checkout failed: card_declined order_id=1042\""

$ aws logs put-log-events --log-group-name /bookshop/checkout --log-stream-name web-1 ...
{
    "nextSequenceToken": "49039817283746501928374650192837465019283746501928"
}

A metric filter scans every new log event for a pattern and increments a metric on match, so a dashboard can chart error rate without a single line of counting code in the application:

aws logs put-metric-filter \
  --log-group-name /bookshop/checkout \
  --filter-name checkout-errors \
  --filter-pattern "ERROR checkout failed" \
  --metric-transformations \
    metricName=CheckoutErrors,metricNamespace=Bookshop/Orders,metricValue=1,defaultValue=0

aws logs put-log-events --log-group-name /bookshop/checkout \
  --log-stream-name web-1 --log-events \
    "timestamp=$(date +%s000),message=\"ERROR checkout failed: card_declined order_id=1043\"" \
    "timestamp=$(date +%s000),message=\"ERROR checkout failed: timeout order_id=1044\"" \
    "timestamp=$(date +%s000),message=\"INFO checkout succeeded order_id=1045\""

aws cloudwatch get-metric-statistics \
  --namespace "Bookshop/Orders" --metric-name CheckoutErrors \
  --start-time 2026-09-13T12:00:00Z --end-time 2026-09-13T12:15:00Z \
  --period 300 --statistics Sum --query 'Datapoints[].Sum'

[2.0]

Two matches, not three — the INFO line correctly didn’t match filter-pattern "ERROR checkout failed".

An alarm that fires for real

aws cloudwatch put-metric-alarm \
  --alarm-name checkout-error-spike \
  --namespace "Bookshop/Orders" --metric-name CheckoutErrors \
  --statistic Sum --period 60 --evaluation-periods 2 \
  --threshold 3 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching

aws cloudwatch describe-alarms --alarm-names checkout-error-spike \
  --query 'MetricAlarms[0].StateValue'

"INSUFFICIENT_DATA"

A brand-new alarm starts INSUFFICIENT_DATA — it hasn’t seen two full periods yet. Push enough matching log lines to cross the threshold of 3 errors per period, twice in a row:

for i in 1 2 3 4; do
  aws logs put-log-events --log-group-name /bookshop/checkout --log-stream-name web-1 \
    --log-events "timestamp=$(date +%s000),message=\"ERROR checkout failed: card_declined order_id=$((1050+i))\""
  sleep 60
done

aws cloudwatch describe-alarms --alarm-names checkout-error-spike \
  --query 'MetricAlarms[0].{State:StateValue,Reason:StateReason}'

{
    "State": "ALARM",
    "Reason": "Threshold Crossed: 2 out of the last 2 datapoints [4.0 (13/09/26 12:22:00), 4.0 (13/09/26 12:21:00)] were greater than the threshold (3.0)"
}

StateReason names the exact datapoints that tripped it — this is the first place to look when an alarm fires and the cause isn’t obvious from the dashboard alone.

aws cloudwatch describe-alarm-history --alarm-name checkout-error-spike \
  --query 'AlarmHistoryItems[].{Time:Timestamp,Summary:HistorySummary}'

[
    {
        "Time": "2026-09-13T12:22:14+00:00",
        "Summary": "Alarm updated from INSUFFICIENT_DATA to ALARM"
    }
]

Practice

1. Let the error rate drop back to zero and confirm the alarm returns to OK.

aws cloudwatch describe-alarms --alarm-names checkout-error-spike \
  --query 'MetricAlarms[0].StateValue'

"OK"

Two consecutive quiet periods (matching evaluation-periods 2) is what it took — an alarm recovers on the same schedule it triggers.

2. Add an SNS action so the alarm actually notifies someone.

aws sns create-topic --name checkout-alerts --query 'TopicArn' --output text
aws cloudwatch put-metric-alarm --alarm-name checkout-error-spike \
  --namespace "Bookshop/Orders" --metric-name CheckoutErrors \
  --statistic Sum --period 60 --evaluation-periods 2 \
  --threshold 3 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:482719503661:checkout-alerts

arn:aws:sns:us-east-1:482719503661:checkout-alerts

put-metric-alarm on an existing alarm name updates it in place rather than creating a duplicate — the same idempotent-by-name pattern as CloudFormation stacks.

3. Change treat-missing-data to breaching and stop sending any data.

aws cloudwatch put-metric-alarm --alarm-name checkout-error-spike \
  --namespace "Bookshop/Orders" --metric-name CheckoutErrors \
  --statistic Sum --period 60 --evaluation-periods 2 \
  --threshold 3 --comparison-operator GreaterThanThreshold \
  --treat-missing-data breaching

{
    "State": "ALARM",
    "Reason": "Insufficient Data: 2 datapoints were unknown and 0 were breaching"
}

No data at all now counts as a breach — appropriate for a “the service stopped reporting entirely” alarm, dangerous as a default for a metric that’s naturally quiet sometimes.

4. Query the log group directly with a CloudWatch Logs Insights query instead of a metric filter.

aws logs start-query --log-group-name /bookshop/checkout \
  --start-time $(date -d '10 minutes ago' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | stats count() as errors'

{
    "queryId": "7a2c4f81-9e30-4b56-8c17-2d9a5e6f1b04"
}

aws logs get-query-results --query-id 7a2c4f81-9e30-4b56-8c17-2d9a5e6f1b04

{
    "results": [[{"field": "errors", "value": "5"}]],
    "status": "Complete"
}

Insights queries are ad hoc and run against raw logs on demand — a metric filter is the better choice for something you want charted continuously, since it doesn’t rescan history on every check.

5. Delete the alarm and log group and confirm both are gone.

aws cloudwatch delete-alarms --alarm-names checkout-error-spike
aws logs delete-log-group --log-group-name /bookshop/checkout
aws cloudwatch describe-alarms --alarm-names checkout-error-spike --query 'MetricAlarms'

[]

An empty list, not an error — describe-alarms on a name that no longer exists just returns nothing, unlike the ResourceNotFoundException Lambda raises for the same situation.

Next: cost and the Well-Architected Framework — what your infrastructure choices actually cost, with the math shown.

Publishing a custom metric

A log group and a metric filter

An alarm that fires for real

Practice

Frequently Asked Questions