Observability and Debugging

Nobody could tell if it was up

• October 1, 2013: HealthCare.gov launches and breaks under its first traffic
• Mid-October: about 3 in 10 visitors can get on at all
• The rescue team finds no dashboard, "no quick way for engineers to measure what was going on"
• "Nobody could even tell us if the system was up."

The first fix was a dashboard

• The rescue team built the dashboard first
• Then they could rank the problems and fix the worst ones first
• By December 1: uptime from 43% to 95%, errors from 6% to 0.5%

Nine months without a trace

• January 2026: I upgrade Project Pulse from Spring Boot 3 to 4
• Spring Boot 4 sets up tracing only through its own modules
• Every test passes, and tracing is off
• No trace IDs, nothing in Zipkin, for nine months

That, then why

Three kinds of telemetry

• Metric: a number measured over and over. Tells you that something is wrong
• Trace: one request's path, step by step. Tells you where
• Log: a line of text about one event. Tells you why

Three of each, from Project Pulse

How do you know?

Actuator: the diagnostic port

Try it on production

curl https://projectpulse.team/actuator/health
curl https://projectpulse.team/actuator/info
curl https://projectpulse.team/actuator/metrics

Five dependencies, no code

spring-boot-starter-actuator            the /actuator endpoints
spring-boot-starter-micrometer-metrics  counts and times
micrometer-registry-prometheus          the Prometheus format
spring-boot-micrometer-tracing-brave    trace IDs and spans
spring-boot-starter-zipkin              sends spans to Zipkin

A dashboard reads a collector

Logs go to standard output

• The app writes to standard output; the platform collects it
• Project Pulse: Logback's console output, no log files
• Azure App Service: the log stream

Written for a machine

09:02:14 ERROR [6af11fb6...-0860298d...] UserInvitationService : Could not send an invitation email for section 2 of course 1

{"log":{"level":"ERROR", "logger":"team.projectpulse.user.userinvitation.UserInvitationService"},
 "message":"Could not send an invitation email for section 2 of course 1", "traceId":"6af11fb6..."}

One ID per request

• A request arrives; Brave gives it a trace ID
• Every log line that request writes carries the ID, in every class
• One search for the ID returns the whole story

Change the level, no redeploy

curl -X POST localhost/actuator/loggers/team.projectpulse.user \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"configuredLevel":"DEBUG"}'

Metrics have tags

/actuator/metrics/http.server.requests
/actuator/metrics/http.server.requests?tag=uri:/api/v1/students&tag=status:500

The four golden signals

Machines and business

Count it yourself

try {
    this.emailService.sendInvitationEmail(userInvitation);
    meterRegistry.counter("invitations.sent", "outcome", "success").increment();
} catch (RuntimeException e) {
    LOGGER.error("Could not send an invitation email for section {} of course {}", sectionId, courseId, e);
    meterRegistry.counter("invitations.sent", "outcome", "failure").increment();
}

What does this number measure?

• Only requests that reached the application
• Starts at zero on every deploy and restart
• Once it becomes a target, people improve the number instead of the thing

Your SLO, measured

95 of every 100 views

management:
  metrics:
    distribution:
      percentiles:
        http.server.requests: 0.95

A probe that cannot fail

curl -s -o /dev/null -w '%{http_code} %{content_type}' https://projectpulse.team/health

200 text/html: the single-page app, not the health check.

Traces and spans

• A trace is one request's journey
• A span is one step: when it started, how long it took, which step called it
• Across five services, only the trace shows which one was slow

A trace shows what someone instrumented

@Observed(name = "email.send", contextualName = "send invitation email")
public void sendInvitationEmail(UserInvitation userInvitation) {

Turn the annotation on

management:
  observations:
    annotations:
      enabled: true

Keep one in ten

management:
  tracing:
    sampling:
      probability: 0.1

September 6, 2026

• Project Pulse's /actuator/env answers any anonymous caller
• It returns the database and mail passwords
• The settings came from a local development tutorial and applied to every profile

The fix, in application.yml

management:
  endpoints:
    web:
      exposure:
        include: health,info        # the base for every profile
  endpoint:
    health:
      show-details: when-authorized # details for an admin only
      roles: admin
  info:
    git:
      mode: simple                  # the commit, not the JVM version
    java:
      enabled: false
    os:
      enabled: false

Is the app down?

• November: staging's health says DOWN
• The cause: a stale mail password in staging's Key Vault
• Mail was broken. Everything else worked

Liveness and readiness

Monday's takeaways

• A metric says something is wrong, a trace says where, a log says why
• Logs to standard output, with a trace ID on every line
• Before you trust a number, say what it misses
• A trace shows only what someone instrumented
• Management endpoints are attack surface

Who is asking?

Reading the numbers

• Dashboard: 583 of 1,979 requests failed. 29%!
• Rows or requests? 1,564 rows stand for 1,979. Sum itemCount
• What failed? Every 4xx counts: 574 were 404s, mostly bots
• What should the SLI count? Our errors, 5xx: 99.95%

Debugging from evidence

• Narrow it down with a metric: when, and which route?
• Find one failing request by its trace ID
• Give the agent the evidence, not the symptom
• Reproduce it with a failing test
• Fix, then watch the metric return to normal

Evidence, not the symptom

Where this work lives

The agent and your telemetry

1. Tour Actuator

• Log in as the seeded admin, copy the token
• /actuator, /actuator/health, /actuator/metrics/http.server.requests
• Raise one logger to DEBUG, no restart

2. Read the dashboard

• Availability, errors, uptime: what does each number count?
• Grafana stores nothing. Where do the numbers come from?
• The scrape signs in as the admin. Without that: 401. Why?
• Two alerts email Mailpit: any 5xx, any ERROR line

3. Stop the mail server

• Add the @Observed line, sampling to 1.0, send an invitation
• Stop the Mailpit container
• Send another. Does Project Pulse degrade, error, or hang?
• The alert fires. Where is its email?

4. Find the 500 on production

• Production's first day: one 500. On which route?
• Join everything with its operation_Id
• Read the cause
• Hand the agent the evidence; ask for a failing test first

One request, joined

let op = toscalar(requests
    | where timestamp > ago(2d) and resultCode == "500"
          and name == "POST /api/v1/evaluations"
    | take 1 | project operation_Id);
union requests, dependencies, exceptions, traces
| where operation_Id == op
| project timestamp, itemType, name, resultCode, message
| order by timestamp asc

Hand it to the agent

5. Turn one metric into an SLO

Your week

• Choose your SLO's source and write what it misses under the requirement
• Make one request fail on purpose in your deployed system
• Find its log line in the platform's log stream, not your terminal