Error Handling, Logging, and Observability in Node.js Backends: Knowing What Your System Is Doing

Part 6 guarded the entrance: rate limits, CORS, CSRF, validation and headers. Every one of those controls turns some requests into rejections, and some requests will fail for reasons nobody planned. This part is about the moment after something goes wrong.

A backend that handles errors badly fails in one of two ways. It crashes in a way you did not intend, or it fails quietly: a request that hangs, an exception swallowed by a catch, a log line that says "error" and nothing else. Both look the same from the outside, a user who is stuck, and both are expensive because you cannot tell which one you have.

The three topics here belong together. Error handling decides what the program does when something fails. Logging records what happened. Observability is the broader ability to ask new questions about a running system without shipping new code. The examples are in Node.js, and I tested the Node and library behavior I describe on Node 24, Express 4 and 5, Fastify and pino. Where a claim comes from a standard or documentation I name it; where it is my recommendation, I say so.

Quick answer

• Separate expected failures from bugs. A failed payment or a missing record is a normal outcome you handle. A TypeError in your own code means the program is in a state you did not plan for.
• Know what Node does by default. A rejected promise with no handler terminates the process; a handler for uncaughtException changes that in a way that is easy to get wrong.
• Check how your framework treats async errors. In Express 4 an async handler that throws leaves the request hanging; Express 5 and Fastify turn it into a normal error response.
• Map every failure through one error handler that logs the details and returns a generic, consistent response.
• Log structured JSON, redact secrets, and attach a request id to every line.
• Watch four things: latency, traffic, errors and saturation. Alert on symptoms users feel, not on every possible cause.
• Add traces when requests cross services. A trace id in your logs ties the three signals together.
• Shut down gracefully. Stop accepting work, finish what is in flight, then exit. There is a keep-alive trap in the middle of that.

failure happens
   |
   +-- expected (validation, not found, conflict)  -> typed error -> 4xx response
   |
   +-- unexpected (bug, dependency down)           -> log with context -> generic 500
   |
   +-- fatal (unhandled rejection / uncaught)      -> log, flush, exit non-zero -> supervisor restarts

every path:  one structured log line  +  a metric  +  (if cross-service) a trace

Two kinds of failure

It helps to name the difference, because the right response is different.

Expected failures come from the world, not from a bug: the database refused a connection, a validation rule failed, the record does not exist, the payment provider timed out, the caller lacked permission (Part 5). You can enumerate them in advance, so you handle them deliberately: return a clear status code, retry if it is safe, or degrade.

Unexpected failures come from a mistake in your code or from a state you did not anticipate: reading a property of undefined, a function that returns the wrong shape, a violated invariant. You cannot meaningfully recover in place, because you do not know what else is wrong.

Node's own documentation draws the line for the second kind in blunt terms. For uncaughtException it says that unhandled exceptions "inherently mean that an application is in an undefined state", and that attempting to resume normally can be like "pulling out the power cord when upgrading a computer": nine times out of ten nothing happens, and the tenth time the system is corrupted. The correct use of that event, per the docs, is synchronous cleanup before shutting down, and it is not safe to resume normal operation afterward.

What Node does when you do not handle an error

I tested the defaults, since they decide what an unhandled mistake costs you.

An unhandled promise rejection terminates the process. The Node docs state that --unhandled-rejections defaults to throw, which raises the rejection as an uncaught exception. In Node 24:

node -e "Promise.reject(new Error('boom')); setTimeout(() => console.log('still alive'), 200)"
echo $?   # 1  -> the process died; 'still alive' never printed

A handler for uncaughtException changes the exit behavior, and not in the direction people expect. The docs say that adding a handler overrides the default of printing the stack and exiting with code 1; unless the handler sets process.exitCode, the process then exits with 0. I tested it:

node -e "process.on('uncaughtException', e => console.log('handled:', e.message));
         setTimeout(() => { throw new Error('x') }, 10);
         setTimeout(() => console.log('continued after throw'), 100)"
# handled: x
# continued after throw     <- the program carried on in an undefined state
echo $?                      # 0  <- and exited as if nothing had gone wrong

So a handler that only logs turns a crash into a silent success: the program keeps running, and a supervisor or CI job sees exit code 0. If you add one, the safe pattern is: log the error, release resources synchronously, set a non-zero exit code, and let the process end so something restarts it.

process.on('uncaughtException', (err) => {
  logger.fatal({ err }, 'uncaught exception, shutting down')
  process.exitCode = 1
  shutdown() // stop accepting work, drain, exit (see the shutdown section)
})
process.on('unhandledRejection', (reason) => {
  logger.fatal({ err: reason }, 'unhandled rejection, shutting down')
  process.exitCode = 1
  shutdown()
})

The design assumption behind this is that something outside the process (a container orchestrator, a process manager, a serverless platform) restarts it. If nothing does, crashing is worse than it sounds, and that is a deployment question for Part 10.

Async errors and your framework

The most common way unhandled rejections appear in a web backend is a request handler that is async and throws. How the framework treats that varies, so I tested three.

app.get('/async-throw', async () => { throw new Error('async boom') })
app.get('/sync-throw',  () => { throw new Error('sync boom') })
// plus an error-handling middleware that returns 500 JSON

The Express 4 row is the dangerous one. The request never gets a response, the client waits until it times out, and on current Node versions the raised rejection terminates the process by default (in my test I trapped the event to observe it). Two ways to avoid it on Express 4: upgrade to Express 5, or wrap async handlers so a rejection is forwarded to next:

const wrap = (fn) => (req, res, next) => Promise.resolve(fn(req, res, next)).catch(next)
app.get('/orders/:id', wrap(async (req, res) => { /* ... */ }))

Whatever framework you use, test this behavior once on purpose. It is a two-line route and it tells you whether a forgotten await takes the server down.

Designing an error model

Handling errors consistently is mostly a matter of having one vocabulary and one place that translates it.

Typed errors

Give expected failures a type that carries what the handler needs: a stable machine-readable code, an HTTP status, and whether the message is safe to show.

class AppError extends Error {
  constructor(
    message: string,
    public readonly status: number,
    public readonly code: string,
    options?: { cause?: unknown; expose?: boolean },
  ) {
    super(message, { cause: options?.cause })
    this.name = 'AppError'
    this.expose = options?.expose ?? status < 500
  }
  readonly expose: boolean
}

class NotFoundError extends AppError {
  constructor(what: string) { super(`${what} not found`, 404, 'not_found') }
}
class ConflictError extends AppError {
  constructor(message: string) { super(message, 409, 'conflict') }
}

Keep the cause

When you catch a low-level error and throw a higher-level one, keep the original. I confirmed that new Error(message, { cause }) preserves it:

try {
  try { JSON.parse('{bad') } catch (e) { throw new Error('could not read config', { cause: e }) }
} catch (e) {
  e.message      // 'could not read config'
  e.cause.name   // 'SyntaxError'
}

Without the cause you log "could not read config" and lose the part that tells you why. Structured loggers can serialize the chain, so it appears in the log line.

One handler at the boundary

Translate at one place, at the edge of your application, rather than in every route:

app.use((err, req, res, _next) => {
  const known = err instanceof AppError
  const status = known ? err.status : 500

  // log once, with context; details stay in the logs, not the response
  // (req.log and req.id assume a request logger such as pino-http)
  req.log[status >= 500 ? 'error' : 'warn']({ err, code: known ? err.code : 'internal' }, 'request failed')

  res.status(status).type('application/problem+json').json({
    type: 'about:blank',
    status,
    title: known && err.expose ? err.message : 'Something went wrong',
    ...(known ? { code: err.code } : {}),
    requestId: req.id,
  })
})

This follows Part 6: generic messages for anything unexpected, the right status code, and RFC 9457 problem details as the response shape. Including a requestId in the response gives a user something to quote to support, and gives you something to search for in the logs.

Log an error once

A common failure mode is an error logged at every layer it passes through: the repository logs it, the service logs it, the route logs it, and one incident becomes four log entries that look like four incidents. A reasonable rule (my recommendation, not a standard): lower layers either handle an error or throw it with its cause; the boundary handler logs it, once.

Time limits and retries

An unbounded wait is an error you have not noticed yet. Put a timeout on every outbound call. Node supports this directly: I confirmed that AbortSignal.timeout(ms) makes fetch reject with a TimeoutError:

const res = await fetch(url, { signal: AbortSignal.timeout(2_000) })
// rejects with TimeoutError after 2 seconds if there is no response

Retry only operations that are safe to repeat (reads, or writes protected by an idempotency key), use exponential backoff with jitter, and cap the attempts. Retrying a non-idempotent write that actually succeeded is how customers get charged twice.

Logging

A log line is a message to a person debugging at a bad time. Make it answer: what happened, to which request, for which user, in which service, and how bad.

Structured, not free text

Write JSON lines, with fixed fields, rather than sentences with values glued in. They are searchable, filterable and countable. Pino is a popular logger for this in Node, and it produces output like this (an illustrative example; the real field set includes pid and hostname):

{"level":30,"time":1761650000000,"pid":4242,"hostname":"api-1","req":{"id":"7f3a"},"msg":"request"}

Structure also solves a security problem. The OWASP Logging Cheat Sheet defines log injection as untrusted data in a log message, and tells you to prefer structured formats such as JSON and to sanitize CR and LF characters. I tested what that means. The same hostile input was logged two ways:

const evil = 'ok\n{"level":30,"msg":"admin login ok","user":"root"}'

logger.info({ email: evil }, 'signin failed')  // 1 physical line: the newline is escaped inside the JSON string
console.log(`signin failed for ${evil}`)       // 2 physical lines: the attacker has forged a second log entry

With string concatenation the attacker's text becomes a line of its own that looks like a legitimate event. With a structured logger the newline stays inside one string value.

Redact secrets by default

The OWASP cheat sheet lists what should not be recorded directly: authentication passwords, access tokens, encryption keys and other primary secrets, and session identifiers (hash them if you must correlate), with sensitive personal data de-identified. Do this in the logger's configuration so it does not depend on every developer remembering. I tested pino's redact option:

const logger = pino({
  redact: {
    paths: ['req.headers.authorization', 'req.headers.cookie', '*.password'],
    censor: '[Redacted]',
  },
})

logger.info({ req: { headers: { authorization: 'Bearer abc', cookie: 'sid=1', accept: 'json' } },
              user: { password: 'hunter2', id: 5 } }, 'request')
// authorization, cookie and password all logged as "[Redacted]"; accept and id untouched

Redaction works on paths you list, so it is a safety net with gaps, not a guarantee. The stronger habit is to not put whole request or user objects in logs at all, and to log the specific fields you need.

What to log

OWASP's list of events worth recording is a good starting point: authentication successes and failures, authorization failures, input validation failures (suspicious, though not proof of an attack), session-management failures, use of administrative privileges, and key rotation. That matches the earlier posts in this series: Part 4's failed sign-ins and Part 5's denied permission checks are exactly the events you want to find after an incident. OWASP also says to synchronize clocks across servers and to alert when logging itself stops.

Choose levels with intent: error for failures that need a person, warn for handled problems worth watching, info for significant business events, debug for development. If error fires constantly for ordinary client mistakes, people stop reading it.

A request id on every line

Logs from concurrent requests interleave. To follow one request, give it an id and attach it to every line it produces. Passing that id through every function signature is tedious; Node's AsyncLocalStorage carries it for you. The docs describe it as state that follows a chain of callbacks and promises (similar to thread-local storage), and mark it stable since v16.4.0. I tested that the context survives the places people worry about:

import { AsyncLocalStorage } from 'node:async_hooks'
const als = new AsyncLocalStorage<{ id: string }>()

app.use((req, res, next) => als.run({ id: req.headers['x-request-id'] ?? crypto.randomUUID() }, next))

// anywhere in the call chain:
const requestId = als.getStore()?.id

My test read the store after an await, inside a setTimeout callback and inside Promise.all, and got the same id each time, and undefined outside the run. Libraries with callback-style APIs can lose context; the Node docs point to AsyncResource for binding those cases.

Metrics: the four golden signals

Logs tell you about individual events. To see the shape of the whole system, count and time things. Google's SRE book gives a short list worth memorizing, the four golden signals:

• Latency: the time to serve a request. The book stresses separating successful from failed requests, because a fast 500 can make an overall average look healthy, and "a slow error is even worse than a fast error".
• Traffic: demand on the system, such as requests per second.
• Errors: the rate of requests that fail, whether explicitly (a 500), implicitly (a 200 with the wrong content), or by policy (slower than a committed time limit).
• Saturation: how "full" the service is, for the most constrained resource. The book notes that systems often degrade before they reach 100% utilization.

Three practical consequences:

Use distributions, not averages. The SRE book's example: a 100 ms average can include requests that take 5 seconds. Record latency in buckets (a histogram) and watch a high percentile such as the 99th.

Mind label cardinality. Every distinct combination of label values is another time series with memory and storage cost. The Prometheus instrumentation guide suggests keeping the cardinality of a metric below 10 as a rule of thumb, and investigating alternatives for metrics over 100 or able to grow that large. Its guidance is about total cardinality; the rule that a user id or email must never be a label is my inference from it. Put high-cardinality detail in logs and traces instead.

Export zero. The same guide notes that a time series which does not exist until something happens is hard to alert on, so initialize counters you expect (such as error counts) to 0.

For a backend like the ones in this series, the signals worth adding are concrete: request count, latency histogram and error count per route (use the route pattern, not the raw URL, as a label), plus rate-limit rejections (Part 6), failed sign-ins (Part 4), authorization denials (Part 5), and saturation measures such as event-loop delay, database pool usage and queue depth.

Traces: following a request across services

When one user action fans out across several services or a database and a third-party API, a log from each is not enough to show where the time went. A trace is the path of one request through the system, made of spans (one unit of work each).

For spans from different services to join into one trace, the services must pass context along. The W3C Trace Context standard defines the traceparent header for this:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             |  |                                |                |
             |  trace-id (32 hex chars)          parent-id        trace-flags
             version                             (16 hex chars)   (2 hex chars)

Per the specification, an all-zero trace id or parent id is invalid and must be ignored, and the least significant bit of trace-flags is the sampled flag, which tells the next service that the caller may have recorded trace data. A companion tracestate header carries vendor-specific data.

OpenTelemetry is the common vendor-neutral way to produce this. Its documentation lists the signals it supports as traces, metrics, logs and baggage. For Node, the getting-started guide uses the Node SDK and the auto-instrumentation package, started from a separate instrumentation file that must run before your application code, loaded with the --import flag:

node --import ./instrumentation.mjs app.js

The most useful thing you can do with a trace id is put it in your logs. Then an alert points to a slow trace, the trace points to the slow span, and the span's id finds the log lines for exactly that request. That is what makes three separate tools behave like one.

Tracing everything is expensive, so production systems sample. Record all errors and slow requests if your tooling lets you, and a fraction of the rest.

Alerting

Collecting data is not the same as knowing something is wrong. The SRE book's guidance is a useful filter: monitor both what is broken (the symptom) and why (the cause), but put much more effort into catching symptoms than causes, limit cause-based alerts to very definite and imminent causes, and make every page actionable instead of firing because "something seems a bit weird".

For an API, that translates to alerting on user-visible symptoms: error rate above a threshold, latency at a high percentile above a threshold, or traffic collapsing to near zero. A CPU graph is a useful dashboard panel and a poor pager. OWASP adds one more: alert when logging itself stops, because silence is indistinguishable from "all quiet" until you check.

Health checks and graceful shutdown

Two related endpoints help an orchestrator decide what to do with your process. A liveness check answers "is the process stuck?" (restart it if so). A readiness check answers "can it serve traffic right now?" (stop sending requests if not, for example while the database is unreachable or during shutdown). Keep liveness trivial: if it checks the database, a database outage restarts every healthy instance.

When the process is told to stop (a deploy, a scale-down), the sequence should be: stop accepting new work, finish what is in flight, release resources, exit. In Node that means handling SIGTERM:

let shuttingDown = false

async function shutdown() {
  if (shuttingDown) return
  shuttingDown = true
  setTimeout(() => process.exit(1), 10_000).unref()   // hard stop if draining hangs

  server.close(() => {
    // all requests finished and connections closed: release the rest
    db.end().finally(() => process.exit(process.exitCode ?? 0))
  })
  server.closeIdleConnections()
}

process.on('SIGTERM', shutdown)
process.on('SIGINT', shutdown)

I tested a version of this and found a trap that the happy path hides. The in-flight request finished and new connections were refused (ECONNREFUSED), as expected. But server.close() did not complete: the connection that served the last request stayed open on keep-alive, and close waited for it. Measured from the start of shutdown, closing took about 6 seconds (the default keepAliveTimeout is 5,000 ms), which is long enough for an orchestrator to give up and kill the process, and long enough to cut off requests you thought were safe.

The fix I tested: once shutdown has begun, respond with Connection: close so each connection is closed as its response finishes.

app.use((req, res, next) => {
  if (shuttingDown) res.setHeader('Connection', 'close')
  next()
})

With that, server.close() completed in about 250 ms in the same test. closeIdleConnections() only closes connections that are idle at the moment you call it, so a connection that becomes idle a moment later is not covered. Also make sure the orchestrator's grace period is longer than your own drain timeout.

In runtimes without a long-lived process, such as edge or serverless platforms, there is no SIGTERM to handle and per-request context and log shipping depend on the platform. The principles in this post still apply; the mechanisms differ.

Putting it together in an incident

Suppose an alert fires: error rate on the checkout route has crossed its threshold.

1. Metrics show which route, since when, and whether latency moved with it. The change started at 14:05, errors are 500s, and latency rose first.
2. Traces for failing requests show where the time goes. Every slow trace has the same long span: a call to a payment provider.
3. Logs, filtered by one of those trace ids, show the same timeout error on each, with the cause chain: a TimeoutError after 2 seconds.
4. Mitigation: you already set a timeout, so the failures are fast and contained. You tighten the circuit (or turn off the provider path) while it recovers.

(That scenario is hypothetical, to show how the signals connect.) Without all three signals, each step becomes a guess.

Common mistakes

Interview answer vs production answer

The interview answer: "Use try/catch, log errors, and add monitoring."

The production answer is about the details that decide whether you can diagnose an incident at 3 a.m.:

• Defaults matter. An unhandled rejection kills Node by default; an uncaughtException handler turns that into exit code 0. Know which you have.
• Frameworks differ on async errors. Test yours once, deliberately.
• Errors need a boundary. One handler that logs once with context and returns a generic, consistent response.
• Logs are a security surface. Structured output prevents forged lines, and redaction keeps secrets out.
• Correlation beats volume. A request id and a trace id in every line are worth more than ten times the logging.
• Percentiles and symptoms. Alert on what users feel, measure the tail, not the mean.
• Shutdown is part of error handling. Draining connections correctly is what makes deploys boring.

FAQ

Should I catch every error?

No. Catch the ones you can do something about: retry, fall back, translate to a status code. Let the rest propagate to the boundary handler. Catching and ignoring an error hides a bug and moves the failure somewhere harder to debug.

Is it ever right to keep running after an uncaught exception?

The Node documentation says it is not safe to resume normal operation. Log it, clean up synchronously, and exit so a supervisor restarts a clean process.

What is the difference between logs, metrics and traces?

Logs are records of individual events. Metrics are numbers aggregated over time, good for dashboards and alerts. Traces show the path and timing of one request across components. They answer different questions and work best joined by a shared request or trace id.

How much should I log?

Enough to reconstruct what happened to a request and to detect the security events listed earlier, and no more. Volume costs money and buries signal. Prefer structured fields to long messages, and sample high-volume, low-value logs.

Do I need OpenTelemetry?

Not on day one. Structured logs with a request id and a handful of metrics get you far. Tracing earns its place when requests cross several services or when you cannot tell where the time goes.

Why are my average latencies fine while users complain?

Averages hide the tail. A small share of very slow requests barely moves the mean but is what those users experience. Record a histogram and look at a high percentile.

Should the health check call the database?

For readiness, often yes: it tells the load balancer to stop sending traffic if the dependency is down. For liveness, usually no, because a dependency outage would restart healthy instances and make things worse.

Why does my deploy take several seconds to stop the old version?

A likely cause is open keep-alive connections. In my test server.close() took about 6 seconds until I responded with Connection: close during shutdown, after which it took about 250 ms.

Glossary

The end-to-end mental model

 request arrives
      |
      v
 assign request id  (AsyncLocalStorage)  +  continue trace (traceparent)
      |
      v
 handler runs ------------- failure ---------------------------+
      |                                                         |
      v                                                         v
 success: count + time it                          typed error / unexpected error
                                                                |
                                                                v
                                                  boundary handler: log ONCE with
                                                  cause chain + request id + trace id
                                                                |
                                         +----------------------+---------------------+
                                         v                                            v
                                 generic problem+json                         error metric + (maybe) alert
                                 to the client (Part 6)                       on symptoms users feel

 fatal: unhandledRejection / uncaughtException -> log -> exitCode=1 -> drain (Connection: close) -> exit -> supervisor restarts

The first six posts of this series were about who may do what and how to keep strangers out. This one was about seeing what the system is doing when it fails. The next step goes one layer down, to where most of the interesting failures actually originate: the database. Part 8 covers connections, pools, transactions and ORM patterns, including how to keep queries fast and consistent under load.