Production Troubleshooting

Introduction

Production issues—startup failures, OutOfMemoryError, slow SQL, thread pool exhaustion—need systematic diagnosis. This chapter maps symptoms to checks: logs, Actuator metrics, thread dumps, and rollback habits so you recover quickly without guessing.

Prerequisites

Spring Boot Actuator and Observability
Packaging and Deployment
Query Performance Basics for SQL slowness

Startup Failures

Read the root cause at bottom of stack trace.

Symptom	Common cause	Action
Port already in use	Old process on 8080	Kill process or change `server.port`
Failed to configure DataSource	Wrong URL/credentials	Fix env vars; test DB from host
Bean creation exception	Missing dependency, typo in `@Autowired`	Run with `--debug`, read condition report
Flyway/Liquibase validation	Schema drift	Align migration or repair

bash

java -jar app.jar --debug 2>&1 | tee startup.log

See Startup Flow.

Application Not Healthy

bash

curl -s http://localhost:8080/actuator/health | jq

DOWN components point to failing dependency—Redis, disk space, custom indicator.

Check recent deploy config change before deep debugging.

OutOfMemoryError

Types:

Error	Meaning
Java heap space	Too many objects in heap
GC overhead limit exceeded	GC thrashing
Metaspace	Class loader leak / too many classes

Capture heap dump (once):

bash

jcmd <pid> GC.heap_dump /tmp/heap.hprof

Analyze with Eclipse MAT or similar—find retained largest objects.

Mitigations:

Increase -Xmx only after understanding leak
Fix unbounded caches
Paginate large queries

Slow HTTP / Timeouts

Actuator metrics — http.server.requests p95 latency
Logs — correlate slow request id / trace id
Database — enable slow query log on MySQL
External APIs — timeouts on RestClient (chapter 21)

Thread dump for stuck requests:

bash

jcmd <pid> Thread.print > threads.txt

Look for blocked threads on JDBC pool or locks.

Connection Pool Exhausted

HikariCP messages: Connection is not available, request timed out.

Check:

yaml

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      connection-timeout: 30000

Causes:

Long @Transactional holding connections
Pool size too small for concurrency
Connection leak (not closed)— rare with Spring JDBC if used correctly

Thread Pool Exhausted (@Async / Tomcat)

Tomcat max threads default 200—under load all busy.

@Async queue full—rejected executions.

Tune:

yaml

server:
  tomcat:
    threads:
      max: 300
spring:
  task:
    execution:
      pool:
        max-size: 32
        queue-capacity: 500

Fix root cause: slow downstream, not only raise limits.

SQL and ORM Issues

Symptom	Check
N+1 queries	JPA `@EntityGraph`, fetch join, or DTO query
Full table scan	`EXPLAIN`, add index
Lock wait timeout	Short transactions, index FK columns

Enable SQL log only temporarily in prod.

Emergency Rollback

Stop bad deployment traffic (load balancer drain)
Redeploy previous known-good image/jar
Restore DB from backup if bad migration applied—last resort
Post-incident: timeline, root cause, action items

Keep last three releases artifacts in registry.

Observability During Incident

text

Logs (what happened) → Metrics (how bad) → Traces (which span slow)

Do not enable TRACE globally in prod long-term—disk and CPU cost.

FAQ

Cannot reproduce locally?

Compare profile, data volume, and env vars—prod-only secrets wrong common.

Restart fixes it?

Suspect memory leak or external dependency glitch—still investigate.

Kill -9 safe?

Last resort—prefer graceful shutdown for in-flight requests.

Who owns DB vs app?

Split escalation—network blip vs query regression.

Spring Boot Admin?

Third-party UI aggregating Actuator across instances—optional ops tool.

More learning?

Spring Boot Up and Running and official troubleshooting guides.