DEV Community

Cover image for More servers vs right-sizing: choosing your response to a capacity problem
binadit
binadit

Posted on Originally published at binadit.com

More servers vs right-sizing: choosing your response to a capacity problem

Scaling vs fixing the bottleneck: how to tell which incident you actually have

Your p95 latency spikes, Slack lights up, and someone says "just bump the instance size." Thirty minutes later the graphs look fine and everyone moves on to the next fire. But did you fix anything, or did you just make the same inefficiency more expensive?

This is the call every on-call engineer eventually has to make under pressure. Let's break down when scaling is the right move, when it's a band-aid, and how to tell the difference fast.

Why "add more servers" works until it doesn't

Scaling, horizontal or vertical, is the fastest lever you have mid-incident. No schema migration, no code review, no root cause investigation required. That speed is genuinely valuable when customers are feeling the pain right now.

It's the right call when:

  • Traffic growth is real and sustained, not a spike from an inefficient query
  • Your app is stateless and actually horizontally scalable, so new nodes spread load instead of just shifting it downstream
  • You need breathing room to investigate safely without customers noticing
  • The added cost tracks real revenue or usage growth, not a single bad code path

A common good pattern: scale out web tier capacity ahead of a launch, buy two weeks of headroom, then use that time to actually profile the slow endpoint. Scaling as a buffer, not a fix.

Where scaling quietly fails

Here's the trap: you scale out your app servers, but they all hit the same Postgres instance. Now you've tripled your compute bill and connection count or lock contention is the new ceiling. Latency barely moves.

# Looks like scaling fixed it
app_servers: 2 -> 6

# Reality check
pg_stat_activity.count # climbing
lock_waits             # climbing
response_time_p95      # basically unchanged
Enter fullscreen mode Exit fullscreen mode

Scaling also multiplies misconfiguration. Clone a server with a bad config into a fleet of five, and you now have five copies of the same bug instead of one fixed server. This overlaps heavily with configuration drift problems, where small inconsistencies between nodes become the actual source of intermittent incidents.

And there's a cost ceiling: doubling instance count doesn't double your headroom if the workload itself is inefficient. It just doubles your bill while the percentage improvement in latency keeps shrinking.

When to profile instead of provision

Fixing the root cause means treating the slowdown as a signal, not a capacity gap. Pull up your request traces and look at query plans, N+1 queries, cache hit ratios, slow external calls, thread pool exhaustion, connection pool limits.

This is the right move when:

  • The same handful of endpoints or queries keep showing up in slow traces
  • CPU and memory sit comfortably under 50% while latency is still bad, pointing to contention, not raw capacity
  • A recent deploy correlates with the degradation, suggesting a regression, not organic growth
  • Scaling costs have already outpaced revenue growth, meaning the architecture can't scale economically even if it can scale technically

One real example: a team kept adding nodes to a cluster that was actually constrained by inefficient pod scheduling and oversized resource requests, not genuine load. Removing the orchestration overhead and right-sizing the workload did more than another autoscaling rule ever would have.

The catch with root-causing under pressure

Profiling takes time you might not have mid-incident. Testing an index change against production-like data and validating it doesn't regress something else can take days. "Let's fix the query" is not an acceptable answer to an active P1, even when it's the correct long-term one.

It also requires expertise your team might not have on hand, especially for database internals or distributed systems behavior under load. Without someone who can read a query plan confidently, you'll spend more time guessing than fixing, and end up scaling anyway.

And sometimes the fix really is more capacity. If your user base tripled and the architecture was built for the old scale, no amount of query tuning replaces infrastructure that matches current reality.

Quick decision framework

  1. CPU/memory/connections genuinely saturated, sustained real traffic growth? Scale. This is legitimate capacity planning. Just verify your horizontal scaling is actually distributing load, not just adding nodes that all hit the same downstream bottleneck.
  2. Moderate resource utilization, latency still bad? Don't scale yet. Profile first. This is almost always lock contention, slow queries, or a misbehaving dependency.
  3. Same component keeps showing up across incidents? Stop patching with capacity. Fix it.

The healthiest pattern: scale to stabilize, buy time, then do the root cause work properly. The failure mode is scaling as a permanent substitute for diagnosis, every single time load increases.

Full breakdown with a comparison table on cost, operational burden, and risk profile: More servers vs right-sizing

Originally published on binadit.com

Top comments (0)