DEV Community

OneLifeVibe
OneLifeVibe

Posted on

Keep the Sessions Alive While You Replace the Gateway: A Draining Plan for Private Cross-Cloud Links

Every private link between clouds eventually meets a maintenance event its designers did not schedule around. A gateway appliance reaches end of support, a virtual network gateway needs a SKU change, an encryption requirement forces new hardware, or a provider retires the exact tunnel configuration the estate was built on. The replacement itself is usually the easy part. A new gateway comes up, routes converge, health checks turn green. Then the tickets arrive, because the old gateway was not carrying traffic in the abstract. It was carrying sessions, and sessions do not move just because routing did.

A database connection opened six hours ago, a long lived gRPC stream, a file transfer at hour three, a replication channel that reconnects only when it breaks: all of them are pinned to the path they started on. Cut that path and they do not migrate. They die, each at a different layer, each with a different error, and the postmortem fills up with application bugs that were really one network decision. This article is about draining a cross-cloud gateway the way a good load balancer drains a backend: deliberately, in phases, with proof at each step.

Start With a Session Inventory, Not a Traffic Graph

Traffic graphs answer the wrong question. They show bytes and flows per hour, which makes a gateway look busiest at midday and quiet at night. Sessions have their own lifetimes, and a link that looks idle at 2 AM may still hold a replication stream that started at noon and matters more than everything else combined.

Before scheduling anything, inventory what actually lives on the path:

  • Long lived TCP connections: database links, message queue connections, replication channels, and tunnelled management sessions, with their typical and maximum lifetimes.
  • Stateful middlebox bindings: NAT translations, firewall state entries, and load balancer persistence records that tie a flow to this specific gateway.
  • Quiet but critical flows: monitoring agents, backup control channels, and time or directory synchronisation that send little and fail silently.
  • Reconnection behaviour per application: which clients retry cleanly, which retry in a storm, which cache a dead endpoint, and which need a restart to let go.

That last item decides the whole plan. An estate where every client reconnects gracefully can drain in an hour. An estate with one legacy service that holds connections until they error needs a window sized to that service, or a fix to that service first.

Drain in Four Phases, and Gate Each One

The common failure is a two step plan: build the new gateway, then cut over. Draining inserts the missing middle.

Phase one: prefer the new path for new sessions only. Shift route preference, DNS answers, or gateway priority so that newly opened connections land on the new gateway, while existing sessions keep the old one. Both gateways stay fully healthy. This is the step teams skip because nothing visibly changes, and it is the step that does most of the work, because short lived traffic migrates itself within minutes.

Phase two: watch the old gateway empty. Session counts, NAT table size, and state entries on the old gateway should fall along a curve that matches the inventory. A plateau is information: whatever is still holding sessions after the expected lifetime has passed is the thing that would have broken in a hard cut. Identify it while the old path still works, not after.

Phase three: shrink, do not sever, the remainder. For the sessions that remain, narrow the old path deliberately. Lower its advertised preference further, or rate limit new attempts on it to zero while keeping established flows passing. If a supposedly drained application opens a fresh connection on the old gateway at this stage, there is a hard coded route, a cached answer, or a forgotten static entry somewhere, and finding it now costs a log line instead of an outage.

Phase four: hold a rollback that is actually warm. Keep the old gateway configured, addressed, and routed, just depreferred, for a defined soak period after the last session closes. Rollback that requires rebuilding a gateway is not rollback, it is a second project started during an incident. The soak ends when monitoring shows zero flows for a full business cycle, not when the change window closes.

Test With a Synthetic Long Lived Flow

Short health checks prove a path exists. They say nothing about whether a session survives the transition, because they open and close in seconds, exactly like the traffic that was never at risk.

Before the real drain, run a synthetic witness: a client in one cloud holding a connection through the gateway to a server in the other, sending a small heartbeat every minute and logging gaps. Start it days before the change. During phase one it should stay on the old gateway and stay unbroken. During the final close it should fail exactly once, reconnect through the new gateway, and show that reconnection in its log. If the witness flaps during the early phases, the preference change is leaking into established flows, and the real applications would have felt it too.

Repeat the witness per traffic class that matters: one for a database style connection, one for a streaming flow, one for whatever the inventory flagged as fragile. Three witnesses that behave predictably are worth more than a dashboard of green checks that all test the same easy path.

The Traps That Make Drains Fail

A few patterns account for most failed gateway replacements:

  • Asymmetric return paths. The new gateway forwards the session, the reply returns through the old one, and a stateful device in the middle drops traffic it never saw open. Draining phases must move both directions of a flow together, or the stateful layers must be told not to care, explicitly and in advance.
  • Persistence configured in the wrong place. A load balancer in one cloud may keep sending new sessions to the old gateway because its own health check still passes. Depreference has to happen at every layer that makes a path decision, not just in routing.
  • Idle timeouts shorter than the drain. If an intermediate device kills idle sessions after ten minutes, a quiet replication channel may already be dead on the old path and simply not know it yet. The drain plan should include a keepalive or a planned reconnect for those flows, before the gateway change gets blamed for their failure.
  • Change windows sized to the build, not the drain. Provisioning a gateway takes an hour. Emptying one can take a day if the longest session lifetime is a day. The inventory sets the schedule, not the other way round.

What Good Looks Like

A well drained replacement is boring to watch. New sessions appear on the new gateway. Old session counts fall on a predictable curve and the exceptions get names while there is still time to handle them. The final close produces a handful of clean reconnects, each already rehearsed by a synthetic witness, and the old gateway sits warm and empty until its soak expires and it can be retired without ceremony.

This is the kind of unglamorous sequencing work that decides whether a multi-cloud estate can be maintained or only built once and feared. It is a regular part of the private connectivity design and managed operations work at CloudVerg, where cross-cloud links built on VergMesh and delivered through VergForge are planned for their replacement day as carefully as for their first day, because a network that cannot change its gateways without dropping its sessions is not finished, however fast it benchmarks.

Top comments (0)