Solving connections, not requests

2026-06-30 · 5 min

For a server-side rendered page, the bottleneck is often network requests. The slowest one sets the floor for time to first byte (TTFB), and the Web Vitals that follow inherit the delay. For requests running in parallel, render time is at least max(...requests). If they run sequentially, it would be the sum.

That makes TTFB worth measuring per region, where customers actually are. A global percentile hides the customers furthest from where the services run.

On an AWS serverless stack, high percentiles are mostly driven by customers furthest from the origin. This post explores one of them: a dynamic request from Australia to a European origin.

The problem

Not every service runs in every region. On a serverless stack that's about dependencies, not cost. A service that depends on an external SaaS product that isn't multi-region should be co-located in the same region as that product for reliability. If the SaaS is hosted in Europe, the service should run in a European or neighbouring region relative to customers. An AWS Lambda in ap-southeast-2 calling a European region pays at least one ~265ms round trip per call, and three on a cold connection. Timeouts and retry behaviour are set by the vendor, so every millisecond spent in transit comes out of a fixed budget. A short hop leaves more of that budget for the dependency itself.

This is problematic where the data is both dynamic and specific to a user.

One round trip isn't the problem

Sydney to Europe and back is roughly 265ms. That's one round trip (RTT).

One round trip describes a request on a connection that already exists. A cold connection has to be set up first:

1. TCP handshake, 1 RTT. SYN out, SYN-ACK back.
2. TLS 1.3 handshake, 1 RTT. ClientHello out, ServerHello and certificate back. TLS 1.2 needs two.
3. The request itself, 1 RTT.

That's 3 × 265ms, about 800ms before the first useful byte.

Most of that is setup before any data is sent. It's about solving connections, not requests: a reused connection pays one round trip instead of three.

The connection in question isn't the caller's. A request enters AWS CloudFront at an edge location close to the customer, and that hop is short. The expensive one is CloudFront's own connection from that edge back to the origin, which could be across the globe.

Keeping connections warm

CloudFront pools its connections to the origin and reuses one if it's open. A warm connection skips TCP and TLS, so the request costs 1 RTT, not 3.

The obvious lever is the origin keep-alive timeout, which is how long CloudFront keeps an idle origin connection open. The default is 5 seconds, and the origin's own idle timeout caps it.

Will raising it solve the problem? It will help, but it won't produce a big drop. Why?

Because there isn't one connection pool. CloudFront consists of many points of presence (PoPs, the edge locations), each holding its own connections to the origin. A connection only stays warm if another request arrives on it before the idle timer runs out, so what matters is how often requests reach each pool. Keep-alive doesn't change that.

For a route that sends one request every 2 seconds through a single pool with a 5 second timeout, the connection never goes cold. Spread across ten pools, each sees a request every 20 seconds, and most arrive cold. Australian traffic to a European origin is the second case.

So it's still a roll of the dice. Is there a way to fix the distribution itself? Yes, but it's an unusual one.

AWS CloudFront Origin Shield

Origin Shield adds one more CloudFront cache layer, in a single chosen region, between every edge location and the origin:

without:  caller → edge location → origin
with:     caller → edge location → Origin Shield → origin

It has two primary features:

• Caching. Edge locations around the world don't share a cache. With Origin Shield, a miss at any edge can be a hit at the shared layer, so the origin does less work.
• Request collapsing. When many edge locations ask for the same object at once, Origin Shield sends one request to the origin and shares the response.

Neither helps a dynamic request. The response is different for every customer, so there's nothing to cache or collapse, and AWS's guidance is that Origin Shield may not suit dynamic content that's just proxied to the origin.

The hypothesis

Both features depend on every origin-bound request, from every edge location, passing through the same place. If the requests converge, so do the connections. The edge locations stop holding their own connections to the origin. Origin Shield holds them, in one region.

So the hypothesis: if origin connections are pooled centrally and all of the traffic flows through them, they should stay warm. A request from Australia wouldn't need a TCP or TLS handshake with the origin at all.

It isn't literally one pool. Origin Shield is a fleet across at least three availability zones, so fewer, hotter pools is the accurate way to put it. And it sits in the same region as the origin, so when it does need a new connection, each round trip of the handshake takes a millisecond or two.

Using a warm connection would go from a sporadic gamble to the default.

Results

Australian customers saw p75 TTFB drop by 300-380ms on public pages. Values are scaled and not exact:

It isn't free. Origin Shield bills per request for dynamic content, which includes any GET with caching disabled. That came to around $7 a day at this traffic volume. Ingress regional traffic incurs no additional cost.

Origin Shield earned its place purely as somewhere for connections to converge, and AWS Solutions Architects were surprised, pleasantly of course, by the result. One AWS post on dynamic workloads does note in passing that Origin Shield "can also increase the connection reuse rate", but the main guidance presents it as another cache layer.

CloudFront's Server-Timing response header includes cdn-upstream-connect. Zero means an existing origin connection was reused, and anything else is the time spent on TCP and TLS setup. With Origin Shield enabled, the measured reuse rate was 81%.