In a previous post I described load balancing for TCP-based HTTP with connection offloading, and mentioned that HTTP/3 can do even more. By the power of AI writing an HTTP/3 version of both the load balancer (sni-router-h3) and the webserver (tarweb-h3), I have experimented enough to have more to say about how that works.

To be clear: this post is 100% human-written.

In this post I will use this terminology:

  • client: E.g. a web browser.
  • router: A load balancer / traffic router, that inspects SNI and forwards to matching backends. Mine’s called sni-router.
  • backend: A webserver. Mine’s called tarweb.

First, the TCP story

While it is possible to use TCP_REPAIR to transfer an established connection between machines (from router to backend), it requires some more infrastructure. The kernel on the chosen backend must be synced up on the connection state (sequence numbers, etc) that TCP_REPAIR provides, but that’s actually just the easy part. It must also be able to send packets to the client with the original server’s IP address as source, and packets going from the client to the backend must somehow find their way to the same backend, in a network-efficient way.

There are other problems too, like what to do if a client sends a TCP SYN with a source port equal to an existing connection. Which connection should win for the routing state machine?

Take the example where the load balancer in New York wants the backend to be in Oslo, because the client is near there.

For backend-to-client packets, some network hop on the way may drop the packets due to reverse-path forwarding (RPF), because it thinks that IP address should be in New York, and therefore the packets look like spoofing.

For client-to-backend packets, there’s no way to announce to the Internet the difference between initial packets that should go to the load balancer, and those that should go to the backend. The difference is just the client IP address and port. And the Internet (specifically BGP) does not work that way.

If you have a worldwide presence and private fibre network on the scale of Google or Amazon, this is physically possible. “Be everywhere” and then on every new connection you can “just” reprogram all your edge router infrastructure to route packets arbitrarily over your own world wide (physical) network, to traffic engineer however you want. But this is basically only if you have your own world wide “internet” that treats “the Internet” as an equal.

Yes, for client-to-backend you could use stabilized anycast if you’re merely medium sized, but this can only solve the problem if the backend and the load balancer should take the same path. You’ll need all backend locations to also be load balancing locations, and vice versa, to have it be optimal. And it can break for longer lived connections in a dynamic world.

(A full discussion about anycast TCP is out of scope for this post, but though I could have made a mistake or have been unclear here, and invite you to correct me on, if you’re thinking “well, actually you can just…” right now, you can safely assume that I know and I’ve deliberately left it out.)

What HTTP/3 enables

HTTP/3 is UDP based, and is not picky about “the 4 tuple” (the values of source and destination IP addresses and ports). Migrating a connection therefore doesn’t involve BGP peering or other network tricks. HTTP/3 can thus let the Internet be a simple packet delivering network that just takes the closest path between smart endpoints. As was always the dream.

Case 1: A new connection

For a new connection’s TLS handshake, a client will send the SNI. A SNI router can then make a routing decision, and forward the packet to the backend.

During this handshake, the router needs to remember the mapping from Connection ID (CID) to the backend it selected when it saw the SNI.

Case 2: Router sees a packet from an unknown connection

After the handshake is complete, the client and the backend can open new connections with new CIDs. This is so that a passive evesdropper cannot track connections, including as they migrate if the client roams somewhere else. Since the router also never sees the plaintext, it is not aware of the new connection ID.

It could fall back to remembering the client host-port pair, but clients can migrate, and that should not break connections. It can be used if the client migrates during the handshake, though.

Even if it did know about the new Connection ID (which it doesn’t), or remembered the client IP-port pair (it can’t, because client address migrations), it shouldn’t need to know about this forever, e.g. across router restarts.

This case is solved by QUIC-LB. Outshort, the backend will put its “backend ID” into the new Connection ID, and the router can then use that to make sure it forwards to the same backend, even without keeping any state.

An entire QUIC Connection ID is at most 160 bits, so there isn’t spare room to put a whole 128 bit IPv6 address plus 16 bit port in there. The solution is to have a lookup table from backend ID to backend address:port.

The Connection ID with the backend ID is then optionally encrypted, so that attackers can’t control it for whatever nefarious purpose. If we were to stop here, it could be to prevent attackers from targeting a specific backend. But handling case 3 nicely, below, forces us to publicly expose backends anyway.

QUIC-LB is a draft standard, but in my case both router and backend follow the same standard, and therefore work together.

For sni-router, the config looks like this:

quic_lb: <
        config_id: 0
        key_file: "/etc/sni-router/quic-lb.key"
        servers: <
                server_id: 123 addr: "192.0.2.123:8444"
        >
        servers: <
                server_id: 124 addr: "192.0.2.13:8444"
        >
>

The config ID is there to allow changing the backend list over time, as a generation number. Though it’s only 3 bits, so few config generations can be active at the same time.

But case 2 shouldn’t happen, because…

Case 3: Backend-announced preferred address

During the TLS handshake, which always goes via the router (though the router only forwards. It doesn’t “speak” TLS), the backend may announce a “preferred address” (a separate one for IPv4 and IPv6), in other words that backend’s real IP address.

The client should then stop sending packets to the router, and instead send directly to the backend.

From the backend’s point of view, the client just migrated, and it too will start sending packets directly, bypassing the router.

The router is now no longer part of the connection.

Unfortunately it’s not mandatory that clients respect “preferred address”. And the backend can’t unilaterally send directly to the client on its own, because If a client receives packets from a new server address when the client has not initiated a migration to that address, the client SHOULD discard these packets.

So if the client doesn’t support preferred address, then we fall back to Case 2, and the router has to take the network load of the traffic. At least it doesn’t have to decrypt anything.

In my brief test, Chrome does respect “preferred address”, but curl does not.

For command line testing, ngtcp2-client has a client that works.

Does it work?

Yeah. Outthis tcpdump (on the client) we see a connection setup seen by running:

gtlsclient good.habets.uk 443 https://good.habets.uk/

# TLS handshake between client and router.
09:00:12.366608 eth0  Out IP Client.3968 > Router.443: UDP, length 1200
09:00:12.372773 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
09:00:12.372849 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
09:00:12.372895 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
09:00:12.412559 eth0  Out IP Client.3968 > Router.443: UDP, length 1200
09:00:12.413771 eth0  In  IP Router.443 > Client.3968: UDP, length 928
09:00:12.447141 eth0  Out IP Client.3968 > Router.443: UDP, length 364
09:00:12.448748 eth0  In  IP Router.443 > Client.3968: UDP, length 344
09:00:12.448788 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
09:00:12.448804 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
09:00:12.448820 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
09:00:12.448861 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
09:00:12.448879 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
09:00:12.448913 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
09:00:12.448942 eth0  In  IP Router.443 > Client.3968: UDP, length 268
09:00:12.451582 eth0  Out IP Client.3968 > Router.443: UDP, length 1406
09:00:12.452652 eth0  Out IP Client.3968 > Router.443: UDP, length 44
09:00:12.487682 eth0  In  IP Router.443 > Client.3968: UDP, length 44

# Handshake complete. Traffic is now almost entirely directly between the
# client and the backend. I think the remaining packets are just during the
# migration.
#
# Note that the client port changed, because my NAT router thinks this is a
# completely unrelated connection. That's not a problem.
09:00:12.490258 eth0  Out IP Client.64998 > Backend.8444: UDP, length 1200
09:00:12.490375 eth0  Out IP Client.64998 > Backend.8444: UDP, length 1200
09:00:12.490537 eth0  Out IP Client.3968 > Router.443: UDP, length 151
09:00:12.490645 eth0  In  IP Backend.8444 > Client.64998: UDP, length 1200
09:00:12.490758 eth0  In  IP Backend.8444 > Client.64998: UDP, length 1200
09:00:12.490908 eth0  In  IP Router.443 > Client.3968: UDP, length 43
09:00:12.522221 eth0  Out IP Client.3968 > Router.443: UDP, length 1444
09:00:12.523153 eth0  In  IP Router.443 > Client.3968: UDP, length 43
09:00:12.526550 eth0  Out IP Client.64998 > Backend.8444: UDP, length 1406
09:00:12.527011 eth0  In  IP Backend.8444 > Client.64998: UDP, length 1200
09:00:12.527271 eth0  In  IP Router.443 > Client.3968: UDP, length 1200
# Looks like no more packets to/from Router past this point.
09:00:12.527833 eth0  Out IP Client.64998 > Backend.8444: UDP, length 42
09:00:12.528160 eth0  In  IP Backend.8444 > Client.64998: UDP, length 1200
09:00:12.560274 eth0  Out IP Client.64998 > Backend.8444: UDP, length 1444
09:00:12.560699 eth0  Out IP Client.64998 > Backend.8444: UDP, length 1200
09:00:12.560910 eth0  In  IP Backend.8444 > Client.64998: UDP, length 43
09:00:12.566818 eth0  Out IP Client.64998 > Backend.8444: UDP, length 1200
09:00:12.583812 eth0  Out IP Client.64998 > Backend.8444: UDP, length 109
09:00:12.584162 eth0  In  IP Backend.8444 > Client.64998: UDP, length 43
09:00:20.007812 eth0  Out IP Client.64998 > Backend.8444: UDP, length 39
09:00:20.010182 eth0  In  IP Backend.8444 > Client.64998: UDP, length 47

If you run this test yourself against my example setup, don’t be confused by the router and backend being on the same address, just different ports. They happen to only have different ports, but it would work equally well if they were on different machines.

I have not dug into the exact packets to/from the router post-handshake. My guess is that it’s probably the client tearing down that part. But remember, the router does not speak QUIC at all. Anything the client sent to the router ends up as-is on the backend. And anything the backend replied back to the router goes untouched back to the client.

In my setup the backend allows new connections created directly for it. That can easily be prevented, by having the backend blocking new connections that don’t come from the router. Clients may be able to spoof the source to appear to be the router, but then they should not get any replies, so the connection establishment will fail.