TCP_MD5 (RFC 2385) is something that doesn’t come up often. There’s a couple of reasons for that, good and bad.

I used it with tlssh, but back then (2010) it was not practical due to the limitations in the API on Linux and OpenBSD.

This is an updated post, written after I discovered TCP_MD5SIG_EXT.

What it is

In short it’s a TCP option that adds an MD5-based signature to every TCP packet. It signs the source and destination IP addresses, ports, and the payload. That way the data is both authenticated and integrity protected.

When an endpoint enables TCP MD5, all unsigned packets (including SYN packets) are silently dropped. For a signed connection it’s not even possible for an eavesdropper to reset the connection, since the RST would need to be signed.

Because it’s on a TCP level instead of part of the protocol on top of TCP, it’s the only thing that can protect a TCP connection against RST attacks.

It’s used by the BGP protocol to set a password on the connection, instead of sending the password in the handshake. If the password doesn’t match the TCP connection doesn’t even establish.

But outside of BGP it’s essentially not used, which is a shame. If we could enable it for any TCP service it’d add a preshared key and if nothing else completely replace the silly port knocking. It probably couldn’t replace user passwords, but it could add a layer and greatly reduce attack surface much more than, say, a TLS certificate.

It’s MD5. Sure, MD5 isn’t great. But it still doesn’t have any preimage attack. Well, none that’s feasible anyway.

So that should be fine. And if not then there’s already TCP AO which is about the same but with other algorithms. But TCP AO doesn’t actually seem to be implemented very widely.

How to use it

For the server (on Linux)

const char* password = "hello";
struct tcp_md5sig sig;
memset(&sig, 0, sizeof(sig));
memcpy(&sig.tcpm_addr, &peer_sockaddr, peer_sockaddr_len);
sig.tcpm_flags = TCP_MD5SIG_FLAG_PREFIX;
sig.tcpm_prefixlen = 0;                  // Match any address.
sig.tcpm_keylen = std::min(TCP_MD5SIG_MAXKEYLEN, strlen(password));
memcpy(sig.tcpm_key, password, sig.tcpm_keylen);
if (setsockopt(s, IPPROTO_TCP, TCP_MD5SIG_EXT, &sig, sizeof(sig)) == -1) {
  fprintf(stderr, "Failed to setsockopt(): %s\n", strerror(errno));
  exit(1);
}

For the client (on Linux)

const char* password = "hello";
struct tcp_md5sig sig;
memset(&sig, 0, sizeof(sig));
memcpy(&sig.tcpm_addr, &peer_sockaddr, peer_sockaddr_len);
sig.tcpm_keylen = std::min(TCP_MD5SIG_MAXKEYLEN, strlen(password));
memcpy(sig.tcpm_key, password, sig.tcpm_keylen);
if (setsockopt(s, IPPROTO_TCP, TCP_MD5SIG_EXT, &sig, sizeof(sig)) == -1) {
  fprintf(stderr, "Failed to setsockopt(): %s\n", strerror(errno));
  exit(1);
}

On the client you can use TCP_MD5SIG instead of TCP_MD5SIG_EXT, since it won’t need to set the prefixlen.

The sad reason it’s not used: It doesn’t work through NAT

Because it signs the source and destination address and port. This is getting to be less and less of an issue as the world goes IPv6. So maybe we can see more of this in the future.

This was never really a problem for BGP, since production BGP doesn’t run through NAT.

It used to be impossible to use on Linux for most applications

The TCP_MD5SIG socket option on Linux requires specifying exactly what the remote address is. This doesn’t make sense for listening sockets, like an OpenSSH server, which won’t know ahead of time what the remote address is.

BGP doesn’t really have clients and servers, just mutually configured peers, so it works fine there.

It used to be possible (back in 2010) to enable MD5 after a connection is established. And indeed this is what I did with tlssh back in 2010. But trying that now it results in EINVAL. Which is odd, because TCP MD5 was made for routers, and routers most certainly allows enabling TCP MD5 on an existing connection.

Back when you could enable it for an existing connection you could actually make this practical without TCP_MD5SIG_EXT, and it’s what I did with tlssh: You enable TCP MD5 immediately after the connection is established. If the two ends don’t have the same password set, then no packets will go through in either direction, and the connections will just time out. Not the best experience, but it worked. Nowadays with TCP_MD5SIG_EXT it works perfectly and as expected.

The latest code for tlssh uses the newer better way.

On OpenBSD it’s a system-level setting

On OpenBSD you set up the TCP MD5 as a security association between the two hosts, and then the program just enables MD5 on the socket. And it “just works” on both the client and the server. Well, a bit annoying that the key is set in hex.

# cat > /etc/tcpmd5.conf
tcpmd5 from 2001:db8::1111 to 2001:db8::2222 spi 0x100 authkey 0x68656c6c6f
tcpmd5 from 2001:db8::2222 to 2001:db8::1111 spi 0x101 authkey 0x68656c6c6f
^D
# ipsecctl -f /etc/tcpmd5.conf

const int on = 1;
if (setsockopt(s, IPPROTO_TCP, TCP_MD5SIG, &on, sizeof(on)) == -1) {
  fprintf(stderr, "Failed to setsockopt(): %s\n", strerror(errno));
  exit(1);
}

Netcat with the -S option does the latter, but the password still needs to be set in the config.

So on OpenBSD it doesn’t really mesh well with adding TCP MD5 to any and all programs. Even if you set up the Security Association inside your program instead of in your config it seems more a state of the system than a state of your connection. Which makes some sense with BGP, but almost nothing else.

I’ve not fully explored how feasible it is to add the SA from the program itself yet, due to lack of time. I’ll get back to it.

Patches

I’ve patched OpenSSH to enable TCP MD5. Works great. Right now it just uses a static password, but that still protects you against internet-wide spread port scans that don’t know what you’re running on port 2222.

$ sudo tcpdump -M openssh -nlpi lo port 2222
[…]
16:04:12.212566 IP6 ::1.2222 > ::1.43934: Flags [P.], seq 1828532:1828704, ack 1117, win 179, options [nop,nop,md5 valid], length 172
[…]

(note the md5 valid)

This sure is better than portknocking. The only problem is that it doesn’t work through IPv4 NAT. So just use IPv6.

The future

It would be awesome if everyone added TCP MD5 support for everything. Imagine how hard it would be to portscan if every port has a password. Even if it’s a simple password, or an organization-wide password, that still makes it harder for hackers to create databases of all listening ports over the whole internet, and what’s behind them.

Instead of “is this port open? Yes? Then let’s check banners and see what that is” it’ll be “Is there an SSH running on this port? No? Then maybe a mysql? No? Then maybe…”. The problem becomes NP instead of P.

Update: I’ve added Linux TCP MD5 support to openbsd-netcat and got it upstreamed, so if you want to play with it then it’s not available. I also made tcpauth to shield SSH, though it’s general should be usable for much more.