Using Mosh With a Socks Proxy

If you haven’t used Mosh, you’ve been living in a cave. Seriously. You don’t know what you’re missing.

As part of my grand experiment to live entirely on mobile data (2GB at LTE speeds, then throttled down to 3G speeds for the remainder of each month), I need robust and responsive SSH connections. Mosh supports intelligent local echo, roaming, and intermittent connectivity; making it ideal for use on mobile networks. It’s obvious Mosh is nearly perfect for my needs.

There’s still one issue. fail2ban is for the birds. SSH access should be restricted by a firewall that allows a whitelist only. I implement this policy whenever possible. As awesome as Terraform is, editing and running plans to allow my current IP address every time I want to ssh into a box is a pain in the ass.

The first step in solving this was to set up Dante, a SOCKS server, on a bastion host. In doing so, I realized that adding a single IP address to firewalls is not only less of an operational burden, but it also enhances security. Because of CGN, allowing a carrier’s IP means potentially thousands of other customers on the same carrier can access the same ports you can. Having an exclusive IP address to proxy out of decreases risk. However - for this to work - UDP ports 60000-61000 (Mosh ports) must be allowed from any IP.

Since I am already using WireGuard, it made sense to ensure that the SOCKS server can only be accessed over the VPN. WireGuard so far has been much more reliable for me than other VPN protocols over mobile networks (pay attention to MTU). It seamlessly roams, just like Mosh. However, SSH connections don’t seem to benefit from this, and die just like they would normally when the connection gets spotty.

Adding the following to my ~/.ssh/config did the trick of forwarding all SSH (and Git) through the proxy:

Host !192.168.128.1 *
	ProxyCommand nc -x 192.168.128.1:1080 %h %p

Enter Mosh

Mosh uses SSH to connect to the remote host and start an unprivileged mosh-server process:

[~] % mosh-server


MOSH CONNECT 60001 ECvWuq87+f2bcM0V3nEHaw

mosh-server (mosh 1.3.2) [build mosh-1.3.2]
Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 15441]

mosh-server gives the client a UDP port number and encryption key so it can connect. Because Mosh roams, the IP address that hits the port doesn’t matter. That means we can proxy our SSH connection to the server and then Mosh can connect from our actual IP. Sounds easy, right?

At the time of this writing, Mosh has an experimental option --experimental-remote-ip. The manpage states “Options named --experimental- are subject to change or removal in future versions of Mosh; their design or function is not yet final.” The default is proxy, which uses SSH’s --ssh-proxy-command. This effectively ignores the ProxyCommand option I set previously.

What if I try this magic?

[~] % mosh --experimental-remote-ip=remote <host>

Tada!

Further explanation of how this works comes from the manpage: “With remote, the server’s SSH_CONNECTION environment variable is used. This is useful for environments where ssh forwarding is used, or the --ssh-proxy-command option is used for other purposes.”

That’s great! But what happens if the Mosh developers decide to change or remove it? While it has been around since 2016, I’m not sure if that’s a good or bad thing.

In any case, here is another way to do it with a shell function:

mosh () {
	MOSH_CONNECT=$(ssh "${1}" mosh-server | grep 'MOSH CONNECT') 
	MOSH_KEY=$(echo "${MOSH_CONNECT}" | awk '{print $4}') mosh-client "${1}" $(echo "${MOSH_CONNECT}" | awk '{print $3}')
}

What this does more or less is manually sets up the Mosh session by first connecting over SSH through the proxy to run mosh-server on the remote host, capturing the parameters it needs, and then running mosh-client.

Or simply create the following alias:

alias mosh='mosh --experimental-remote-ip=remote'