Introduction

I do happen to travel from time to time and in order to stay online I have to use insecure public wifi. Everyone who understands a bit about network security knows the risks of connecting to unsafe networks and how dangerous they could be.

One of the most common solutions to this problem, including mine, is a VPN service. That’s all great, however, more and more network administrators put effort into blocking VPN traffic.

That annoys the heck out of me so in this post I’ll show you how I’ve essentially bypassed any VPN filtering with some simple tweaks.

Prerequisites

Now before reading this blogpost, I recommend getting yourself familiar with basic OpenVPN as well as systemd concepts.

My setup consists of a OpenVPN instance so whenever I say “vpn” I refer to OpenVPN entity.

Let’s get started!

How is OpenVPN traffic filtered?

Having a quick look at the network traffic, we can easily spot the OpenVPN traffic:

The image shows the beginning of the communication - firstly the DNS query, followed by the OpenVPN TLS handshake and finishing with the first packets of encrypted data sent over the secure channel.

By default, OpenVPN runs on port 1194/UDP so filtering it can be as simple as

# iptables -A FORWARD -p udp --destination-port 1194 -j DROP

Nowadays, network admins would probably use a more GUI-like user interface to do that, but that’s essentially what it boils down to.

Port 1194 is what Wireshark recognises as OpenVPN traffic as well:

What can we do about it?

Whenever you connect to a secure website (including my blog) you are essentially sending HTTP requests over an encrypted channel that is set up beforehand. The SSL/TLS protocols (see difference between SSL and TLS) lie at the heart of web security these days and there are plenty of other protocols that rely on SSL/TLS to securely transfer data around. Once this channel is initialized, the outside world cannot see any data that travels inside it.

OpenVPN also uses TLS for its encrypted channel. The TLS handshake is standardized and does not depend in any way on the protocol it carries.

So could we run the OpenVPN service on port 443/tcp to mask is as regular HTTPS traffic? Absolutely! Blocking 443/TCP would mean breaking the internet for the common user so no one will ever do it.

The issue of running multiple OpenVPN instances

A funny thing is that in all the forums I read, everyone’ opinion was that you can run multiple instances of OpenVPN simultaneously, but no one gave any guidelines on how exactly to do that. From the scarce information that I found, 2 things became clear to me:

I really didn’t want to go the bridging way so I kept on looking for an alternative. What’s more I didn’t want to spin up an entirely separate instance - it was a must to reuse my config files and client certificates.

systemd hidden gems

I looked for a more Linux-y way of solving this issue so I had a think - OpenVPN is just a daemon ran by systemd. By default it has the following config:

# /lib/systemd/system/openvpn.service

# This service is actually a systemd target,

# but we are using a service since targets cannot be reloaded.

[Unit]
Description=OpenVPN service
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecReload=/bin/true
WorkingDirectory=/etc/openvpn

[Install]
WantedBy=multi-user.target

I had a second look at the directory where the service file lives (/lib/systemd/system/) and I noticed there are several openvpn related service files:

root@vpn:/lib/systemd/system# ls openvpn
openvpn-client@.service openvpn-server@.service openvpn.service openvpn@.service
root@vpn:/lib/systemd/system# ls openvpn

When managing the OpenVPN service I’ve always used the openvpn unit file, but what’s that fancy openvpn-server@.service thingy?

That “@” looks dodgy, let’s see what it stands for in the unit file name.

The answer lies in section 5 of the systemd manpage (man 5 systemd.service)

Table 1. Special executable prefixes
┌───────┬───────────────────────────────────────────┐
│Prefix │ Effect │
├───────┼───────────────────────────────────────────┤
│"@" │ If the executable path is prefixed with │
│ │ "@", the second specified token will be │
│ │ passed as "argv[0]" to the executed │
│ │ process (instead of the actual filename), │
│ │ followed by the further arguments │
│ │ specified. │
├───────┼───────────────────────────────────────────┤

Woah! systemd can pass arguments to the daemon executables it runs, how awesome is that!

Having a look at the openvpn-server@.service file we can see that it’s more complicated that the first one:

 1# /lib/systemd/system/openvpn-server@.service
 2
 3[Unit]
 4Description=OpenVPN service for %I
 5After=syslog.target network-online.target
 6Wants=network-online.target
 7Documentation=man:openvpn(8)
 8Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
 9Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO
10
11[Service]
12Type=notify
13PrivateTmp=true
14WorkingDirectory=/etc/openvpn/server
15ExecStart=/usr/sbin/openvpn --status %t/openvpn-server/status-%i.log --status-version 2 --suppress-timestamps --config %i.conf
16CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE
17LimitNPROC=10
18DeviceAllow=/dev/null rw
19DeviceAllow=/dev/net/tun rw
20ProtectSystem=true
21ProtectHome=true
22KillMode=process
23RestartSec=5s
24Restart=on-failure
25
26[Install]
27WantedBy=multi-user.target

I’m interested in the ExecStart directive as this is what is being executed as a ‘service’. Obviously, this is the OpenVPN daemon being started and I’m keen on learning what this ‘%i’ stands for since the config I want it to use is referenced as ‘%i.conf’ (at the end of line 15)

Having a look at man 5 systemd.unit shows a neat table that explains is well:

Table 4. Specifiers available in unit files
┌──────────┬────────────────────────────┬─────────────────────────────┐
│Specifier │ Meaning │ Details │
├──────────┼────────────────────────────┼─────────────────────────────┤
│"%i" │ Instance name │ For instantiated units this │
│ │ │ is the string between the │
│ │ │ first "@" character and the │
│ │ │ type suffix. Empty for │
│ │ │ non-instantiated units. │
├──────────┼────────────────────────────┼─────────────────────────────┤

Aha! So I could start a openvpn-server@some_name.service and that would start an OpenVPN instance using some_name.conf as a config file.

By default, there is no server directory in /etc/openvpn so after we make that we should get our hands on creating the new tcp config file:

# mkdir /etc/openvpn/server && cp /etc/openvpn/openvpn.conf /etc/openvpn/server/tcp.conf

We want to have the same config, to make use of the same client certificates and what not, but run on port 443/tcp. So the config has to be mostly the same with only a few changes :

Let me explain why each of the changes is needed:

The rest of the config is the same. So let’s start the second instance and keep our fingers crossed:

# systemctl start openvpn-server@tcp

That seemed to have done the trick since the vpn instance was up and running.

Then I quickly crafted a client certificate that would be an exact copy of an existing one, with the exception of the port and the ip address.

Now if you’ve read my post about my home network topology, you know that the only port I have on my disposal on the site where I’m hosting my vpn server is 1194.

However, I’d like to run another instance of OpenVPN that listens on another port. So I’ve hit a wall there …

… or have I?

OpenWRT to the rescue

Running OpenVPN over tcp hurts performance significantly due to the way of how tcp works. I’m setting up this instance not to be fast, but to be stealthy so I am ready to sacrifice a little bit of performance. Now given that I desperately need hole through which I can smuggle my service and that performance is not a priority, perhaps I could make it run via my home router as I’ve done previously with this site.

This would mean running a OpenVPN instance inside another OpenVPN instance.

The meat of this hack

Let’s step back and see what that means. Hopefully this picture clarifies things a bit:

Firstly, UDP clients are not affected in any way. My home router is one and I use its udp vpn connection to send tcp vpn traffic from the client on the left. When that nested conenction reaches the server, the udp one is stripped away and then the tcp connection afterwards so in the traffic logs on the server you could tell both clients appart even though one carries the traffic of the other.

Let’s see if this even works:

╭─viktor@yuhu 11:31:25 ~/
╰─\$ sudo openvpn viktor-tcp.ovpn
Tue Jan 08 19:09:02 2019 OpenVPN 2.4.6 x86_64-redhat-linux-gnu [SSL (OpenSSL)][lzo] [LZ4][epoll] [PKCS11][mh/pktinfo]
[AEAD] built on Oct 6 2018
Tue Jan 08 19:09:02 2019 library versions: OpenSSL 1.1.1 FIPS 11 Sep 2018, LZO 2.08
Tue Jan 08 19:09:02 2019 TCP/UDP: Preserving recently used remote address: [AF_INET]213.191.184.70:443
Tue Jan 08 19:09:02 2019 Attempting to establish TCP connection with [AF_INET]213.191.184.70:443 [nonblock]
Tue Jan 08 19:09:03 2019 TCP connection established with [AF_INET]213.191.184.70:443
Tue Jan 08 19:09:03 2019 TCP_CLIENT link local: (not bound)
Tue Jan 08 19:09:03 2019 TCP_CLIENT link remote: [AF_INET]213.191.184.70:443
Tue Jan 08 19:09:04 2019 [vpn.samitor.com] Peer Connection Initiated with [AF_INET]213.191.184.70:443
Tue Jan 08 19:09:05 2019 Options error: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:1:
block-outside-dns (2.4.6)
Tue Jan 08 19:09:05 2019 TUN/TAP device tun0 opened
Tue Jan 08 19:09:05 2019 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Tue Jan 08 19:09:05 2019 /sbin/ip link set dev tun0 up mtu 1500
Tue Jan 08 19:09:05 2019 /sbin/ip addr add dev tun0 local 10.3.2.1 peer 10.3.2.2
Tue Jan 08 19:09:05 2019 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to
prevent this
Tue Jan 08 19:09:05 2019 Initialization Sequence Completed
So we should be done by now right?

Ehm, not quite:

╭─viktor@yuhu 19:09:20 ~/
╰─\$ ping 192.168.254.1
PING 192.168.254.1 (192.168.254.1) 56(84) bytes of data.
^C
--- 192.168.254.1 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 158ms

There is neither ping on any of the machines on the internal network, nor access to public addresses.

Is it the firewall or the routing?

So the OpenVPNpn client log indicates that a connection with the vpn server on the other side has been made successfully, but there is no internet access. Now there are a few probable reasons for that:

Putting the debugging hat on

The good news is that I don’t have to debug the OpenVPN thingy but instead a networking issue. Firing up tcpdump everywhere yeilded an odd result - the ping is being received by the vpn vm, however, nothing is sent back.

So it’s probably iptables that’s stopping my traffic isn’t it? Let’s stop it for a moment: 

# iptables -P {INPUT,OUTPUT,FORWARD} ACCEPT

Nope, still no ping back.

Then, surely, it must be an issue with the routing table. The weird thing about the whole thing was that I could cleary see the echo requests on tcpdump, but could not see any echo replies - if it was a routing error I would see the echo replies going out from a wrong interface, but I couldn’t see any!

This took a good couple of days to figure out.

OpenVPN routing quirks

I found this nice blog post from 2014 that explained how to do the entire thing - run a tcp and an udp instances of OpenVPN with the same config and without using a tap interface.

I had done most of the stuff the guy explained, I’ve just missed the last part - the routing. There is this simple script that I’d missed. It takes care of the routing whenever a client connects and disconnects. Basically, each time a client (dis)connects a static /32 route is being added /removed that would tell the kernel to route the client via the according interface - if the client connected via tcp aka tun1 - then route him via that interface. Otherwise route the client via the default tun0 interface:

#!/bin/bash

# /etc/openvpn/scripts/learn-address.sh

##

# learn-address script which allow

# OpenVPN to run on both TCP and UDP

# with the same range of address on both

# protocol.

#

# tgouverneur -- 2014

##

if [ $# -lt 2 ]; then
exit 0;
fi
action=$1;
addr=$2;

case ${action} in
        add|update)
                echo "[-] ${addr} logged in to ${dev}" >> /etc/openvpn/server/tcp-vpn.log
                /usr/bin/sudo /sbin/ip ro del ${addr}/32
/usr/bin/sudo /sbin/ip ro add ${addr}/32 dev ${dev};
;;
delete)
echo "[-] Deleting addr ${addr} -> ${dev}" >> /etc/openvpn/server/tcp-vpn.log
/usr/bin/sudo /sbin/ip ro del \${addr}/32
;;
\*)
;;
esac

exit 0;

Neat right?

We want this script to be executed each time a client (dis)connects.

There is a special directive in the OpenVPN server config for that (have to put it in both configs):

# /etc/openvpn/openvpn.conf

# /etc/openvpn/server/tcp.conf

--- snip ---
learn-address /etc/openvpn/scripts/learn-address.sh

PS: Remember to add your vpn daemon user to sudoers and allow him to execute /sbin/ip as root - it needs to be able to change system routes:

# /etc/sudoers

--- snip ---
vpn ALL=(ALL:ALL) NOPASSWD: /sbin/ip

Finally it should be good to go!

Not yet…

Alas even with this tweak there was no connection going out of the vpn server to the clients…

This time I decided to have a look at the server logs - something must have gone wrong server-side!

 1root@vpn:/etc/openvpn/server# journalctl -f -u openvpn-server@tcp -b
 2-- Logs begin at Fri 2018-09-21 20:18:43 EEST. --
 3Jan 10 21:52:51 vpn openvpn[28814]: Socket Buffers: R=[87380->87380] S=[16384->16384]
 4Jan 10 21:52:51 vpn openvpn[28814]: Listening for incoming TCP connection on [AF_INET][undef]:443
 5--- snip ---
 6Jan 10 21:52:57 vpn openvpn[28814]: 10.3.2.9:13131 [sgs7] Peer Connection Initiated with [AF_INET]10.3.2.9:13131
 7Jan 10 21:52:57 vpn openvpn[28814]: sgs7/10.3.2.9:13131 OPTIONS IMPORT: reading client specific options from: /etc/openvpn/ccd/sgs7
 8Jan 10 21:52:57 vpn openvpn[28814]: sgs7/10.3.2.9:13131 OPTIONS IMPORT: reading client specific options from: /tmp/openvpn_cc_2550293368432e205dcedc71562a78a3.tmp
 9Jan 10 21:52:57 vpn sudo[28844]: vpn : TTY=unknown ; PWD=/etc/openvpn/server ; USER=root ; COMMAND=/sbin/ip ro del 10.3.2.5/32
10Jan 10 21:52:57 vpn openvpn[28814]: sudo: unable to send audit message
11Jan 10 21:52:57 vpn sudo[28844]: PAM audit_log_acct_message() failed: Operation not permitted
12Jan 10 21:52:57 vpn sudo[28844]: pam_unix(sudo:session): session opened for user root by (uid=0)
13Jan 10 21:52:57 vpn sudo[28844]: vpn : pam_open_session: System error ; TTY=unknown ; PWD=/etc/openvpn/server ; USER=root ; COMMAND=/sbin/ip ro del 10.3.2.5/32
14Jan 10 21:52:57 vpn openvpn[28814]: sudo: pam_open_session: System error
15Jan 10 21:52:57 vpn openvpn[28814]: sudo: policy plugin failed session initialization
16Jan 10 21:52:57 vpn sudo[28845]: vpn : TTY=unknown ; PWD=/etc/openvpn/server ; USER=root ; COMMAND=/sbin/ip ro add 10.3.2.5/32 dev tun1
17Jan 10 21:52:57 vpn sudo[28845]: PAM audit_log_acct_message() failed: Operation not permitted
18Jan 10 21:52:57 vpn openvpn[28814]: sudo: unable to send audit message
19Jan 10 21:52:57 vpn sudo[28845]: pam_unix(sudo:session): session opened for user root by (uid=0)
20Jan 10 21:52:57 vpn openvpn[28814]: sudo: pam_open_session: System error
21Jan 10 21:52:57 vpn openvpn[28814]: sudo: policy plugin failed session initialization
22Jan 10 21:52:57 vpn sudo[28845]: vpn : pam_open_session: System error ; TTY=unknown ; PWD=/etc/openvpn/server ; USER=root ; COMMAND=/sbin/ip ro add 10.3.2.5/32 dev tun1
23Jan 10 21:52:57 vpn openvpn[28814]: sgs7/10.3.2.9:13131 MULTI: Learn: 10.3.2.5 -> sgs7/10.3.2.9:13131
24Jan 10 21:52:57 vpn openvpn[28814]: sgs7/10.3.2.9:13131 MULTI: primary virtual IP for sgs7/10.3.2.9:13131: 10.3.2.5
25--- snip ---

From this log snippet I could tell that the daemon has started listening successfully (line 4), client has connected without any problems either (line 6) but then when running the learn-address script there is this “Operation not permitted” on multiple instances.

You could cleary see that root is executing the /sbin/ip command (so /etc/sudoers is read properly) but then why does it still get permission denied?

Debugging this issue was a hard one.

Eventually I found the answer in a debian bug report logs on openvpn.

The issue was in the .service file of the openvpn tcp instance. The CapabilityBoundingSet directive controls what capabilities the started process will run with.

Apparently the default set does not include the capability for changing routes (or some other, I haven’t figure it out).

The easiest way to fix it is just to comment out that line:

# /lib/systemd/system/openvpn-server@.service

--- snip ---
[Service]
Type=notify
PrivateTmp=true
WorkingDirectory=/etc/openvpn/server
ExecStart=/usr/sbin/openvpn --status %t/openvpn-server/status-%i.log --status-version 2 --suppress-timestamps --config %i.conf
#CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE
LimitNPROC=10
DeviceAllow=/dev/null rw
DeviceAllow=/dev/net/tun rw
ProtectSystem=true
ProtectHome=true
KillMode=process
RestartSec=5s
Restart=on-failure
--- snip ---

Obviously the best way to fix it, is to find out what capability it is missing and add it to the list. In my case it’s not mission critical given that I run my default OpenVPN with default capabilities, I don’t see that as an issue for the tcp instance.

After restarting the services I finally got it working properly:

root@vpn:/etc/openvpn/server# journalctl -f -u openvpn-server@tcp -b
--- snip ---
Jan 10 21:56:14 vpn openvpn[29039]: 10.3.2.9:28118 [sgs7] Peer Connection Initiated with [AF_INET]10.3.2.9:28118
Jan 10 21:56:14 vpn openvpn[29039]: sgs7/10.3.2.9:28118 OPTIONS IMPORT: reading client specific options from: /etc/openvpn/ccd/sgs7
Jan 10 21:56:14 vpn openvpn[29039]: sgs7/10.3.2.9:28118 OPTIONS IMPORT: reading client specific options from: /tmp/openvpn_cc_1fd11ab8ec0f2dad7ebbc6eb4de16a8a.tmp
Jan 10 21:56:14 vpn sudo[29072]: vpn : TTY=unknown ; PWD=/etc/openvpn/server ; USER=root ; COMMAND=/sbin/ip ro del 10.3.2.5/32
Jan 10 21:56:14 vpn sudo[29072]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 10 21:56:14 vpn openvpn[29039]: RTNETLINK answers: No such process
Jan 10 21:56:14 vpn sudo[29072]: pam_unix(sudo:session): session closed for user root
Jan 10 21:56:14 vpn sudo[29074]: vpn : TTY=unknown ; PWD=/etc/openvpn/server ; USER=root ; COMMAND=/sbin/ip ro add 10.3.2.5/32 dev tun1
Jan 10 21:56:14 vpn sudo[29074]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 10 21:56:14 vpn sudo[29074]: pam_unix(sudo:session): session closed for user root
Jan 10 21:56:14 vpn openvpn[29039]: sgs7/10.3.2.9:28118 MULTI: Learn: 10.3.2.5 -> sgs7/10.3.2.9:28118
Jan 10 21:56:14 vpn openvpn[29039]: sgs7/10.3.2.9:28118 MULTI: primary virtual IP for sgs7/10.3.2.9:28118: 10.3.2.5
--- snip ---

.. with the appropriate route being set for tcp:

10.3.2.5 dev tun0 scope link

and udp

10.3.2.5 dev tun1 scope link

respectively.

Future work

There’s a problem that I haven’t solved yet - proxying the tcp vpn instance on port 443 at my home router raises an issue. The router at home runs a haproxy instance on 443/tcp (read more here) to forward traffic to my webserver and serve content over HTTPS. I can’t really tell which TLS traffic is for the website and which for the vpn - that was the point in the first place wasn’t it.

Currently as I see it, I’ll probably need another public IP but we shall see. Or since I have the private key for the TLS certificate I might be able to make haproxy act as a router but I need to do more research on the topic.

Conclusion

Running a second OpenVPN instance on the same machine seemed as a pretty easy task, however, there are some caveats that might get in your way. I learnt a lot about systemd, how it operates and what awesome stuff one could do with it. Moreover, it was fun debugging the networking issues after a client has connected and last but not least I’ve never had to debug capabilities issues so that’s good experience as well.

Getting everything to work brought me great satisfaction and a feeling of a time productively spent.

Resources

OpenVPN over TCP or UDP - pros and cons

GitHub issue hinting that multiple OpenVPN instances are possible, but not without further tinkering

OpenVPN source file that searches for config files

OpenVPN tcp config from docs

OpenVPN forums thread that says OpenVPN can be run over tcp but doesn’t specify how

Thomas Gouverneur’s blog bost on the topic that included the ’learn-address’ script

Debian Bug report logs where I found the capabilities issue for systemd

Digitalocean guide on systemd and how to manage services in linux