As some of you know, we use a DNS based load balancer in our VPN setup.
There's the "Global random" option in our Windows widget, and a set of our OpenVPN configs with "Balancer" in the file name.
Both of those use the host name balancer.cstorm.is (or 3 other domains).
Before the upgrade, we used windows-balancer.cstorm.pw and linux-balancer.cstorm.pw (plus 3 other domains).
In both cases, the balancer host name would use a simple form of round-robin DNS so that a single VPN server isn't always chosen, making things more evenly distributed.
The problem with this setup is that it doesn't offer much scalability for whenever new IPs are added, especially to existing servers. If a server has more than one IP, putting those IPs in the DNS balancer will cause VPN clients to connect to that server more than others. The more IPs on a server, the higher the probability the client would connect to that server.
In 2017, someone requested that we get more US/CA IPs, so we did. We also purchased a bunch for some of the non-US/CA servers. For the reasons described in the last paragraph, we couldn't simply add all of those new IPs to the DNS balancer, since some servers have more IPs than others. At the same time, we also didn't want people using the balancer to miss out on these new IPs.
The solution we came up with was to use an iptables based load balancer on top of the DNS based load balancer. The probability option of the iptables statistic module seemed perfect for this.
While testing this solution, we noticed that a lot of tutorials out there about this iptables module are incorrect. They don't take into account that iptables rules are processed sequentially. If one rule doesn't apply to the packet, then the rule is discarded and the next rule is processed.
In most tutorials, they use something similar to:
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -m statistic --mode random --probability 0.2 -j DNAT --to-destination 5.254.96.242:443
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -m statistic --mode random --probability 0.2 -j DNAT --to-destination 5.254.96.244:443
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -m statistic --mode random --probability 0.2 -j DNAT --to-destination 5.254.96.246:443
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -m statistic --mode random --probability 0.2 -j DNAT --to-destination 5.254.96.248:443
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -m statistic --mode random --probability 0.2 -j DNAT --to-destination 5.254.96.250:443
At first glance, these five rules would appear to mean "if UDP packet received on port 443 of 5.254.96.226, then there's a 20% chance of forwarding it to port 443 of either 5.254.96.242, 5.254.96.244, 5.254.96.246, 5.254.96.248, or 5.254.96.250". But that's not how these rules would be interpreted. When the packet reaches the first rule, it has a 1 in 5 chance of going to 5.254.96.244. 4 out of 5 times, it would traverse to the next rule. But since the first rule is now discarded, there are now 4 rules. The second rule is still applying the 0.2 (20%) probability against an outcome that has a 25% (1 in 4) probability. If that rule also doesn't match, then it's discarded and the packet would move on to the third rule. At the third rule, there's now a 1 in 3 chance of matching, but 0.2 probability is still being used. If we make it to the 4th rule, there's only two rules left, so a 50% chance of it matching, while still applying the 20% probability. If we make it all the way to the last rule, then we're left with a 20% chance of it not matching anything.
The correct way to do it is:
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -m statistic --mode random --probability 0.2 -j DNAT --to-destination 5.254.96.242:443
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -m statistic --mode random --probability 0.25 -j DNAT --to-destination 5.254.96.244:443
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -m statistic --mode random --probability 0.33 -j DNAT --to-destination 5.254.96.246:443
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -m statistic --mode random --probability 0.5 -j DNAT --to-destination 5.254.96.248:443
iptables -t nat -A PREROUTING -p udp -d 5.254.96.226 --dport 443 -j DNAT --to-destination 5.254.96.250:443
I.e., increasing the probability according the number of rules that would be left if the previous rule were discarded. Doing it this way ensures that if the packet ever makes it to the last rule, there's a 0% chance of it not matching a rule.
The whole point of all of this is that it allows us to set aside a single IP on servers that have a lot of IPs (5.254.96.226 in the above example), and that single IP can be placed in the DNS balancer instead of all the server's IPs. That gives people using the DNS balancer a fair chance of being able to use all the available IPs, without us having to put all of those IPs in the DNS balancer.
So if you're wondering how you were assigned an IP different than the one in the balancer you connected to, or how we can advertise "600+ available IPs" when the balancer only has 27, that's how.