Thank you, Valve

EDIT: The configuration below has now been superceded by LANCache.

About twice a year I help set up and run a local LAN Party at a Rugby club on the outskirts of town. Compared to more widely known events, ours is tiny in comparison – typically 20-30 people – all squeezed into the club house.

One thing that a LAN party needs to be a success, particularly in the last few years, is a solid connection to the internet so that multi-player server lists can be retrieved, DRM systems can unlock, updates can download, and people can sign into Steam. Guests might also wish to have more general net access for the web, email, IM etc. One of the issues with smaller venues, however, is that they generally have no need for a fast, reliable internet service – the cheapest consumer-grade connection will normally suffice. In some cases, the geography of the less expensive locations – required for small parties – tends to impose technical limits on what services are available anyway. In our case, the clubhouse has an ADSL connection which syncs at about 2Mbit on a good (dry) day.

2Mbit shared between 30 users doesn’t split very well. In fact, 2Mbit for a single user would be quite a trial these days. One of the biggest problems is the amount of traffic created by game updates, which are typically a few hundred megs per game. For multiplayer games – the centre of a LAN event – everyone needs to be on the same version of the game as the server. A far as I can remember, Valve (producer of the many popular Source-based games) has released updates for their games the day before every LAN party event we’ve had in the past few years. That typically leads to a significant number of installs being out of sync with everyone else.

Let’s assume that 50% of installs need to update out of a total of 30, and the update is 200MB big; over a 2Mbit connection, on the assumption the connection is completely unused by other activities, each of the 15 clients will have 133Kbit/s or 17KB/s – the update for each person will take approx 3 hours 15 mins to download. That’s just for updating one of many games. If someone wants to join in with a game they don’t own, they can purchase it on Steam but it’s very unlikely that it will have finished downloading by the time the event is over.

There’s a lot of duplication going on when downloading game updates. Everyone downloads exactly the same files, but until earlier this year it was impossible for people to ‘share’ those downloads. Steam is the most popular game distribution platform for PCs, and for the most of its life it used a proprietary download protocol running over very non-standard TCP & UDP ports. Much to the applause of us network admins, Valve has now changed this to use the SteamPipe system which is based entirely on HTTP. By itself, this doesn’t change very much, but with some tricks it is the breakthrough that should ease the pain of future parties.

With their huge (thousands of guests) i-series LAN events, Multiplay had the same problem as us but on a much larger scale. They were one of the first to take advantage of the new distribution platform, using a combination of DNS overrides and a custom HTTP proxy using Nginx. They very kindly published their approach.

For the technique to work, there needs to be a DNS server which resolves queries for all of the machines that are to use the mechanism, and an HTTP proxy server that will store the received updates. Details aren’t specified on how to configure the DNS server, so it is assumed you already have one and just need to add the spoofing records to make sure SteamPipe traffic goes through the caching server.

If you find this method a little tricky, the same effect can be accomplished with a normal HTTP proxy server (lots of solutions out there for that) but the disadvantage of generic HTTP proxy servers is that they are designed to flush (clear) their caches to remove lesser-used or old content. Ideally we want to keep as much of the retrieved objects as we have space for, which Nginx allows us to do.

For our purposes, there is a slightly modified/enhanced version of Multiplay’s setup created by Astrolox. This one is designed for Ubuntu, which I’m more familiar with compared to FreeBSD. It also caches the general steam client content that can consume a lot of traffic, implements a per-user bandwidth limit and blocks the transmission of crash reports (which can be fairly large).

We may also be using a slightly different network structure to the one shown above. After a fair bit of looking, I’ve found a great router distribution – Endian Firewall – through which all requests are routed. This little gem contains transparent DNS proxying, which can be configured to redirect requests based on specific domains to another DNS server; in our case the requests for steampowered.com all go to our internal DNS server which spoofs addresses where necessary. It also includes transparent HTTP proxying, as well as the typical firewall and request filtering.

We are also looking at other options for connectivity, with satellite being our best bet so far – either by itself or combined with a landline-based service or 3/4G. For large downloads that don’t mind about latency, such as our game updates, satellite would be perfect. For now though, the cache of game updates should dramatically improve the quality of service at our LAN party events. In the update example given earlier, our 200MB download will now only need to happen once and will complete in 1/15th of the time at a little under 15 minutes.