In 2008, Riot built League of Legends’ desktop client on Adobe AIR, which used “an RTMP session-based networking protocol to talk to our servers” [1]. In 2016 its engineers wrote down why they threw it away, and the reason was not performance [1]. It was that the game could not be taken offline long enough to replace the thing players were looking through.

The data underneath it was in worse shape. Bill Clark described the game’s content as “a bunch of loose files sloshing around in a big bucket of a folder called DATA” [6]. Inside that folder, 977 spells carried the same line [6]:

MissileEffect=AnnieBasicAttack_mis.troy

Clark’s own parenthetical is “(certainly ignored)” [6]. Nine hundred and seventy seven spells pointing at one missile effect that no longer meant anything is not a bug report. It is what a data schema looks like when nothing enforces one, and it is the shape of the problem the whole live-operations stack had to solve: content that many people edit, shipped to clients that are always in use.

The wall

The obvious remedy is the one every desktop product gets: freeze the code, rewrite it, ship it in a release. That remedy assumes a release exists. The thing being replaced is the executable players are currently looking through, so there is no window in which it can be closed. Riot could not take the client down to swap it [1].

Two other constraints pointed the same way, and the second was organisational. The client had to be shippable by many feature teams rather than one. McVeigh’s post says it plainly: “The original codebase was structured around a small, highly cohesive team and didn’t give the required level of autonomy and independence as the number of features teams grew” [1]. That is a statement about team structure, not about rendering, and it explains why the replacement was assembled from services instead of written as one program.

And the server could not be stopped for a patch either, for the same structural reason. League has no fixed step to stop at: “League of Legends does not use a fixed-step server clock, but rather a measured frame delta time which we throttle to a target frame rate” [3]. Community sources repeat a 30 Hz tick rate for League; Riot’s determinism write-up describes the opposite, frame deltas “usually less than 0.033 seconds” [2], with client clocks nudged rather than snapped: “we very slowly, almost imperceptibly, speed up or slow down individual frames” [2].

The game is not lock-stepped either: “League does not use the lock-stepped, synchronized peer-to-peer networking common in sports, fighting, and traditional real-time strategy games” [2]. Where authority lives, Riot stated most clearly in its VALORANT netcode post, which is the cleanest published form of the principle: “the server must never trust a client’s view of the world” [4]. The patch pipeline obeys the same rule. Nothing on the player’s machine is allowed to be the source of truth, including the patcher.

Three moves

Replace the front end, keep a service under it. Riot did not repair AIR. “we chose to start again and build around Chromium Embedded Framework (CEF) as our front-end” [1]. Beneath the browser sat a C++ layer: “We called this microservices layer the ‘foundation.’” [1]. The design constraint shows up in the client’s lifecycle rather than its rendering: “when we minimized the client, we were able to kill the CEF UI (and all JavaScript VMs), leaving just the foundation”, and “The foundation only takes up around 20mb in memory” [1]. Twenty megabytes is the price of a client that can be torn down to a service and rebuilt around it.

Widen the version window, then chunk the content. The replacement patcher works against more than one prior build: “RADS uses the second option, and limits it to the previous 8 versions” [5]. A deploy that had taken five hours became this: “deploy a new version of League of Legends in less than a minute instead of the five hours it used to take” [5]. Patch 9.9 measured both patchers: “68 MB for RADS and 83 MB for the new patcher”, with the new one finishing in “less than 40 seconds, compared to over 8 minutes” [5]. The mechanism is chunking: “over 300,000 chunks, but we group them into less than 5,000 bundles”, and “Each version of League shares over 85% of the bundles with the previous version” [5]. Reliability moved with it: “The failure rate dropped from 2.2% to 0.3%” [5].

Move editable data into a daemon. The 977 spells became a service. “The foundation is RiotGameDataServer.exe … They all communicate with GDS via a JSON-based RPC API” [6]. What replaced the folder was a layer model that records per-field changes: “GDS simply marks the delta of each field that has been changed. The before and after values are saved to reconcile merge conflicts later” [6].

The numbers those moves produced:

QuestionAnswer
What replaced AIR?CEF front end over a C++ “foundation” [1]
Foundation memoryAbout 20 MB [1]
Versions RADS diffs againstThe previous 8 [5]
Chunks and bundlesOver 300,000 chunks into under 5,000 bundles [5]
Bundle overlap with the previous versionOver 85% [5]
Deploy timeFive hours to under a minute [5]
Patch 9.9 download and apply68 MB vs 83 MB, under 40 seconds vs over 8 minutes [5]
Patch failure rate2.2% to 0.3% [5]

The shadow

Every one of those moves installed a component that now has an uptime requirement of its own.

The foundation is the clearest case. The 20 MB is not a saving, it is a floor: a browser engine plus a native service on every player’s machine, where before there was a single AIR client [1]. The minimizing behaviour that makes it cheap is also the admission that the UI is disposable and the service is not, and the players are the ones who inherit the second process.

GDS is the same trade in a different place. The quote names a .exe, which means a process on the player’s machine, an RPC surface, and a merge path that has to keep working when two designers touch the same field. The layer model moved conflict resolution out of a folder of text files and into a tool that someone has to maintain, with promotion rules that are now part of shipping a patch. That cost lands on the designers first and on release engineering second.

The patcher inherited the most load. A pipeline that turns 300,000 chunks into bundles can make a deploy fast, but it is a build step between the designers and the players, and its output is the thing every client downloads. Bundle sharing at 85 percent means the interesting failures live in the other 15, in the bundles that changed, which is exactly where a bad rollout is hardest to see before it reaches everyone [5].

The bill is paid continuously. The deploy that took five hours now takes under a minute, and what runs in its place is a daemon, a chunk store, and a layer system with promotion rules, each with its own way to fail. The client that could not be closed was replaced in the end. What sits underneath it now cannot be stopped either.

References

[1] Andrew McVeigh, “Architecture of the League Client Update”, Riot Games. https://www.riotgames.com/en/news/architecture-league-client-update

[2] Riot Games, “Determinism in League of Legends: Unified Clock”, Riot Games. https://www.riotgames.com/en/news/determinism-league-legends-unified-clock

[3] Riot Games, “Determinism in League of Legends: Implementation”, Riot Games. https://www.riotgames.com/en/news/determinism-league-legends-implementation

[4] Riot Games, “Peeking into VALORANT’s Netcode”, Riot Games. https://www.riotgames.com/en/news/peeking-valorants-netcode

[5] Javier Blazquez, “Supercharging Data Delivery with the New League Patcher”, Riot Games. https://www.riotgames.com/en/news/supercharging-data-delivery-new-league-patcher

[6] Bill Clark, “Content Efficiency in Game Data on the Server”, Riot Games. https://www.riotgames.com/en/news/content-efficiency-game-data-server