02
Netcode
Explaining how fighting games use delay-based and rollback netcode
October 16, 2019
Infil

The Technical Side of Rollback Netcode

On the previous page, we talked about how rollback works and why it’s good at hiding bad connections. But now, we need to talk about how rollback netcode gets made, what a game that implements rollback must be able to do, and why some games have trouble creating good rollback.

As much as fighting game fans wished creating good netcode was simple, implementing rollback takes work and requires commitment and a strong belief from the studio that high quality online play is a priority.


How a Game Must Be Built

As a brief reminder of how rollback works, when remote inputs are unknown, the game simulates forward anyway with predicted inputs. If those predictions are incorrect when the real input is received, the game must roll back by loading a previous game state and simulating several in-game logic frames.

Therefore, the primary requirement of a rollback game is that it can save and load different game states and simulate game logic in the background, and it must be able to do this all very rapidly. There is already a lot to unpack with just this seemingly simple statement.

The act of converting objects in computer memory to a format that can be saved and loaded is called serialization, and the game state for every single frame must be serialized and saved in case we need to roll back to it later. This means your game state serialization, which can be a time-consuming operation depending on the data, must be lightning fast and very well optimized. Depending on how the game has been built, you may even need to change how the data is stored and how the various systems of your game use it to gain the necessary performance. If you build your game with rollback in mind from the start, it can lessen the burden, but adding rollback to an existing game can mean a lot of system-wide changes to the very core of your game. Michael Stallone, engineer at NetherRealm Studios, estimates that it took his team 2 man-years to build and optimize serialization systems in Mortal Kombat X, when they retrofitted the game with rollback netcode.

If your game needs to roll back a large number of frames, once the previous state has been deserialized and loaded, the game must re-simulate all these frames back to the present (and then also serialize all of them!), and this must be done quickly and in the background. This means your game logic must be independent from everything else in your gameplay loop; you need to be able to run many frames of logic without rendering to the screen, or waiting on the network, or using any number of other systems that normally operate every frame. Depending on the engine you chose, or how much customization to the engine-provided game loop systems you’ve done, separating and turning off these subsystems may be surprisingly hard.

Because all these things need to be performed in 1 game frame, in addition to all your normal game loop operations, optimization in general is another key concern. Perhaps those costly CPU cycles that your fancy particle system or cloth simulation uses are now putting you over your performance budget. Or maybe you’ve been playing fast and loose with your console’s power, because the need to be extremely efficient wasn’t there when you initially built the game. Stallone says that MKX, under its old delay-based solution, took only an average of 10ms of its 16ms budget to run a complete frame due to the extra horsepower of the PS4 and Xbox One. When initially switching to a nonoptimal, naive rollback solution, he says the cost to run a frame more than tripled to 32ms, which forced his team to re-evaluate almost every single in-game system and look for performance gains.

A small sampling of the things NetherRealm Studios needed to optimize to get MKX running fast enough to use rollback. It all sounds very complicated. (source)

Note that any deterministic game can use rollback netcode as long as it is able to save and load game states and perform multiple frames of game logic quickly enough to fit inside its game loop. To dispel a common myth about rollback, there are no limitations based on the genre of game. Rollback could be (and has been) used for 2D or 3D fighting games alike, including Street Fighter III 3rd Strike: Online Edition, Killer Instinct, Brawlhalla, and For Honor. It’s also used outside of fighting games; Iron Galaxy incorporated rollback netcode into the 4-player action game Dungeons & Dragons: Chronicles of Mystara. Even games like Rocket League incorporate rollback to make online play just as responsive as offline play.


Keeping the World in Sync

All netcode solutions, whether driven by input delay or rollbacks, must keep both systems running completely in sync. This means both systems are showing each player the same frame at the same time, or as close to this as possible. Even though most fighting games try to lock the framerate at 60fps, it is not simply enough to sync at the beginning of a match and assume the players will remain in sync for the whole fight. Many issues outside of networking, such as consoles overheating, computers performing operating system functions, or someone temporarily overloading their CPU, can cause performance drops which will put them out of sync with their opponent.

The consequences of not correcting this sync are catastrophic. Consider a rollback system where player A is being shown frame 20, but player B is being shown frame 23, across a network delay of 3 frames. Player B will send input for their frame 23 to player A. Because they are on frame 20, the network delay means they will receive the input on their own frame 23, which is just in time to execute it and will not have to roll back at all. Meanwhile, player A sends their input for frame 20 to player B. With the network delay, player B will receive this input on their own frame 26, which will repeatedly cause lots of very uncomfortable 6-frame rollbacks.

An example of what extremely one-sided rollback might look like for an unlucky player. The other side sees a much more normal version of the same match. Playing in these conditions is impossible.

Note that this rollback is entirely one-sided; only the player who is ahead of the other will experience the effects of the bad connection. Tony Cannon, in many ways the founder of modern rollback with his invention of GGPO, theorizes that this is one of the flaws of Street Fighter V’s rollback system, and proper clock syncing might go a long way to addressing SFV’s poorly received online play. Some fans have even run their own tests and come to similar conclusions. To help illustrate the point even further, Zinac shows the one-sided rollback effect in action using a very simple project designed to teach about the benefits of rollback.

Zinac's networking example has P1 controlling red and P2 controlling blue; he made it so P2's game is several frames ahead of P1 with no way to resync. When blue moves, P1, who is behind on the sync, sees the blue box move smoothly. But when red moves, P2, who is ahead on the sync, sees tons of huge rollbacks. (source)

Solving the problem is relatively straightforward. Most rollback systems will briefly pause the player who is ahead for 1 or 2 frames so the player who is behind can catch up. If the game keeps on top of this system and never lets the games drift very far apart, the correction will be quick and painless. Note that these pauses are not due to missing input or other networking difficulties; they are designed to fix the fact that different computers will run software at slightly different rates in unpredictable ways, and losing 1 frame every 10 seconds to maintain the sync is unnoticeable to even the most astute players.


What Exactly is GGPO?

If you spend any time around the fighting game community, you have undoubtedly heard about GGPO, the rollback library (that is now free and open-source!) written by EVO co-founder Tony Cannon. You’ll often hear “GGPO" and “rollback" used interchangeably among fighting game fans. GGPO’s first release in late 2006 was designed to test the concept of rollback with popular legacy fighting games running on emulators, and shortly after, Cannon released a version of the library that could be licensed and incorporated into modern games.

The main function of GGPO is to provide a robust solution for syncing machines and game states. For example, GGPO can tell your game when two machines are out of sync, and by how much. It can also keep track of the inputs from all connected parties and tell your game when to roll back, by how much, and what new inputs to apply during your rollbacks.

Because GGPO is well-tested and proven as a solution to handling these parts of the rollback framework, it is an excellent asset towards making games that want to use rollback netcode. However, the act of actually performing the rollbacks still falls on the developers. They have to make sure their game is performant, their game logic is properly split away from other parts of their game loop, and all edge cases are handled intelligently. GGPO simply tells them when and why to roll back, and what new inputs you should simulate your game with. It does not concern itself with the particular implementation of how the game achieves these goals, which makes it a flexible tool but not the complete suite of functionality needed for rollback to work. It also means that some games, including Killer Instinct, implement rollback netcode without directly using the GGPO library by coding their own in-house solution for tracking the game state and syncing the clients.

If you are further interested in the exact ways GGPO handles rollback and talks to games, I recommend reading this excellent article in Game Developer Magazine written by Tony Cannon. It has a bit more of a technical focus, but may interest you if you’re a programmer or curious on the exact things GGPO provides your game. You can also peruse the source code and read Tony’s own description of the theory behind rollback.


Be Careful of Edge Cases

As if rollback wasn’t complicated enough, it’s worth briefly mentioning just a few of the edge cases you will run into when you support a game loading up the past.

Creating and destroying objects becomes more difficult. Let’s pretend the local player throws a fireball, and the opponent holds down-back in front of the fireball, ready to block it. Rollback uses predicted inputs to play out the opponent blocking this fireball on the local player’s screen. The fireball is destroyed when it is blocked, and particle effects around the fireball are shown. But then remote inputs come in and it turns out the opponent did an invincible move through the fireball right before it was blocked, which means it was actually not destroyed and must continue to travel on the screen. In many games, recreating the fireball after it’s been destroyed is expensive and could lead to odd visual popping. A solution many rollback games take is to simply save objects for many frames after they’re destroyed, in case rollback needs to un-destroy them, which may require some changes to your game logic. Particle systems also require a lot of special attention.

Audio is also a particularly difficult problem in rollback games. Sounds that were played in the predicted version often have to be cut short and stopped in the rolled back version, and perhaps sounds that were never played on the local version must now be played several milliseconds in when the rollback changes the game state. This is especially problematic when you remember that game logic needs to be divorced from all other aspects of the game loop. In early versions of rollback in modern games, like Street Fighter x Tekken, there were horrific sound bugs where the sound would cut out for seconds at a time, or sound that was played would sharply pop or play out of sync. Tony Cannon gives some advice for how to handle audio in a rollback engine in his Game Developer Magazine article, specifically the “Separating Game State from Rendering" section.

There may also be times you never want to roll back. Stallone mentions how rollbacks are very undesirable during drastic camera shifting events, like brutalities in MKX or stage transitions in Injustice 2, which means their game must prevent those events from executing until the local player is 100% positive the event occurred and cannot be rolled back. This leads to extra engineering and debugging efforts across all the gameplay system teams, and therefore extra expense.

You also have to be careful to correctly copy the last known frame during your prediction algorithm. This includes what directions the remote player was holding, as well as what buttons they were holding (and not holding). If you mess up this prediction, you may cause a remote player to do something they never actually did, which will be incredibly disorienting when it gets rolled back.

The remote Cody player does EX Zonk Knuckle, a move which is triggered by holding and then releasing buttons, which is then somehow rolled back. My best guess for why this happens is because SFV did not correctly remember what buttons Cody was holding during the prediction phase, which made him do a button-release move. (source)


Is Rollback Worth the Effort?

While there’s no denying that adding rollback to a game takes effort and expense, I don’t want the takeaway from this section to feel like implementing rollback is too difficult to be feasible. The reality is somewhere in the middle between “copy and paste GGPO into your game in an afternoon" and “invest your entire budget on rollback systems".

It largely depends on when you decide to implement rollback. Adding rollback to a game after its gameplay systems have been built is often very challenging. When NetherRealm Studios decided to patch in rollback netcode to Mortal Kombat X, a game that had already shipped, it took approximately 8 man-years of time split between many programmers over 10 months. The very small studio behind Rivals of Aether spent a lot of development time over a few years exploring how to convert their finished game from delay-based to rollback, but eventually had to move on when it was proving too costly. (2022 update: Rivals of Aether now has rollback! Well done team!) On the other side of the spectrum, Mike Zaimont, lead programmer on Skullgirls, added GGPO to his game in about 2 weeks, early in development using his custom engine likely well-planned out for rollbacks. Therefore, deciding that rollback is a priority early in development will greatly reduce the amount of investment required.

Implementing rollback also gives you some added benefits in other areas of your game. For example, if your game has a robust method for saving and loading game states, you can use it to provide better training mode options for your players. And when you’ve implemented rollback once, you can reap the benefits of the knowledge in future games. After putting in the hard work for MKX, NetherRealm and its fans now enjoy smooth netplay in all their future titles that were built off modifications of the same engine, including Injustice 2 and Mortal Kombat 11.

But in reality, rollback netcode is far from the first worthwhile advancement in technology to be more work than the previous method. Animation techniques such as inverse kinematics are now standard in many games despite being a higher initial investment for many studios to implement effectively, and after years of working on the problem, developers now have better tools in place to improve it even further. The value for rollback is there, and developers should not be wary of prioritizing that benefit in their game. After all, is there anything more important for a modern fighting game than smooth, reliable, consistent online play that can connect communities worldwide?

Back to the blog index.

Fightin' Words