Revisiting the ghost’s shell: proprioception in gaming

[Featured image from PS4Fans.net]

Anyone who has played a local-multiplayer game (multiple players sharing one screen) knows how messy the initial conditioning is. You wiggle your controller’s thumbstick, press a few buttons, determine which of the viewports shown is your own (if split-screen first-person) or which of the on-screen characters running around is yours (if third-person). And even then, at some point in the game massive explosions happen, or you need a toilet break, and when you’re finally back with full attention you have to re-spot your character all over again.

This can get really messy with games like Assault Android Cactus. It is not uncommon to mix up another player’s character for your own, especially when there is lots of on-screen movement.

Assault Android Cactus [Comicbuzz]
Assault Android Cactus [Comicbuzz]

There are even games that exploit this difficulty of matching intention to movement—proprioception in medical parlance. The player who recognises his character first is much more likely to win in that game.

Proprioception in the body #

In first-person or third-person shooters, single-player mode, there is no such difficulty. The viewport moves with your intended movements (presumably correlated with your actual movements through the controller).

This recognition of something as being part of your body, as something you can control with your mind, is a complex process, a mini-miracle in itself. Field sobriety tests check for the lapse of this sense in inebriated persons: an inability to touch one’s nose with one’s fingers with eyes closed, for instance.

The rubber hand illusion shows us one way to exploit, to hack this sense: in the spot where a subject expects their real hand to be, a rubber hand is placed (a trick using mirrors). Both the rubber hand and the real hand are stroked at the same time; the subject feels the real hand being stroked but sees the rubber hand being stroked. The result is an eerie sensation, “a convincing feeling of body ownership”.

What gamers do—recognising an on-screen character as being under their control—is one level removed from this phenomenon. After all, we are not trying to control those characters directly with electrical impulses from the brain. Our control over the on-screen characters is mediated by an interface, the game controller. But just consider for a moment the ease with which your mind enters the game again after your toilet break. Before you know which character is yours, you are completely helpless. Once you identify your character, ass-kicking resumes almost immediately (depending on how long your situational awareness takes to kick in). Wherefore this difference?

Proprioception and VR #

The Oculus Rift is a VR helmet that maps your viewport to your head position and orientation. That brings game control one step closer to the mind; your neck muscles directly control your viewport, with no controller interface to mediate. This has been demonstrated to work remarkably well with first-person shooters, but with third-person shooters the effect is likely much diminished. I would guess the proprioceptive effect is not as strong when viewport changes do not match muscle movements so closely.

When motion-control technology catches up, and gamers can control their character’s motions directly, the invisible miracle of proprioception will happen again. Animal tests on rhesus monkeys show that they have little difficulty adapting to new robot legs or simulations of hands, so long as the proprioceptive illusion is not broken. Testing on humans even shows that we have little difficulty adapting to a smaller virtual body, and a lower viewport.

But proprioception isn’t just for building up “muscle memory” around a new body part or virtual part. It is how we are able to walk in the dark, how we are able to reach for our phones or alarm clocks without looking, how we can scratch our own backs if we are flexible enough. I feel that this aspect of proprioception isn’t exploited or explored enough in gaming.

Proprioception in gaming #

We do some pretty amazing things with our bodies thanks to proprioception. Without it, we end up like Christina, who lost her sense of proprioception due to a spinal cord infection [Wikipedia]. The result, after compensating for this loss with other senses, is “a stiff and slow movement and nearly normal speech, which is believed to be the best possible in the absence of this sense. She could not judge effort involved in picking up objects and would grip them painfully to be sure she did not drop them.”

Most games don’t require anywhere near such proprioceptive effort. You are seldom required to position your character just right to fit through a small gap, or even to know where your character is at all times. And so you seldom get the chance to know what its like to stretch the gaming proprioceptive sense to the limit.

There is expert gameplay where the player seems to know where all the enemies will be, and where to aim so as to get them all with headshots. There is expert gameplay where the player seems to know the game mechanics like their back of their hand, they know just how to bounce their grenades, exactly how far explosive splash damage stretches, how long the projectile lead time is. And there is expert gameplay where the player just seems fully in control of their character, they seem to know how many grenades they have and whether their superbar is full and they are fully aware of their character’s weaknesses and adjust their gameplay accordingly. If we map this to Sun Tzu’s mantras, they would be “know thy enemy”, “know thy environment”, and “know thyself”.

The third is different from the first two though. In gaming, “know thy enemy” is pattern recognition, figuring out which triggers the enemy AI uses and how it reacts, and its attack patterns. “Know thy environment” is knowing the game engine, knowing how it calculates body kinematics and determines interactions, figuring out hacks that are allowable in that engine. “Know thyself”, aside from being familiar with enemy patterns and game engine basics, requires a strong sense of proprioception—“knowing without looking” where your character is, having all the appropriate instinctive responses, being able to make snap judgements about whether you are able to pull off what you have in mind. Many games don’t let you do this, platforming games for instance: you have to look at the screen to know if you landed on the platform right, and if you didn’t you will require an entirely different set of responses. And, of course, you can’t do this with turn-based games.

I don’t play first-person games (for reasons I might explain in future blog posts), so I don’t know how this might manifest itself in such games. but thus far I have only encountered two games that have stretched me this far, and I think the proprioception they require is interesting to explore. I don’t think this is the same phenomenon as flow, a state of mind in which the task at hand has full focus and attention, which is not the development of a sense that identifies another entity as part of your body and treats it as such.

Burnout 3: Takedown #

Burnout 3 is an arcade racer with one key feature: the ability to cause other cars to crash—what is known as a “takedown”. Any illusion that there is no pattern to takedowns is disbursed after 10 hours with the game. The enemy AI is simple and easy to figure out for the most part, and playing each stage a few times gives you a pretty good sense of the environment—the safe zones, how to “hit a wall correctly” so you don’t crash, the different ways to take down opponents.

Proprioception takes much longer to set in though. Muscle memory, visual patterns, actions and their expected consequences … in all, it may take 20 hours or more for that to steep in one’s brain. But when it sets in, a curious thing happens.

Let me say first that proprioception is not necessary for a large number of takedowns. Here is a gameplay video of a player getting a higher record seemingly without using proprioception (and with a slower vehicle). Proprioception doesn’t always make you a better player, but it is definitely necessary for some playing styles.

For the uninculcated, there are some things about Burnout not readily apparent in that video. The greatest level of expertise always looks the easiest. The first couple of hours in Burnout, the first thing that strikes you is just how bloody difficult it is to make an opponent crash. Contrary to impressions from that video, randomly slamming into opponents does not make them crash. They may wobble a bit, then recover and continue on their way. Some time later, you learn that at higher speeds, slams and shunts are a lot more effective. At higher speeds they impart more momentum, and opponents are much more likely to lose control and hit a wall. But you will also learn that at higher speeds, you are much more likely to crash as well. You can choose to play slow, like in the link above, and accept the slower pace of takedowns. Or you can choose to develop your takedown proprioception, and try living life in the fast lane.

A curious thing happens when you take the red pill, the path of takedown proprioception. First, you begin to lose conscious awareness of your car’s state. It’s not critical to your game performance—the only thing that’s important is how many crashes you have left (the game helpfully informs you when the next crash will end your run). The car becomes part of your “game body”, and you stop paying attention to its various features until your focus lapses and you notice your bonnet is hanging loose.

Next, you stop getting distracted by opponents. They, for the most part, largely follow the same behavioural patterns, and are differentiated only by their paintjobs and their weight (which affects their slamming effectiveness). In the early part of the game, it takes a lot of effort and a lot of tries just to take down an opponent. You slam into them once, twice, they escape, and you finally manage to edge them into a gap in a turnoff. By the time you’ve mastered “thy enemy” and “thy environment”, your body knows how to take down your opponents, knows it as a sequence of thumb twitches, faster than conscious thought. Once an enemy appears, you more or less know where it is going to be, and where you need to be to take it down. Your body responds, and already you’re looking at the next oppponent.

And then the next curious thing happens. You stop noticing car takedowns, and your takedown count (except when you take a short breather). The orchestral hit informs you if your takedown is successful, so you know if you should be expecting any shunts from your six. Your gameplay becomes a cycle of recognition-response-verification, represented by the phases of detection-cognition-orchestration (I know the last one’s not technically accurate, but any Burnout player will have that auditory cue so ingrained as a habitual “hit” that I think it works well enough).

In truth, it took me a lot of thinking to even try to recognise these phases. Swatting an opponent, to a player with proprioception, is a bit like reaching for a glass of water in your peripheral vision, or reaching for the remote control. Mastering this allows you to play at speeds at which it would be near impossible to rely on conscious cognition. It takes more than mere knowledge of enemy patterns and the environment, it’s the kind of knowledge that has to be encoded in instinct, in bodily responses. That’s another way of saying it’s not something you can pick up from a wiki.

Assault Android Cactus #

Assault Android Cactus (AAC) is a twin-stick shooter, which generally refers to shooting games where the camera (your viewport) is fixed at a distant point (does not follow over your player’s shoulder), one of your controller thumbsticks controls movement absolute to the arena, and the other thumbstick controls your aiming direction. In such games, the player is usually beset upon by successive waves of enemies which must be fought off.

In this, AAC is following the tradition of other games that came before it, such as Crimsonland. But Crimsonland plays differently; it is a conscious, cognitive game, with moments of interspersed panic when one begins to get overwhelmed. AAC is tuned differently. With a timer game mechanism, and with powerups appearing not randomly but from the bodies of fallen foes, it is a game that wants you to be up close with enemies. That requires a different kind of playing style, a style that is heavily conducive to gaming proprioception.

Games like AAC very quickly overwhelm newcomers: there is simply so much to manage at a go, a multitasking requirement that quickly stretches the cognitive ability to the limit. You have to remember to fire your secondary weapon when it recharges after 5 seconds, you have to watch your back while aiming at enemies while looking for safe zones to move to while watching for the battery drop while collecting powerups for your weapon. Many gamers try to play this like Crimsonland, keeping a safe distance that lets them handle all this cognitively. AAC punishes this play style with a relatively fast timer. You have to get up close with those enemies, to pick up powerups easily (and tot have an easier time aiming), but that means you can be overwhelmed in less than a second if you’re not careful.

Enter gaming proprioception. Gamers with a strong sense of proprioception quickly learn to “outsource” tasks to their subconscious. They learn to aim in general directions rather than at specific enemies (being near enemies helps too). They learn firing patterns that are cognitively cheap (require little conscious attention), for instance getting enemies within a firing arc and then letting loose within that arc. They learn to cover their back subconsciously, for instance by firing backwards every two seconds (this is easier to do with twin-stick shooters than in traditional first- and third-person shooters). They do all this so they can free up the conscious mind for other tasks, like monitoring an internal 5-second countdown for the secondary weapon. If you watch gameplay videos of AAC players where they don’t fire their secondary over a stretch of 20 seconds or more, that is usually where they are getting overwhelmed.

The enemies in AAC, like in Burnout, are also quite easily predictable (thought not necessarily easy to hit). But the mind still has to figure out how different groups of enemies behave (a mix of big spider tanks and melee spiders is different from a mix of big spider tanks and cannon spiders), and what the safe response patterns are. You want to stay away from melee spiders because they hit hard, but get in close to cannon spiders because they don’t. But if you have an accelerate powerup you can zoom past melee spiders, so long as you keep moving. You need some time with the game to slowly outsource these response patterns to the subconscious.

In AAC, precision matters. A slight shift in position can mean the difference between taking a full barrage of bullets, or narrowly sidestepping them. You can achieve precision through conscious cognition, or through proprioception. Once your subconscious calibrates itself to how your character moves, you longer need to keep an eye on the on-screen character; you know instinctively how far it has moved, and how quickly or slowly it will move. This frees up more mental “bandwidth” to focus on enemies, or to listen out for the battery that drops every so often to extend your stage timer. AAC helpfully provides audio cues for lots of important events so that you can listen out for them rather than overstretch the capabilities of your peripheral vision—one way of making a game more conducive to proprioception development.

The end result for both games is something akin to “battle dancing”, a fluid motion of the body that involves little conscious awareness of where its various body parts are and how they are moving.

That is a gameplay video (the game is still in early beta, hence the lack of background textures) of one of the hardest stages in AAC. An S+ rank means that all enemies were killed without the kill combo lapsing (which happens if you allow more than ~2 seconds to pass between enemy kills), and without a single player death. I’ve sunk over 40 hours in AAC and have not achieved that yet. That is an amazing demonstration of gaming proprioception in AAC that I hope to pull off one day.

Game longevity #

The games I keep returning to are games tuned to invoke proprioception, games that overwhelm one’s ability to maintain conscious situational awareness. With such games, the goal becomes an incremental growth towards greater and greater gaming proprioception, what some may call “a sense of oneness” with the game.

I still don’t have any real idea of how this might play out in VR/AR, but I think this bringing this sense of proprioception into new game forms would be an interesting challenge to tackle.