I read a column by Brendan Sinclair where he suggests that AR for gaming doesn’t have much of a future beyond the novelty factor. On Mastodon, he was even more blunt, suggesting that AR is good at attracting venture capital but will inevitably run the same course of underwhelming reality as current-day VR has.
My overall take on that column is that Sinclair makes short-term observations that I entirely agree with, and then he makes conclusions that I think are myopic and unimaginative. To be fair: the column is about games as business, which is all about analysis of existing product more than speculation about the future, and Sinclair acknowledges as much in the column.
For instance: it’s tempting to point to Pokemon Go’s success as a sign that AR is a potential gold mine, but as Sinclair points out, that game was successful because of its IP and its geolocation more than its AR functionality. We’re in agreement there, but I disagree that you can extrapolate much about the viability of AR games from that.
Pokemon Go’s AR element was doomed to be uninspiring (in my opinion at least) for two reasons: first, it had to be compatible with a broad range of devices, which limited it to the lowest-common-denominator in terms of AR functionality. Second, it had to work with a game that was literally designed to be played anywhere on earth, with zero predictability in terms of environment.
The main takeaway I got from Sinclair’s column is that most people’s thinking about the realistic potential of AR and VR — including my own! — has been both defined by the limitations of existing implementations, and also set to an impossibly high standard.
The devices — and by extension, experiences — that we’re familiar with have all been limited by necessary compromises: some of them because the tech just isn’t there yet; some of them because companies rushed products to market before they were fully baked; and some of them because the devices were intended to be prototypes, to generate ideas about what the future of VR or AR could be instead of presenting any finished and polished technology.
And to be clear, when I talk about products being rushed to market before they’re “ready,” I don’t think it’s entirely sinister. I fundamentally disagree with Meta’s overall take on VR, for instance, but I do think it was a reasonable decision to emphasize lower cost, wider adoption of headsets over the absolute best and most expensive technology.1In retrospect, I don’t agree with Google’s versions, though, even though I thought they made sense at the time. Sure, technically Google Cardboard and Daydream brought the potential of VR to more people who otherwise wouldn’t be able to afford it or be interested in it, but it also set expectations impossibly low for what VR could be.
But it’s also put us in a weird position in which current implementations of VR haven’t lowered the bar, but raised it. In other words, for some reason, it’s not enough just to fix the problems with the existing technology. Supposedly, we have to make it perfect, or it’s not worth pursuing at all.
Sinclair writes:
If AR gaming is going to be anything grander than a new way to play games entirely possible in other formats, then the games need to understand and interact with the player’s environment. And that only makes sense if the environment where you play – the specific arrangement of furniture and things around you – is meaningful to gameplay.
Which is essentially like saying that AR is a useless novelty until it achieves a level of object recognition and environment mapping that is at least five to ten years away. Even five years feels overly optimistic, but considering that mapping out and recognizing an environment is exactly the kind of problem that fits ML modeling, I expect it’s going to advance very quickly.2Apple’s ARKit already does a pretty good job of detecting walls, floors, ceilings, and tables, and can generally differentiate between different types of furniture, but at least on the phone it’s not yet at the state where I’d trust an entire experience to be based around it.
Double Standards and Pong
I’m realizing that I’ve been guilty of that thinking, too. The Apple Vision announcement has gotten me excited about the potential of AR and VR again, and I’ve been eager to think of some killer idea that I could develop for it, mostly to reinforce my weak justification that I really really want to buy one. And every time, I come to the conclusion that unless it’s a breathtaking, only-possible-in-AR type of experience, it’s not worth doing at all.
But the thing we all should remember is that no other medium3At least that I can think of; feel free to comment if I’m missing an obvious one has ever developed that way. Sudden breakthroughs that change everything are exceedingly rare; what happens the most often is we make gradual improvements on things that already exist. After all, the first big hit in home video games wasn’t some hugely complex experience made possible only by state-of-the-art processors, but ping pong.4One might even say it was only half of ping pong.I’d originally called it a “breakout hit” but that was an entirely different game. Altogether.
There’s plenty of potential in finding thoughtful ways to make existing games and apps just a little bit more compelling or convenient to use in AR and VR. I thought it was interesting that one of Apple’s developer videos about designing apps for the Vision Pro recommended doing exactly that: choosing a single key feature or element that would benefit most from “spatial computing,” instead of overwhelming users with flashy AR/VR presentations or interactions just for their own sake. That kind of restraint has been missing from a lot of VR game development, where there’s this unspoken assumption that you have to completely re-invent the user experience. This often makes them kind of exhausting to use, and contributes to the feeling that an experience has to be absolutely mind-blowing to warrant all the hassle and inconvenience of putting on a headset.
I don’t want to see the AR equivalent of shovelware, obviously, but I also don’t want to see people over-correcting, assuming that there’s nothing to be gained by presenting familiar stuff in a new way. I don’t believe that the only way AAA developers can make in-roads to AR is Sinclair’s example of Kratos inhabiting your kitchen. I think it sounds pretty compelling just to play God of War (or any other AAA game you choose) on a 3D TV screen as big as your wall.
One Day, Your Computer Can Be a Big-Ass Table
“Gigantic 3D monitor” is still the killer feature of the Vision Pro, as far as I’m concerned, even though it’s the one that’s most often dismissed as mundane. And for what it’s worth, I think the people dismissing 3D TV as just a fad are missing the point entirely. I don’t think 3D TVs failed because they weren’t cool, but because they weren’t cool enough on their own to justify the extra cost and the special glasses.
Anyone looking at the Apple Vision Pro showing a 2D screen and scoffing “Oh great I can pay $3500 just to play NBA2K”: just get out of here with that disingenuous nonsense, please. You don’t buy a $1000 TV just to watch Game of Thrones, just like you don’t buy a $500 PS5 just to play Elden Ring, and it’s 2023 and we’re all well acquainted with the idea that computers do multiple things.
But about that Apple presentation: at different points, they showed someone playing a 2D basketball game with a PS5 controller, and the Disney demo of someone watching a basketball game with a diorama playing out on their coffee table and stats and scoreboards scattered around all over the living room. One got dismissed as being too mundane, and the other got (rightfully, IMO) dismissed for being too over-the-top. But I can totally imagine a compelling experience that’s somewhere in the middle.
I can totally see playing a game like Jedi: Fallen Order or Spider-Man in a big screen with supplemental views off to the sides. You don’t have to hide your UI to keep it from breaking the immersion of the main view. You could even pop up reminders of combos that you can perform, a persistent mini-map, or sections from the in-game lore. And incoming messages from other characters, too — one of the things that Disney attractions do well is have multiple channels of information playing out in different screens all around the “main” experience.
And I can see that being even more compelling in a strategy game, like XCOM or if real-time strategy games ever make a comeback. Devote the large view to close-up, short-term battles or tactics, and keep the larger strategy map just below the screen. And of course, give the player the option to put that strategy map on their coffee table if they want.
Do You Understand the Tables Are My Corn?
And speaking of tables: I think that the most interesting potential application of AR to gaming, at least in the short term, is the possibility of augmented tabletop games. Not just RPGs, but board games as well.
It’s here that Apple’s insistence on an AR-first headset, with excellent passthrough, is going to make all the difference. There has been absolutely no shortage of attempts to do board games in HMD-based VR or phone-based AR, but it’s rare that it’s seemed like more than just a gimmick. Playing a game in VR removes all the inherent appeal of playing with physical game components. And all of the AR possible to date has required holding up a phone or a tablet, which usually means brief, one-off interactions. (Full disclosure: I worked on the iOS app for the Infinite Rabbit Holes Panic in Gotham City game, but not in a design capacity. All these opinions are my own and don’t represent anyone else’s, etc).
But if you can wear a HMD that lets you see everything in the room — and, presumably, is comfortable enough to wear for hours — then that opens up a ton of possibility to have truly augmented experiences with “analog” games. In the short term, at least, I don’t see these being at all suited to in-person social experiences, but they’d be great for remote games. Take everything from a system like Roll 20, lay it out on the table in front of you, and leave open the option to interact with your minis. Depending on how distinct your minis are (and how well they’d be recognized by object detection), you could move pieces around locally and have them reflected in a remote player’s view.
And while I’m skeptical that the Vision Pro is going to be good in social environments — at least for a few iterations, as Apple does what they do best, making the thing smaller, lighter, and less obtrusive — it’s a no-brainer for adding a solo mode to board games that typically require multiple players.
Master of None
All of the HMDs that I’ve used have required some amount of hassle to get started. The Vive required the external lighthouse sensors, and even the ones that do inside-out tracking have required setting up a play space. And all of them have been kind of hot and uncomfortable, and have meant closing yourself off from everything and everyone in the room.
If you can wear a Vision headset for an extended period without feeling too cut off or too uncomfortable, and you’re able to see and interact with physical objects in the room, then there’s a ton more opportunity for genuinely augmented experiences. The complaint about the Vision Pro being “isolating” will likely be around forever5Considering that the “Apple tax” is still a persistent complaint, long after it’s been shown over and over that it’s not a thing when you’re taking about comparably-powerful computers, but there’s not much merit to it: books are isolating; the only difference is how quickly and easily you can interact with the real world while you’re using it.
I haven’t yet played the game Inscryption — I’ve been warned that it’s got scenes of eye trauma, which is one thing I can’t handle — but I think it’s a much better example of the near-term future of AR gaming than either AAA console games, or Niantic’s location-based games. Imagine a card game where your cards change and react to you, where they affect the environment around you, and where there’s always another player available (even if they are a creepy villain).
At some point in the future, I have no doubt, it’ll be great to have some kind of high-budget, high-production experience where a fantastic world with fantastic characters builds itself into your existing environment. But I think it’s comically short-sighted to dismiss an entire medium if it doesn’t jump immediately to that stage with no interim period.
Are board games and indie game darlings too niche for publishers to care about? That’s entirely possible, but I’m not entirely sure that it matters. AAA games have been getting bigger and more expensive as they’ve diverged from their roots and gone farther in the direction of blockbuster movies. I like the idea of not having to choose between action movies, video games, or tabletop games, because I’ve got a device that can improve on all of them.