… for Visual Effects

Tip #1763: The AR Elephant in the Ballroom

Larry Jordan – LarryJordan.com

AR allows you to move the camera.

TVBEurope reports that UK-based Strictly Come Dancing has introduced augmented reality into this year’s series. Here’s a detailed look at what they did.

It’s not been easy to bring the ballroom back in the year of the pandemic, and the production team deserve all the plaudits they’ve received for their hard work to get the show on air. Strictly’s production team has been particularly innovative by bringing augmented reality into the mix. From the racing cars in week one, to the elephant that appeared during Bill Bailey’s Quickstep, augmented reality has featured every week during the live shows.

The use of AR has been a real team effort, involving both the lighting and audio teams, as well as companies, Mo-Sys and Potion Pictures. While this year is the first time AR has been employed in the show, it’s something the team has been considering for a while. “We’ve previously used perspective in the floor to create the illusion that the dancers are standing on top of a lighthouse or wedding cake, or skyscraper,” explains Potion Pictures’ managing director David Newton, who also serves as the show’s graphics designer.

“That’s been really effective but the big drawback is you can’t move the camera.”

Newton was asked by Strictly’s producers to look into the possibility of AR, and chose to use Unreal Engine as they were already starting to use it for real time rendering. “Mo-Sys’s name kept coming up in relation to AR and camera tracking, and Epic Games said Mo-Sys have a plugin that works great with their software and we were comfortable with using Unreal so it all sort of added up.”

Using AR looks great on screen, but there’s always the possibility that the couples will end up dancing right through it. How do the team ensure that doesn’t happen? Newton cites Clara Amfo’s recent jive, which featured an AR record player. “The first draft of the record player had it much further down stage, we thought the original starting position was going to be camera right. So we sort of changed how a record player works to actually have the arm on the other side. But it all changed, and it went further upstage and there were a few more meetings about where it was going to go, what colour it should be, what side the arm was going to be, how many letters of complaint we would get if the records spun in the wrong direction,” he laughs.

As well as the graphics, Strictly’s lighting is a key component of the augmented reality employed in the show. David Bishop, the show’s lighting director, says it’s been interesting to explore the relationship between lighting and AR. “In my mind there are two routes with AR, you have to either make something that is real, and therefore it has to appear on the screen as being absolutely real, or you have to make something that’s very clearly an ethereal dimension and isn’t meant to be real,” explains Bishop. “For example, if you’re inside an AR-created house, as we’ve used this series, and there’s a light bulb inside then the person that’s standing inside that house needs to be lit as though it’s coming from that fake light. That means I have to find a real light in the same direction, which sort of does the same job.

Bishop continues: “The tricky thing about that is that our spot ops can’t see the things they’re trying to point the lights at. So they’d be pointing the spotlight in one direction and then I would be saying left a bit, down a bit. It’s that sort of workflow that’s become quite new to us but it is absolutely key, getting the lighting angles right is what’s making the AR even more believable, and that’s certainly something that’s improved throughout the series.”

Read the full article here.