Augmented reality shopping is about to level up, thanks to a new capability introduced in iOS 16. The new Apple RoomPlan API uses the LiDAR scanner in recent iPhone Pro models to create a 3D room plan – and an impressive new demo shows just what it’s capable of.
Shopify has created a proof of concept that scans a room and then allows you to virtually empty it, ready to see how new furniture would look in the space. Alternatively, you can remove specific items, like a couch, and then slot new ones into the space …
If that sounds familiar, it’s because Ikea has already done the same thing in cruder form, but we noted at the time that RoomPlan was likely to let furniture companies create more sophisticated apps.
Shopify’s Russ Maschmeyer says there has been lots of interest in the capability. He refers to the feature as a reset button for a room, and explains what it can do.
AR makes it easy to try new couches, but the couch already in your space can get in the way. What if every room had a reset button? Space Eraser models spaces in high-fidelity—capturing room-defining objects, their size, position, orientation.
Live pixel data combines with LiDAR depth data to enable a lifelike digital twin of your room that can be overlaid onto your real space using AR and then edited digitally.
With an empty room, shoppers can now swipe through room sets which leverage the orientation of existing furniture to arrange themselves automatically.
With a digital twin of your room and your furniture you could simply swipe your original sofa away to explore options that better match the rest of the room.
Here you can see the reset process:
And here, removing individual items from the room:
Maschmeyer talks through how they created it, and some of the challenges they faced.
RoomPlan outputs an untextured USDZ model composed of unit cubes; one for each door, window, wall, and room-defining object but, surprise! Models don’t contain floor or ceiling elements. We align the model with the real world using ARWorldMap.
Since RoomPlan exports untextured models, we explored ways we might texture it automatically. The best way would be to localize and map the texture when capturing, but we make do with an aligned model and realtime camera feed.
We reversed the graphics pipeline: instead of sampling from a texture and writing to the screen, we sample from the camera and write to a texture. RealityKit made this hard, so we used SceneKit instead.
This approach creates issues when furniture obscures a surface. In video above our couch obscured the wall. RoomPlan provides bounding box references for room-defining objects. With this we can determine what parts of the texture to redact.
We can now use inpainting techniques to fill in the redacted texture. We were pleasantly surprised by the results produced by off-the-shelf tools like Photoshop and theinpaint.com.
AR and #RoomPlan are exciting tools for commerce. The ability to select and remove real objects from a scene brings us one step closer to a fully editable reality.
I’m a big fan of augmented reality shopping, providing a good sense of how something would look in your home without anything as barbaric as visiting a shop in person. What do you think? Please share your thoughts in the comments.
FTC: We use income earning auto affiliate links. More.
Comments