Carnegie Mellon students invent AI pet that merges virtual reality with the physical world

When Carnegie Mellon University graduate student Brian He proposed developing a fully virtual pet, the idea was rejected for lacking innovation.

Despite the setback, the Physical Presence Pet team took the idea a step further and roughly 14 weeks later, He and five other Entertainment Technology Center students had created the prototype “Luceal,” a functioning artificial intelligence pet, which combines the virtual reality (VR) experience with the physical world.

“We can have a virtual companion be overlaid on top of … a physical object that not only has the tactile features and textures of a physical object, but also the virtual expressive animations,” He said.

When they pitched their idea to a committee of faculty members, Entertainment Technology Center Director Derek Ham was surprised in the similarities between the student’s idea and a prototype he had been working on.

Ham felt that the entirely virtual aspect of the AI pet project had been done before, so Ham and the team combine ideas, He said.

Using an Apple Vision Pro VR headsets from CMU’s department, the students, with the help of IDeATe Soft Technologies Professor Olivia Robinson, custom-built textile sensors and animated designs, He said.

Luceal, which was designed in its totality for under $5oo, works similar to a game controller — each section of the plush animal has sensors that when pressed send a signal to the VR googles, the virtual creature then reacts in real time to the touches and gestures.

“I showed them how to create custom soft sensors and textile circuits using conductive fabrics, yarns and threads. E-textiles were new to the team, and they were excited to create custom soft sensors that could be subtly and smoothly integrated into their stuffed animal form,” Robinson said in a news release.

Of the Physical Presence Pet team members, four are international students, who have been unable to bring their pets to campus.

Joy Lim, who selected the fabrics and hand-sewed Leceal’s face together, said she grew up in a household with a number of pets and described herself “heavy pet person.”

“I grew up with a lot of different animals,” Lim said. “As a international student, I couldn’t bring my pet and everywhere I go. So it would be awesome that I can have a … device that is just around me all the time.”

When someone feels the softness of Luceal’s fur help create connection and calmness, Lim said.

He and Jerry Zheng, the game programmer and designer, modeled the idea off of Tamagotchi, which was brand of handheld digital pets developed during the 90s, by Japanese toy maker Bandai.

“We could absolutely leverage the interactivity of this three dimensional space to create some very other worldly kind of creature that normally you wouldn’t be able to interact with. Dogs and cats are always around, but you can’t have a pet such as like a seal or dragon or something exotic, because either they don’t exist or they’re under a heavy animal production,” He said.

Sophie Huang, a second-year master’s student and the animator for Luceal, designed several different skins so users could change what the animal looked like through a mobile app designed by Zheng.

Tech artist Paige Li, had to work around limitations of new technology.

“What I saw when developing on my computer wasn’t the same on the Vision Pro — colors and scale looked completely different. I had to find a lot of workarounds to get the visual effects to work,” the release said.

William Zhang, the producer, said the most complex challenge the team faced was aligning the digital and physical pets in space, the release said.

“If people see a virtual thing floating in space — and the physical plush is somewhere else — they’ll want to touch the virtual one,” said Zhang.

While the student’s semester working on Luceal has ended, He said they plan on showing off their creation at SIGGRAPH, a conference on computer graphics and interactive techniques and will be submitting to the Game Developers Conference, located in San Francisco, next year.