Meta reveals future of realistic avatars at offices in Pittsburgh

The future of ultra-realistic human avatars has started, and technology for it is being developed in Pittsburgh.

Meta, the Menlo Park, Calif.- based global tech company that operates Facebook, Oculus and other tech firms, provided a media tour of its Strip District offices on Tuesday. The company moved into the four-story District 15 building in early 2020. This is the first time the company opened its doors to the news media since the pandemic.

Chuck Hoover, general manager of Meta’s Reality Labs Research center, said hundreds of employees work across Pittsburgh offices. In addition to the main building in the Strip District, Meta has smaller facilities in Oakland and at Carnegie Mellon University.

Pittsburgh stands out compared to other Meta offices across the globe because it is home to complex audio and video technology that Meta developed to help create realistic avatars and collect terabytes of data to advance its virtual reality technology, he said.

Hoover moved to Pittsburgh in 2005 and has been working with Meta for several years. He said Meta has been increasingly successful at attracting top talent to come to Pittsburgh. Hoover said about 40% of Pittsburgh’s Meta employees are from around the world, 30% are from the Pittsburgh region and 30% from elsewhere in the United States.

“We have been able to draw in talent from around the world, and that is something we couldn’t have done 17 years ago,” he said.

Meta has created chambers that capture people’s voices, facial expressions, appearance and movement to help construct highly realistic avatars. The avatars on display Tuesday looked nearly human, a far cry from avatars that are on display in Meta’s Horizon Worlds VR platform, also known as the “Metaverse.”

At Meta office in Pittsburgh, company’s VR headquarters. This realistic avatar demo was created w/ tech on site, including hundreds of cameras & microphones capturing hours of footage and terabytes of data. Meta says scaling this tech is still several research breakthroughs away pic.twitter.com/W4W8YPe1lU

— Ryan Deto (@RyanDeto) August 23, 2022

One of the chambers is called Mugsy, and it uses 171 video cameras and hundreds of lights to record and capture people’s faces and about 16 terabytes of data, or about the same as 4,000 full-length films.

Hearsay uses 120 microphones embedded in silicon ears to capture people’s voices from every possible angle. And Sociopticon, a large circular room, uses 253 cameras to track the entire body’s movements.

There are other, smaller sound and video chambers.

After subjects provide hours inside the chambers, Meta can create ultra-realistic avatars that showcase detail down to strands of hair, pores on the skin, and stubble on beards. The avatars also make and maintain eye contact, which Reality Labs director Yaser Sheikh said is key to moving virtual reality and video conferencing beyond its current limits.

“The first time when I made eye contact with another avatar, it felt like I was making eye contact with the person,” said Sheikh.

The tour comes about a week after an image of Meta CEO Mark Zuckerberg’s avatar was widely ridiculed on social media. Zuckerberg’s avatar, which looked cartoonish, was from the current version of the metaverse that users can interact with today.

Sheikh said Zuckerberg is aware of the research being done by the Pittsburgh team, and he praised him for having “the vision to see where this technology is going.”

As of now, though, Meta’s realistic avatars are not close to widespread implementation.

For one, users would need to visit Pittsburgh and undergo hours of testing to create personalized avatars.

Sheikh realizes this is not possible. He said Meta is working on technology to have people use their smartphones to scan in their appearances, voices and facial expressions, but Sheikh said the company is still “several research miracles” away from that.

“The social metaverse is the grand challenge of the next 10 years,” said Sheikh.

In a roundtable with reporters in June, Zuckerberg acknowledged the challenges of improving and implementing this virtual reality technology, but said he was committed to this vision and would be directing Meta in this direction to fully form the metaverse.

According to Yahoo News, Meta spent $10 billion in 2021 on the metaverse project alone, up from $6.6 billion in 2020.

To advance its realistic avatar technology, Meta wants to capture at least 10,000 identities from across the world. Right now, the company has only been able to capture subjects in Pittsburgh.

Technical program manager Ashley Brewer said Meta has the capacity to take in about four participants a day, which are either public volunteers or Meta employees. Brewer said Meta has completed study of 219 subjects to date and is hoping to scale up to about 900 a year.

Hoover said Meta will need to improve the diversity of its subjects to ensure the realistic avatars truly match their human counterparts.

“The avatars that we have created so far, they do lean a bit towards your weird Pittsburgh cousin,” Hoover said.