ShareThis Page

At CMU's Panoptic Studio, 531 cameras capture the tiniest details of human movement

Aaron Aupperlee
| Wednesday, July 19, 2017, 10:15 a.m.
The Panoptic Studio inside of Newell-Simon Hall on CMU's campus on July 14, 2017.
Nate Smallwood | Tribune-Review
The Panoptic Studio inside of Newell-Simon Hall on CMU's campus on July 14, 2017.
The inside of the Panoptic Studio inside of Newell-Simon Hall on CMU's campus on July 14, 2017.
Nate Smallwood | Tribune-Review
The inside of the Panoptic Studio inside of Newell-Simon Hall on CMU's campus on July 14, 2017.
Hanbyul Joo, one of the students who has worked on the Panoptic Studio, explains the machine's features inside of Newell-Simon Hall on CMU's campus on July 14, 2017.
Nate Smallwood | Tribune-Review
Hanbyul Joo, one of the students who has worked on the Panoptic Studio, explains the machine's features inside of Newell-Simon Hall on CMU's campus on July 14, 2017.
The inside of the Panoptic Studio inside of Newell-Simon Hall on CMU's campus on July 14, 2017.
Nate Smallwood | Tribune-Review
The inside of the Panoptic Studio inside of Newell-Simon Hall on CMU's campus on July 14, 2017.

A two-story dome at Carnegie Mellon University outfitted with more than 500 cameras could help computers diagnose autism, fix a golfer's swing and understand body language.

Hanbyul Joo and Ginés Hidalgo, two researchers in CMU's Robotics Institute, have spent three years gathering data on human movement from inside the Panoptic Studio in the basement of Newell Simon Hall.

"When we are communicating, we are using our bodies as well to convey some information," Joo said. "Each of our body gestures or facial expressions has some specific meaning, but that meaning is really hard to describe or define."

That's where the 531 cameras inside the Panoptic Studio come in. The cameras, a combination of high- and low-resolution rigs, capture human movement inside the dome. That movement is fed into a computer that catalogs and measures it, creating data about how we nod, smile, cross our arms, point our fingers and roll our eyes.

"Ultimately, we want to make machines to understand that kind of information by themselves," Joo said.

Right now, the Panoptic Studio gives researchers a highly detailed, 3-D, 360-degree look at movement. The studio generates 600 gigabytes of data a minute when all 531 cameras are recording.

Athletes could have their swings, kicks, serves and running captured. Joo and Hidalgo have shot video of a dancer, a cellist, a guitar player and a drummer. Joo has watched his son grow up inside the Panoptic Studio, taking him in a couple of times a year to record how he plays, walks and interacts.

Joo and Hidalgo just released software that provides the same tracking and measuring for regular videos shot outside the studio.

RELATED: CMU researchers are teaching a computer to understand body language

This could mean detailed analysis of sports and insight into music, dance or how a child develops, Joo said. A doctor could look at footage of a child, run it through a computer program and determine, based on movement, whether the child is autistic.

And once Joo and Hidalgo collect enough data and develop the right algorithms and artificial intelligence, a computer could make the diagnosis, Joo said.

"To understand, we need to know what is normal interaction and what is not normal interaction," Joo said.

Joo and Hidalgo's research builds on decades of work by Takeo Kanade, a professor at CMU's Robotics Institute. Kanade's work into multi-camera imaging began more than 30 years ago with a handful of low-resolution cameras. In 2001, CBS debuted Kanade's multi-camera technology during the Super Bowl. Viewers could see replays stitched together from 30 cameras. The image could rotate around the action.

Kanade in November was awarded the prestigious Kyoto Prize , Japan's highest private honor for those who contribute to humankind's scientific, cultural and spiritual development, for his work on computer vision, including pioneering and advancing the field of computer vision, critical to the development of self-driving cars.

Aaron Aupperlee is a Tribune-Review staff writer. Reach him at aaupperlee@tribweb.com, 412-336-8448 or via Twitter @tinynotebook.

TribLIVE commenting policy

You are solely responsible for your comments and by using TribLive.com you agree to our Terms of Service.

We moderate comments. Our goal is to provide substantive commentary for a general readership. By screening submissions, we provide a space where readers can share intelligent and informed commentary that enhances the quality of our news and information.

While most comments will be posted if they are on-topic and not abusive, moderating decisions are subjective. We will make them as carefully and consistently as we can. Because of the volume of reader comments, we cannot review individual moderation decisions with readers.

We value thoughtful comments representing a range of views that make their point quickly and politely. We make an effort to protect discussions from repeated comments either by the same reader or different readers

We follow the same standards for taste as the daily newspaper. A few things we won't tolerate: personal attacks, obscenity, vulgarity, profanity (including expletives and letters followed by dashes), commercial promotion, impersonations, incoherence, proselytizing and SHOUTING. Don't include URLs to Web sites.

We do not edit comments. They are either approved or deleted. We reserve the right to edit a comment that is quoted or excerpted in an article. In this case, we may fix spelling and punctuation.

We welcome strong opinions and criticism of our work, but we don't want comments to become bogged down with discussions of our policies and we will moderate accordingly.

We appreciate it when readers and people quoted in articles or blog posts point out errors of fact or emphasis and will investigate all assertions. But these suggestions should be sent via e-mail. To avoid distracting other readers, we won't publish comments that suggest a correction. Instead, corrections will be made in a blog post or in an article.