TEDxUCLA 2016: Push. Pull. Stretch.

3D conversations on history


About Andrew

Andrew Jones is a computer graphics programmer and inventor at the University of Southern California’s Institute for Creative Technology. In 2004, Jones began working in cultural heritage, using 3D scanning techniques to virtually reunite the Parthenon and its sculptures. The resulting depictions of the Parthenon were featured in the 2004 Olympics, PBS’s NOVA, National Geographic, the IMAX film Greece: Secrets of the Past, and The Louvre.


Imagine if you could meet anyone in the world. Anyone. Who would you choose? What questions would you ask? And would you record that meeting so that, years from now, you could relive it?

I work at an Army research lab, the USC Institute for Creative Technologies, and my research focus is on how to make faces in video games and blockbuster films appear less computer-generated and more real. Back in 2010, my team and I, we were invited to showcase this new project we’d invented, essentially a floating 3D head in a box.

Now this display was pretty cool for several reasons. The first was that you could see the depth of the face without needing any special 3D glasses. But secondly, this was a floating head and you could talk to her and she would look you in the eye or turn and look at the person next to you, just like in face-to-face conversations. And that, that’s something you can’t even get in regular video chat. Now most people, when they saw this display, would say something like, “Oh wow, it’s holograms, finally here.” But others, they’ll just lean in and ask you to repeat, “Help me Obi-Wan Kenobi, you’re my only hope.” Just as I was getting tired of hearing that same quote over and over again, then this woman from the USC Shoah Foundation came up. She grabbed my hand, looked me in the eye and said, “Would it be possible to record this 3D conversation, but with a Holocaust survivor?”


Would it be? All I had really was this 3D floating head. And the most we’d ever recorded of that, that was 60 seconds. That’s, that’s not much of a conversation. And it’s just a head. If you really want to have an impact, you’d want the entire body from head to toe.

But there was also this added urgency because the remaining Holocaust survivors, they’re in their 80s and 90s, so in the next decade or so most of these firsthand accounts, they’re going to be lost forever. It all seemed just totally daunting.

I mean, I am a technology guy. What do I know about the Holocaust? But at the same time, how could I pass up this opportunity to meet an actual survivor, help preserve part of history? I knew in my heart there had to be a way to turn this 3D vision into a reality.

As we began, I realized we had two major challenges. The first was, how do we record these survivors in 3D? Now we already had this recording stage that we had built to scan actors and turn them into digital doubles for visual effects films. But this was gonna be different because these were real survivors with unscripted, much longer experiences.

Now the first survivor that I got to meet was Pinchas Gutter. And when I got to talk to him, he was so warm and friendly that I just couldn’t comprehend how between the tender ages of 11 and 16, he had not only survived the Warsaw Ghetto but six concentration camps. But he was incredibly patient. I mean, we were experimenting with different lighting directions, camera angles, because our dream was that one day kids could sit all the way around the 3D Pinchas and listen to his stories. So we placed 50 cameras around the stage and by selecting different cameras, we could be compatible with different displays like traditional 2D televisions, 3D glasses, or whatever new display people might be using 20 years from now.

The second challenge had to do with interactivity and the fact that our target audience were gonna be kids. The USC Shoah Foundation has actually already recorded over 50,000 Holocaust testimonies. But these are more like lectures, so most kids really don’t have the attention spans to watch them all the way through. But if one kid is lucky enough to meet an actual survivor and ask them questions face-to-face, then they start to feel that survivor’s pain, regret, even joy. And it’s these shared emotions that make the lesson stick long after the conversations are over.

So we wondered: could we use the 3D video, but to simulate an actual conversation? Now unlike lectures, conversations are naturally made of much shorter responses because this provides opportunities for the listener to ask their own questions. But this would mean that we would have to record a long interview with Pinchas over many many hours, basically making his entire life into these short, conversational segments. So we began by recording like the, gathering the most common questions that Pinchas receives when he currently speaks in public.

But there were unexpected stories too. For example, one of our crew members apologized to Pinchas for all the terrible things that had happened in his life. But he responded with this Yiddish joke: “When you ask somebody, ‘What is the most terrible thing that can happen to you?’ And your answer is, ‘Terrible? Nothing is terrible. The only thing is terrible is when you swallow an umbrella, the umbrella opens inside, and you want to take it out. That is terrible.'”

So that clip just gives you just that little tiny glimpse of what meeting Pinchas is like. He’s just so warm and he’s amazing. But we had to ask the the big philosophical questions as well like, “Pinchas, do you blame God for what happened?”

“Do I blame God? This is a question that is asked of me many, many times. And my answer is I blame the people. The person, the persecutors, the ones that did all these vile things. The S.S., the Gestapo, the whole ideology of Nazisism. That’s what I blame for the Holocaust.”

As each of these answers, I mean, they’re completely spontaneous and different. So we had to make sure we got it the first time every time, no second takes. But what I hadn’t expected was how these, these heart-wrenching stories, they were going to affect me.

And one example of that is when Pinchas told us about his 11-year-old twin sister. She was murdered the very first day they arrived in the concentration camp. And what pains him the most is he can’t even remember her face.

When I heard that story, I was on the verge of losing it. But we were recording all this audio, so I couldn’t be sniffling in the background. I had to hold it together. But there were happier moments, like we invited children to come interview Pinchas and he would open up. They would ask such innocent questions, like “Do you have any pets?” or “Did you ever meet Hitler?”

In all, we gathered over 2,000 questions over seven days. At the afterparty, I was worried. We gathered so much footage from all these cameras, it could take years for me to sort through this all. But I had to promise Pinchas, no matter how long it took, we would find some way to share his stories so others could feel the same things I was feeling: his strength and his wisdom.

Now the key was that one good story could actually answer many different questions. So the IST Natural Language Group gathered over 20,000 alternate questions and linked them to different answers in this interview. And this allowed us to train a computer to parse a visitor’s question and then search for and play back the closest video response.

Then, five years after this project began, we were finally ready to travel to the Illinois Holocaust Museum and Education Center. Here we were going to introduce the new digital Pinchas to the general public. Now I was still working on that 3D display, but it wasn’t quite ready for prime time. So instead we showed Pinchas on an 80-inch flat screen television so he’d be just as large as he is in reality.

Then one day, fifteen Polish Holocaust survivors came to visit the museum. This was the moment of truth: would they feel that we had done justice to their stories? One of the survivors asked Pinchas: “What is your favorite song?”

(sings a Polish lullaby)

Slowly, each survivor recognized this Polish lullaby and they started to sing along. I mean they were sharing memories of a simpler time before the war.

One of the survivors had actually been in the same concentration camp as Pinchas, but he couldn’t remember any of it. Now Pinchas could share stories about the camps, but he could never be as specific as that survivor really desired. And this turned out to be the system’s biggest challenge.

Now the next step is to have a 3D Pinchas in a museum near you. We’ve just finished building a new one-of-a-kind display. Behind Pinchas there, you can see 200 lights and a horizontal band. Those are actually individual video projectors. So as you walk around the display, you can see different views from different video projectors. This creates a sense of depth and presence so that visitors respond as if Pinchas is actually there. They apologize for his suffering or if they interrupt. One survivor even went up and he tried to touch the screen and then he looked behind. He was trying to see, “Where’s the real Pinchas?”

Now our 3D library with the Shoah Foundation continues to grow. We’ve just finished recording conversations with 11 other survivors, preserving their stories for the future. But what I’ve learned through this journey is that it’s the stories and emotions that really matter. For example, we asked survivors like Pinchas, “What message do you have for your family and the future?”

“The message I leave with them is that the most important thing that they should do is tolerate, be human towards each other, make sure that they accept any other, any other person who is of a different culture, different religion, maybe different color of skin, and that they should all live like brothers.”

When I hear these words, they inspire me. I wonder, what wisdom would my father want to pass on? Because one day, in the not-so-distant future, this technology is going to be available to all of us. Our memories will stretch across the generations so that our great-great-grandchildren, they’ll be able to experience our thoughts, feelings, and dreams.

So, knowing that one day soon it’ll be your turn, I ask you: What stories will you choose to tell? Thank you.