The Holodeck On Your Head: A Virtual Media Studio

I was asked to give the Closing Keynote for the 2013 Audio Engineering Conference this week at Javits in NYC. The convention drew over 18,000 attendees. For my topic I selected “The Future of Audio Production 2020-2050”. Added:  CNET reported here. I spoke about the demise of “physical” post-production. The lecture was accompanied by around forty proprietary research graphs, not included here. By 2050, perhaps even as early as 2030, I showed (with extensive data) that most media post-production will be performed in virtuality, where every functional piece of equipment — every knob, fader, switch, and patch point — will be visible and controllable entirely in virtual space. This paradigm will encompass film editing, sound and music editing, game production, mixing, mastering, and just about any type of aural-visual post-production and delivery. By 2040, we’ll have mostly abandoned the mouse. Physical touchscreens will be largely obsolete. There will be far fewer physical media objects — such as external audio monitors, keyboards, trackballs, personal desktop video monitors, and so forth. Save for a quiet room, a comfortable chair, and innocuous motion trackers, the physical “production studio” will largely be a thing of the past. Certainly, a number of “legacy hardware rooms” will still exist, but they will be dying curiosities. Bottom line: we have moved from a desktop-culture to a hand-held culture, and now we are moving from a hand-held culture to a head-worn culture. Physicality will be replaced with increasingly sophisticated head worn immersion devices. Most of these basic changes will be well in place by 2035. And by 2050, head-worn audio and visual fully-spherical realism will be nearly indistinguishable from real-space. Audio will be mixed for a true three-dimensional sound space (in fact, we are doing this now). Visual production will require three axes of reality (also happening today). During this transition, perhaps the only remaining piece of CEH (clunky external hardware) will be sub-woofers, which cannot be emulated with a headworn device. By 2025, today’s emerging object-oriented 3D audio environments (Atmos, Neo, Auro, etc.) will be commodity delivery formats. By 2025-2030, head motion tracking and hand gestural tracking will also be inexpensive, matured commodities. A single desktop computer in 2050 will be equivalent to roughly 10 billion human brains working in parallel, so media processing power is no longer a bottleneck. The 2050 Internet will be hosting roughly 10,000,000,000,000,000,000,000 bits of data, per second (10 sextillion bits/s). Production and post-production studios of 2030-2040 will give us our familiar working tools:  mixing consoles, outboard equipment, patch bays, audio and visual monitors … or their real space DAW equivalents. The difference is that all of this “equipment” will live in virtual space. When we don our VR headgear, anything we require for media production is there “in front of us” with lifelike realism. It’s a Holodeck on your head. Headworn reality. Matured gestural control (2030-2035) allows us to reach out and control anything in the production chain. Efficiency will be improved with scalable depth-of-field. Haptic touch (emulated physical feedback) will add an extra layer of realism (2030-2040), but it’s probably not necessary for media production emulation. Anything in the virtual room can be changed with one voice or gestural command. Don’t like the sound of that Neve 8086 console? Install the Beatles EMI Abbey Road console. A ten second operation. But why stop there? Let’s dream bigger. Call up a complete AI symphony orchestra that fills your immersive vision stage. Call up a great concert hall (let’s try the Concertgebouw. Hmm, that’s a little too swimmy. Let’s try Boston Symphony Hall). Add a 200 voice choir. Add Yo Yo Ma soloing with his carbon fiber cello. You’re there in front, conducting and refining the orchestra with gestural and voice commands, making refinements to the score and performance, until it becomes exactly as you want it. We achieve a complete Virtual Audio Workstation, or more precisely a Virtual Media Workstation which can be tailored to fit any creative production goal. The future of audio, music, film making, game design, TV, industrial apps  —  any creative media construction, from inception to post-production — becomes truly boundless and limited only to our imagination. Personally, I dream about being able to think of music directly into a recording system:  a non-invasive brain-machine interface. It turns out that this dream is moving from science fiction to reality (link, link, link). And if we assume a two-year doubling period for cortex sensing resolution, by the early 22nd century our non-invasive brain interfaces will be about 20 orders of magnitude more powerful than today. But will that give us the ability to think music and visual art directly into our computers? Or does it simply blur the line between our brains and our computers, so that the entire paradigm of augmented thinking and collective knowledge is radically shifted? At that point … when we have billions of devices globally networked, and each device is trillions of times smarter than the combined intelligence of all humanity … what will our species become? What will our collective thought processes look like? Personally, I think these kinds of paradigm-shifting social questions are coming sooner than we may realize. And I think there’s both great promise and great risk with the technologies that are emerging. Or as my wife reminds me before my lectures, teach them that the heart is always more important than our technology. Or as Bryan Stevenson said, “we will not be judged by our technology, intellect, or reason. Ultimately, the character of a society will be judged not by how they treat the powerful, but by how they treat the poor.” Nevertheless, somewhere in the future, we will create human-to-machine interfaces that respond and adapt directly to our personal imagery and creative ideas; so that one day just about anything we can imagine will become our art. Millennia Media,  

Lux Aurumque

Some years ago, I produced a recording with the St. Olaf choir, widely recognized as one of world’s premier a cappella choirs. Among the repertoire was a piece entitled “A Boy & A Girl” by composer Eric Whitacre. I had never heard of Eric, but the piece was stunning. Shortly after the recording, I contacted Eric directly. He was thrilled to know that we had recorded his piece. To his knowledge it had never been recorded previously. After mastering, I sent him a copy of the recording and we continued to stay in touch. I later helped him with some technical guidance on his opera (which I understand will play Carnegie Hall this year). So I was thrilled when Kevin Kelly made me aware this week of Eric’s latest project, a “virtual choir” of 185 voices performing his work “Lux Aurumque.” It’s a dream-like composition highlighting Eric’s masterful artistry. Besides being an innovative use of virtuality, I see this work as a breakthrough metaphor of new gathering as global humanity grows in its ability to connect and collaborate directly, at a high level, with decreasing need for locality and mediation. Coincidentally, I took the family to see Carmina Burana today at the Sacramento Ballet — with large choirs straddling both sides of the stage. Please take 5 minutes and be swept away by Eric’s beautiful excursion into the heavens. Eric said this, “When I saw the finished video for the first time I actually teared up. The intimacy of all the faces, the sound of the singing, the obvious poetic symbolism about our shared humanity and our need to connect; all of it completely overwhelmed me.” I want to put on my audio engineer’s hat and point out that I’m hearing a recording quality far higher than what one would expect with a PC-grade microphone plugged into a PC recording system. And I’m not hearing 185 voices. I’m hearing perhaps 50 of the best voices, perhaps doubled, thickened, perhaps pitch-corrected (don’t make choir pitch too perfect or it sounds sterile), widened, EQ’ed, compressed, placed-in-space, single-ended noise-reduced with gobs of electronic ambiance added. This is as much a study in post-production as it is in production, which (to me) doesn’t lessen the impact of the project at all. I think it’s brilliant. Here’s an example of a  

Lester Polfuss 1915-2009

A mentor has passed on today. He was a very young 94. What Les Paul did for twentieth century music and technology cannot be overstated. One of my most inspiring experiences was staying up all night, pretty much until daybreak, talking with Les talk about his life and work, and playing guitar with him. Time will remember you fondly, my  

Generative Rhythms

Here’s a brilliant three-minute video of Bobby McFerrin leading an audience experiment. Something profound occurs at the 45 second mark, and again around 2:00. We are ALL pre-wired with certain shared resonant qualities. Innate musical cognition is one of those qualities. While the universe may seem random at times, it is not.  Nothing on earth or beyond happens outside of a perfectly structured set of organic, natural rules. We all operate within those rules. Whether it be the pentatonic primacy of music to the soul, the centrality of love to the spirit, or the human tendency towards selfishness, pride, and narcissism ” certain universal truths are wired into our collective sentience.” If we are here for a reason, then finding, living, and growing within these generative shared rhythms, while learning to eschew naturally degenerative rhythms, seems like our shared mission. But by fear or thoughtlessness we often stop sharing. We dig ourself into a hole that differentiates ourselves from others (us and them, my tribe / your tribe, my religion, etc.). If we individually lived more in this place of shared mindfulness, those we call “enemies” might start to look strangely like family. “No matter where I’m at, every audience gets this” — Bobby McFerrin World Science Festival 2009: Bobby McFerrin Demonstrates the Power of the Pentatonic Scale from World Science Festival on  

Doc Kauffman

When Dan came home from school today, I noticed he was wearing a Fender Since 1946 tee-shirt. This triggered a memory I hadn’t thought about in years. In the early 1940’s, two men formed an Orange County partnership called Kauffman & Fender. Leo Fender and Clayton “Doc” Kauffman initially made Hawaiian lap steel guitars and later experimented with solid body prototype electrics of the Broadcaster (Telecaster) and Precision Bass. Kauffman left the company and Leo renamed it Fender Electric Instrument Company. When I was growing up in Orange County (circa 68-71), I would bring my 1958 Fender Stratocaster to Doc for “tune-ups”. In reality, my guitar didn’t need a tune-up. I just enjoyed hanging out with him. He had stories about electric instruments dating back to the 1920s, designing guitars for Rickenbacker in the 1930’s, and so much more. He never failed to remind me that the tremolo arm on my Strat was his patented design. Doc (1975 photo above by Rob Cusuman in Doc’s living room) would take me into his garage and walk me through all the steps of setting up a Stratocaster on his workbench – which I already knew. It was the only way I could hear more stories. One day, probably the last time I visited him, he grabbed on old piece of plastic — I think it was one of the newer (at that time) plastic milk cartons. He cut out a piece of plastic in the shape of a guitar pick. Then, with a conductors hole punch, he punched three small holes in the shape of a triangle. He handed it to me and said, “this is the Doc Kauffman pick”. Daniel’s tee-shirt reminded me about the Doc Kauffman pick. I remember years ago putting it in my Yamaha 12-string electric case, along with a broken Kluson low-E tuning machine from the 58 Strat. A moment ago, I dug into my case stack – and found it! Clay “Doc” Kauffman was the technical guru behind the first Fender guitars, some of which remain in production today (P-bass, Tele). I think it’s just too cool that, as a kid, I had a chance to hang with inventor of many of today’s most popular electric instruments. Wikipedia lists Doc’s passing in 1990. Apparently (I’ve not seen it) Doc has a section devoted to him in (Microsoft founder) Paul Allen’s Experience Music Project under the Space Needle in  

The New Geometry of Music

I’ve played guitar since I was seven. That’s about 45 years, with some of those early years actually making a living at music, or playing in HS and college big-bands. I’m a huge fan of prog-rock and fusions of classical music with contemporary instrumentation. When I play, I’m not thinking about music theory, but rather the geometry of sound. I think of the guitar in geometric patterns. I harness geometry to evoke a musical experience. British inventor Peter Davies began thinking about sound geometry in the 1980’s when he patented the Note Tracker, a kind of “slide rule” based on musically geometric patterns, and manufactured briefly in the early 1990s. Later, Peter’s idea grew into a 156 key prototype geometrical controller called the “M-Box.” Around 2005, the M-Box had morphed again into something called the Minima, and finally in 3Q08, a true production-level device called the Axis is now shipping. After 500+ years of keyboard design (organs, harpsichords, piano-forte, etc.), little has changed to the fundamental single-axis western keyboard layout – white keys, black keys, arranged in a long single row of half-tone steps. I believe the Axis multi-dimensional music keyboard signals a catalytic event in musical history. Rather than try to explain the details of this brilliant invention, I would encourage you to watch prog keyboardist Jordan Rudess give a hands-on demonstration. If you have any interest at all in music, this is not to be missed. Welcome to the new era of musical keyboards, and what I think will be the beginning of an entirely new paradigm of musical performance based on the fundamental geometry of western  

Transcendent Trane

My friend Scott sent me this great video of John Coltrane’s Giant Steps with typewriter-like musical notation. For whatever reason, the notes appearing on-screen remind me of emerging theological conversations – a complex string of notes (ideas) put together to make music, to be enojyed as beauty and truth in the moment. Ultimately, music isn’t meant to be captured and studied like a lab animal. Nor, I would offer, is spirit. Both should be practiced and enojyed in the flow of the eternal present. And while certain elements of music, like theology, can be parsed and catalogued, the truly life-changing experience given by both music and spirit transcends rational understanding and touches a common place in us all – the shared  

Laptop Orchestra

Last year, one of our interns was accepted into the CCRMA masters program at Stanford — the Center for Computer Research in Music and Acoustics. This weekend I received an e-mail from him pointing me to videos of a new CCRMA project called the Stanford Laptop Orchestra. “SLORK” is a large-scale, computer-mediated ensemble that explores cutting-edge technology in combination with conventional musical contexts – while radically transforming both. Founded in 2008 by director Ge Wang and students, faculty, and staff at CCRMA, the ensemble requires more than 20 laptops, human performers, controllers, and custom multi-channel speaker arrays (all designed and built by CCRMA grad students) to provide each computer meta-instrument with its own identity and presence. From the SLORK website: The orchestra fuses a powerful sea of sound with the immediacy of human music-making, capturing the irreplaceable energy of a live ensemble performance as well as its sonic intimacy and grandeur. At the same time, it leverages the computer’s precision, possibilities for new sounds, and potential for fantastical automation to provide a boundary-less sonic canvas on which to experiment with, create, and perform music. Offstage, the ensemble serves as a one-of-a-kind learning environment that explores music, computer science, composition, and live performance in a naturally interdisciplinary way. SLOrk uses the ChucK programming language as its primary software platform for sound synthesis/analysis, instrument design, performance, and education. This intern came to my attention at an Audio Engineering Society conference some time ago. I was one of three judges in the “Student Design Competition” (one of the other judges was the late Bob Moog, largely credited as the father of commercial music synthesizers.) His student project was a MIDI controlled car using electronic bagpipes as the controller. The SLORK project gives us a glimpse into the direction of music, and future community dynamics in general. Sound, composition, ensemble, orchestration – virtual collaboration changes everything. Just as the modern orchestra is largely a product of the last 500 years (renaissance, reformation, industrial age), I think future music will be defined by the emergence of real-time global collaboration and virtual modeling. Spiritual community, I believe, will be impacted  

Robert Mondavi 1913-2008

I want to offer a tribute to someone who has made an impact in my life, someone who died yesterday at the age of 94. Robert Mondavi is widely considered the man who put California on the map as an icon of fine wine. I met Robert and Margrit two years ago while engineering NPR’s historic first live classical music webcast, from the new Napa Valley symphony hall. Robert was in a wheelchair and couldn’t hear so well. Margrit, his wife, promised to pass along my deep gratitude for their lifelong patronage of Northern California arts. Of all the projects they’ve underwritten, closest to my heart is the Mondavi Center for the Performing Arts on the U.C. Davis campus. The new concert hall could not have been built without their spectacularly generous gift. Mondavi’s new hall represents a pinnacle in world-class acoustical design. I’ve been producing and engineering classical music in N. California for almost 20 years. Every chance I get to record in the new hall is like sneaking into a little bit of heaven. I consider this “my hall.” I was there often during construction, and made the first commercial recording in Mondavi Hall (Delos DE3360). Here are excerpts from that stunning performance. MP3-a MP3-b MP3-c Mondavi’s sons have since sold off the family business to a megalcohol conglomerate, but Robert’s legacy will live on, touching and enriching the lives of millions to come. Thank you Robert Mondavi. From the Chancellor of U.C. Davis In June 2004, UC Davis presented Robert and Margrit Mondavi with the UC Davis Medal, the highest honor bestowed by the university. But what I will treasure most about our friendship with Bob Mondavi was his sense of destiny. I remember him saying once, “If you wish to succeed, you must listen to yourself, to your own heart, and have the courage to go your own way, to find the right direction.” There’s no doubt that Mondavi accomplished his mission – to the betterment of the university, the wine industry, agriculture, the state of California – and beyond. And the Mondavis’ belief in UC Davis emboldened each of us to reach even higher. Through his leadership, Mondavi truly opened a new era of opportunity for UC Davis. He was convinced that the sciences and the arts were essential companions. He reassured each of us – no matter our calling in life – that we were capable of and responsible for creating a magnificent and enduring legacy. – Larry N. Vanderhoef , Chancellor of the University of California. Also this week.. I received a photograph from Portland Oregon-based artist Rebecca Gray. Her new painting of a rose is so stunning I wanted to share it. And finally, with the Jill Taylor post approaching 300 comments, I leave you with the results of my brain lateralization test: Brain Lateralization Test Results Right Brain (58%) The right hemisphere is the visual, figurative, artistic, and intuitive side of the brain. Left Brain (42%) The left hemisphere is the logical, articulate, assertive, and practical side of the brain Are You Right or Left Brained? personality tests by