March 1998
In Search of a Graceful Death
Today's Gear Means All or Nothing


There's nothing like having a bunch of students around to help you find the weaknesses in your gear. Those of us who have worked with audio hardware and software for many years develop an instinct about what we can or cannot do with our tools. We know, for instance, that on a tape deck built before the days of transport logic, if you quickly press all the transport buttons in succession, the result will be several yards of tape spilling all over the floor, or wrapped around the capstan, or both. We know that when a software program gives us a "You're running out of memory..." message, we don't just save what we're doing, we save everything we have been doing and get the hell out, because we know a system crash could be lurking around the next mouse click.

So we're careful. But students and other newcomers to our field don't necessarily know what they're not supposed to do--the air is fresher around them--and so they find ways to screw things up that we older, wiser, more cautious, oft-stung old farts would never stumble across. They figure, in their naïveté, that things are supposed to work the way their makers say they do--and so they are constantly breaking stuff.

And there's nothing like the last week of a semester to bring this point home. This past semester, on the Saturday evening preceding the week my students were to present their big term projects, as they labored frantically around the clock to finish them, I got one of those phone calls I am constantly dreading: A tape deck had eaten a student's master tape. Of course, I won't mention any brand names, but it was a modular digital multitrack deck that uses a transport originally designed for video.

So now her project was in danger. Not only did she fear that all of the music on the tape would have to be redone, but she was even more upset about something else: No one else in the class could work on their projects either until the masticated cassette was removed from the deck. I doubt the problem was caused by incompetence: She is among the small, elite group of students who are trained to assist our facilities director in maintaining all the studios. But while she can be counted on to find an errant normal in a patchbay or isolate a noisy fader in a console, pulling apart an MDM was not part of her training. It wasn't part of mine, either, so I couldn't be of much help. Our facilities director lives 30 miles away, and even if his wife hadn't been in labor, it's doubtful we could have convinced him to come in on a Saturday night and fix the thing; nor would he have felt comfortable walking her through the job over the phone.

As it happens, we have one of the area's top technicians on our payroll, so I told the student to call him. He was very sympathetic over the phone, but we don't pay him enough to come in on a Saturday night either--he lives 30 miles the other way, and all of his time at school that week was dedicated to hearing his students' term projects. All he could do was suggest she take an identical deck out of another studio, where it was also in round-the-clock use by a different group of students, and patch it into the studio where she was working. And, oh yes, forget about the original tape--it's toast.

So for the next five days, until the technician could get the thing into his shop, extricate the tape and figure out what the hell went wrong, we had hot and cold running students plugging and unplugging audio, timecode, MIDI and control connectors from one overused multitrack and running it back and forth between two studios, while frantically rearranging their schedules to make sure that the same deck didn't need to be used in both studios at the same time.

The next day, Sunday, and I'm not making this up, an audio editing program the students were using stopped working in one of the studios (not the one with the dead tape deck). No error messages, no crashes. All of a sudden it would just freeze halfway through the boot-up process. I came in the day after it started happening and spent two hours reconfiguring system extensions and prefs files, running disk analysis programs and reinstalling the program--everything short of initializing the hard disk and starting over. Nothing worked. They'd have to do without. There was no way they could use the software in another studio, since that would have meant disconnecting the various hard disks and dragging them around, and I wasn't going to go there. So I put up a little sign next to the computer: "Sound Patootie [not its real name] has stopped working. Deal with it."

They did, and in the next couple of days, they were somehow all able to finish their projects, and they were all good, despite the insane obstacles that were thrown their way at the last minute. They all managed to get time with the multitrack without forcing deprivation on other students. They all found ways to accomplish what they needed in terms of sound design without the software--either they found other software that could do the job, or they just shrugged and made what use they could of what they already had.

So I was proud of them, but I was still mad as hell. And what made me the most mad was realizing that there is no "graceful failure" built into our tools anymore. Just the way digital is all ones and zeroes with nothing in between, today's gear, when it goes bad, doesn't do anything halfway: It's dead, Jim.

Return with me now to another time, say, 20 years ago, and another place: a new studio in a far-away country with a brand-new console, brand-new 24-track tape deck and brand-new stereo plate reverb. The studio was in a very hot climate, and the air conditioning had been installed by people whose ineptitude was exceeded only by their dishonesty, so the equipment broke down a lot. We were on the phone constantly to the dealer, who, not eager to send someone several thousand miles on a service call, was working just as hard as we were to solve the problems.

One of the output preamps on the reverb failed. First, we patched around it, then we managed to wire the two transducers on the plate to the single working output, so it ended up sounding pretty good, although it was in mono. Another day, the monitor section on the board started to get noisy. While we waited for a new op amp, we borrowed a cue bus that we weren't using much and ran that to the monitor amps. The multitrack Dolby rack started to give out, one channel at a time; we patched around the dead channels and recorded without noise reduction if we had to. The 24-track mysteriously stopped erasing when we were overdubbing on certain tracks; we went to other tracks and finally traced the problem to faulty relays on the channel boards. We pulled the boards out and put the good ones in the channels that we needed to overdub on. In other words, as things deteriorated around us, though we weren't very happy about it, we were quite capable of carrying on, and the clients barely noticed.

Imagine that happening today. Your digital reverb freaks out and garbage starts spewing from the left speaker. What are the chances you'll be able to still get signal through the right channel? How about a multichannel A-to-D converter? If one channel fails, will you be able to patch around it and use the others? If an effects bus on your all-in-one digital mixer goes haywire, do you think the other five buses will continue to work just fine? And what happens when a channel goes down on a modular digital multitrack? Can you still record and play on the others?

The answer to all of these questions is a resounding "No!" When today's equipment craps out, it does so spectacularly and with great finality. There's no such thing as a "workaround" any more. If you need to make sure that something is going to work all the time, you buy two of them. One studio I know advertises itself as "24-track," despite the fact that it owns four of the previously mentioned MDMs. The owner knows that at any given time, it's almost certain that one of his decks will be in the shop.

Why has this happened? It's a sacrifice we've made in the interests of economy. Manufacturers are under constant pressure to come out with gear that packs more punch into the same amount of physical and financial space, and redundancy, field-repairability and user-oriented troubleshooting and diagnostic procedures just aren't part of the equation.

Audio manufacturers are becoming increasingly dependent on custom VLSIs, which, as time goes on, get continuously more sophisticated and include more functionality. That brings costs down but makes it impossible for a unit to fail partially or gracefully. Putting eight D-to-As on a single chip means that the manufacturer only has to buy one chip, not eight. But if one channel on that chip fails, there's nothing anyone can do to salvage the other seven.

Still, even doing repairs at the chip level has become rare: Most manufacturers don't want to have a bunch of proprietary replacement chips floating around outside their factory, where they could be analyzed by the competition. So whole assemblies, even if they're otherwise perfectly functional, end up being replaced.

A lot of this attitude, not surprisingly, is borrowed from the computer industry, which in the early '80s adopted the "don't-fix-it-replace-it" mentality so as to keep the costs of training field technicians down, and to keep tight control over crucial components like operating system ROMs. It has meant repairs can get done quickly (assuming the replacement boards are available), but it has also meant a lot of people end up paying for very expensive assemblies when a cheap diode or capacitor is really all that was needed. Unfortunately, it means really useful service manuals are becoming a scarce commodity, and quick and clean component-level repairs are simply not an option.

Other factors are at work, too. Product development cycles have become so short that there's no time for long-term failure testing, to say nothing of developing practical ways for users to respond to failures. Often, a service manual for a new device doesn't become available until long after the product is out there, as manufacturers are reluctant to invest in the creation of the service manual until they're sure they have a hit on their hands. How many devices do you have that display "error codes" that are explained nowhere in the documentation? One of the reasons that our esteemed editor George Petersen's book on modular digital multitracks has been a best-seller is that for a long time it was the only source of information about those codes for that incredibly popular family of gear.

On the software side, operating systems of desktop computers have become so complex, and users have so many options even with a computer as supposedly simple as a Macintosh, that real-world testing and meaningful diagnostic procedures for dealing with crashes and incompatibilities are extremely difficult to develop.

And even if we try to be cautious, we're pushing our tools harder than ever. The video transport the MDM is based on was not originally designed to be constantly run at the high speed required for multitrack digital audio, and so component and head wear are accelerated, and tolerances must be tighter. The units are also caught in a kind of netherworld between professional and consumer--as video decks, they were originally designed to function both in professional environments (where one presumes they would be carefully maintained by trained technicians) and in consumer environments, where they would not be pushed very hard. But as audio decks, at least in the personal and project studios for which they were originally targeted, they often do not get the attention they need, while they are being pushed just as hard as if they were in a video-editing suite.

On desktop computers, we're asking for DSP and hard disk performance that few others in the industry demand. The video and graphics developers who use PCs as front ends for their systems are mostly smart enough to provide their own hardware to do the heavy lifting, but the trend in our industry is going the other way, to use as much "native" processing as possible, to keep costs down.

There's no easy solution to this problem. As long as we customers demand high performance at a low price, and as long as the manufacturers are obsessed with getting their next generation of product out the door ahead of the next guy's, the cost of field-serviceability, of usable real-world troubleshooting guidelines, and of "graceful death" will be too high to be included in the products we use. But manufacturers could do more to acknowledge that things do indeed fall apart and make more of an effort to address the problems that users will undoubtedly encounter. Could I teach a student to get a wayward tape out of a digital multitrack deck? I bet I could, but the maker of that deck has to let me in on the secret first, and also has to make it a little easier to get inside the thing. If more manufacturers were to trust us, cautious bunch that we are, to be able to deal with some of these problems on our own, there could be a lot less down time in our studios. Of course, then we'd only have to buy one of everything.

Paul Lehrman is trying hard to age gracefully. Meanwhile, he offers congratulations to Bill and Lisa, and thanks to his ever-suffering students for being such good sports.

These materials copyright ©1998, 2001 by Paul D. Lehrman and Intertec Publishing