How Berkeley hackers built the Net’s most fabled free operating system on the ashes of the ’60s — and then lost the lead to Linux.
By the time Bill Joy arrived in Berkeley, Calif., in 1975 to attend graduate school, the fabled capital of leftist radicalism was a bit ragged around the edges. If the 21-year-old programming wunderkind had glanced at the headlines blasting out of the local alternative weeklies, he might have wondered just what kind of insane mess he had gotten himself into. In San Francisco, Patty Hearst was on trial for a bank robbery committed while the newspaper heiress was toting machine guns for the Symbionese Liberation Army. In Oakland, the Weather Underground botched a bombing of a Defense Department building. Even the reliable bugaboo of CIA recruitment on the University of California’s Berkeley campus failed to generate more than a token protest.
Berkeley was burned out, its radical energy wasting away in infantile terrorism, conspiracy theorizing and drug overdoses. The Free Speech Movement that had galvanized the university in the ’60s belonged to another geological age. Ken Thompson, co-creator of the Unix operating system, graduated from Berkeley in 1966 with a degree in electrical engineering. He returned to the university from Bell Labs for a sabbatical in 1975. But the campus on which he had once walked to class through clouds of tear gas had changed. That year, says Thompson, Berkeley “had turned into the most politically apathetic place I’d seen.”
But it was the right place for Joy. “He never looked at those [alternative] papers,” says John Gage, a close friend of Joy’s during the Berkeley years and later at Sun Microsystems, a company co-founded by Joy. Today, Joy calls himself a “staunch Democrat” and has recently carved out a new niche as a techno-skeptical doomsayer, but in the ’70s he was, by his own description, “not an activist.” Joy chose to attend UC-Berkeley instead of Stanford or MIT not because he was attracted by its politics or countercultural reputation but because the computer science department’s hardware was so obsolete that he figured he’d have no choice but to confine his research efforts to studying computing theory — which was exactly what he wanted to do.
But theory turned out not to be Joy’s forte. He started hacking code and never stopped. “His goal was to build something that worked,” recalls Gage. And so he did. During his seven years at Berkeley, Joy and a few other graduate students and staff researchers spearheaded an intensive software development effort that culminated, most famously, in a radically improved version of AT&T’s Unix, known simply as Berkeley Unix or, more commonly, as BSD,* for Berkeley Software Distribution.
Talk about your killer apps! Berkeley Unix worked so well that DARPA* chose it to be the preferred “universal computing environment” linking together Arpanet* research nodes, thus setting in place an essential piece of infrastructure for the later growth of the Internet. An entire generation of computer scientists cut their teeth on Berkeley Unix. Without it, the Net might well have evolved into a shape similar to what it is today, but with it, the Net exploded.
How did the small group of Berkeley programmers pull off such a feat? Well, for one thing, there was Joy, a programmer around whom legends accrue like so many iron filings stuck to a magnet. He could read at age 3, play chess at 4 and, during his oral exams, invented a “sorting algorithm”* on the fly that so stunned his examiners, one of them later compared the experience to “Jesus confounding his elders.”
But too much focus on Joy, a favorite target for business magazine hagiography, obscures the larger picture. Berkeley’s most important contribution was not software; it was the way Berkeley created software. At Berkeley, a small core group — never more than four people at any one time — coordinated the contributions of an ever-growing network of far-flung, mostly volunteer programmers into progressive releases of steadily improving software. In so doing, they codified a template for what is now referred to as the “open-source software development methodology.” Put more simply, the Berkeley hackers set up a system for creating free software.
BSD itself wasn’t originally free software. Joy sold it, with the University of California’s blessing, at a nominal cost only to people or institutions that had already purchased licenses permitting them access to the source code of AT&T Unix (although, in practice, Joy’s efforts to verify whether would-be buyers really did own licenses may not have been overly vigorous). But in spirit, Berkeley Unix was indeed free: As Dennis Ritchie, Thompson’s collaborator in creating Unix, observes, anyone who wanted to hack on Unix usually had access to the source code, one way or another. And if those hackers sent their modifications to Berkeley, and they were deemed good enough, they became part of a code base maintained by programmers who wanted nothing more than for their software to be widely used, for as low a cost as possible.
Berkeley Unix has morphed through multiple phase shifts since its inception some 20 years ago, from the Joy-dominated era of the late ’70s and early ’80s to the more collaborative period that began after Joy’s departure to Sun in 1982. But in the early ’90s, after a bitter confrontation with AT&T, BSD finally did become “freely redistributable,” and descendants of BSD — led by FreeBSD,* but also including OpenBSD* and NetBSD* — are vigorous participants in the contemporary battle for operating-system supremacy. Yahoo, arguably the world’s busiest Web site, runs on FreeBSD. And yet, despite its proud heritage, BSD’s current status doesn’t quite match up to its early fame. A victim of schisms within its own developer community, bruised by the battle with AT&T and wounded by the defection of Joy to Sun, BSD is currently a small player, especially as compared with Linux. Linux-based operating systems have seized the public imagination.
BSD patriots argue that the battle is far from over, that BSD is technically superior and will therefore win in the end. That’s for the future to determine. What’s indisputable is BSD’s contribution in the past. Even if, by 1975, Berkeley’s Free Speech Movement was a relic belonging to a fast-fading generation, on the fourth floor of Evans Hall, where Joy shared an office, the free-software movement was just beginning.
The connection between the two movements is clear, if not direct. By demonstrating the power of cooperative software development, and by strengthening the software backbone of the Internet so it could further nurture such development, BSD helped enable the creation of a medium that will do more to spread free speech than anything hitherto constructed. Power to the people, from the code.
Many nice, upper-middle-class Berkeley backyards boast a redwood patio, possibly a hot tub, perhaps a vegetable garden complete with thriving rosemary bushes and marauding raccoons. Bob Fabry’s backyard has a radio tower that wouldn’t look out of place at a major Air Force base. Capable of telescoping upward to a height of 100 feet, and built mostly by Fabry himself — the hub that rotates the antenna was scavenged from a 1940s airplane propeller — the radio tower looms beside Fabry’s home like a not-so-miniature Eiffel Tower. One look at it and you realize you are in the presence of a very dedicated geek.
Fabry takes his ham radio “hobby” seriously. He once even helped organize a trip to the uninhabited wilderness of Heard Island, about 1,000 miles north of Antarctica, just to set up a ham radio station for a few days so amateur-radio enthusiasts all over the world could enjoy the pleasure of exchanging radio signals with the faraway station.
But Fabry’s most impressive achievement scales far beyond his tower or his expeditions. He is the Berkeley computer science professor who orchestrated the creation of Berkeley Unix. Not that he wrote a lot of code — that honor belonged primarily to Joy and the other members of Fabry’s Computer Science Research Group, an all-star band of programmers whose roster included names like Sam Leffler, Kirk McKusick, Mike Karels and Keith Bostic. But while Joy and others were hacking for 36 hours at a stretch, improving file systems,* networking performance, memory utilization and a hundred other arcane but crucial elements of Unix, Fabry was running interference — maneuvering through the formidable bureaucracies of the University of California and AT&T, dealing with departmental politics and backbiting and, most important, writing the grant proposals that brought a steady flood of DARPA money into Berkeley.
Fabry was personally responsible for bringing Unix to Berkeley. His reasons were simple, and offer an early example of the pragmatist bent that has characterized BSD development ever since.
Unix was cheap. AT&T had been forced to practically give it away for free by government order. But Unix was also, fundamentally, a hack designed to work on cheap hardware. Back in 1969, Thompson wanted to get a computer game called Space Travel working on a castoff PDP-7. So what did he do? He wrote an entire operating system that made it possible. Kind of like using a nuclear missile to hammer a nail — but then that’s often standard operating procedure for obsessed hackers.
Fabry was entranced by Unix’s affordability, along with the ease with which it could be adapted to different computer hardware. In addition to his academic research focus on operating systems, he was involved in setting up computing resources for the UC-Berkeley student body. In the mid-’70s, this could be an expensive proposition. Typically, operating systems that would allow multiple users of a mainframe* to work at individual terminals were designed only for extremely expensive computers. The costs per user could easily reach $50,000 a terminal, which made such systems impractical for pedagogic purposes. But Unix, used in combination with a relatively inexpensive PDP-11 from DEC, ended up costing closer to $5,000 “per seat.”
Even better, a $99 license fee bought you access to the Unix source code — to the blueprints, the magic recipe, the key that unlocked all hidden mysteries. For researchers, teachers and students, this was priceless. Researchers working on cutting-edge operating system technology could experiment with already existing source code and modify it for their needs; students who wanted to learn how an operating system really worked could find out by getting their hands dirty with the code. Duane Adams, the DARPA contract “monitor” who administered the Berkeley Unix contracts, notes that the availability of source code was an explicit reason why DARPA chose Berkeley Unix instead of contending aspirants like DEC’s* VMS.* Never mind that VMS had been designed from the bottom up for the DEC VAX computers that were the most popular hardware for Arpanet nodes; VMS was a closed, proprietary system. You couldn’t get in and muck about, so it just wasn’t attractive to researchers.
DARPA was also recognizing reality. Prominent researchers, hungering for the magnetic tapes carrying Berkeley’s latest distributions like so many desperate junkies, demanded that DARPA adopt Berkeley Unix because that’s what they were already using.
“What was driving DARPA,” says Fabry, “was that almost all of their contractors were telling them that they were running Berkeley Unix and it was superior to anything else available.”
As one popular explanation has it, Unix’s source code became widely available through a lucky accident — as an unanticipated consequence of the consent decree that forbade AT&T from commercializing its non-telephony-related inventions. But that’s only a part of the story. Unix was always more than just a bit player in a showdown between the world’s largest government and the world’s biggest corporation. Unix was, at least in the mind’s eye of scientists like Fabry, “a thing of beauty.” And from the very beginning, Unix benefited from a communal vibe that spread directly from its creators, Ritchie and Thompson.
Fabry recalls grasping the hidden wonders of Unix one week in 1975 when Thompson conducted a “reading” of Unix over several successive nights.
“The first meeting of the West Coast Unix User’s Group had about 12 or 15 people,” recalls Fabry, a mild man, now 60 years old, who clearly delights in his 25-year-old memories. “We all sat around in Cory Hall and Ken Thompson read code with us. We went through the kernel* line by line in a series of evening meetings; he just explained what everything did … It was wonderful.”
The reading of the code: Thompson’s primeval act of deconstruction was an initiation into the Unix cabala, a ritual passing down of code lore. Fabry may have brought the first physical manifestation of Unix to Berkeley, but Thompson’s reading embedded it in Berkeley’s soul. Eric Allman, *” who was later to write sendmail,* the open-source-software mail transport program that still shuttles the vast majority of Internet mail across the Net, was an undergraduate at Berkeley when he attended the readings. He still has his marked-up “listings,” reams of cheap, flimsy computer paper with notes scribbled on them, detailing the obscurities of the C programming language and other Unix arcana.
“The really bizarre thing is that Ken Thompson did a free tutorial on Unix kernel internals,” recalls Allman, “and everyone fit into a rather tiny room.” Today, you’d need to rent a ballroom.
Fabry marched against the Vietnam War while he was a graduate student in Chicago, and notes proudly that in his entire 12-year tenure at Berkeley he never once wore a tie. But although some historians have later described the Berkeley hackers as freedom fighters — especially in the context of their battle with AT&T (which came well after Fabry had retired) — neither Fabry nor the hackers themselves saw what they were doing in such explicit ideological terms. But when I ask Fabry if there was ever a moment when the goal crystallized in his head that software should be free, he turns the question around:
“Where did it come from that code should cost money? I think that’s the fair question,” says Fabry. In the mid-’70s, most programmers had grown up in an era where software was usually included with hardware and not considered a separate revenue source or proprietary intellectual property. Joy saw his work on Unix as research, to be shared with the rest of the academic research community the same way professors had been sharing the fruits of their labors for thousands of years. Ritchie and Thompson wanted their software to be used — they did everything in their power to help the Berkeley programmers fix bugs and make improvements.
None of them saw himself as a crusader. But a trickle of idealism still occasionally leaked out.
“I think the spirit in which we were putting this all together was much the spirit that was picked up later by the Free Software Foundation and the various people who were trying to build ‘software for the people,’” says Fabry. “The idea is that there is no duplication cost for software, so it ought to be basically free, and we were all working together to try to produce this ideal system that we would all love to have, and love to be able to use ourselves. That was the goal of a lot of people, and of course that was the original goal of Ken Thompson and Dennis Ritchie in starting Unix.”
Fabry retired at age 43, tired of the DARPA treadmill and eager to focus his energy on his ham-radio tinkering. But Berkeley Unix’s record of success still thrills him.
“Berkeley Unix was clearly the most successful university software project that has ever gone on,” says Fabry in a rare moment of assertiveness, before backtracking slightly. “I don’t know, I haven’t been keeping up since 1983 and maybe there’s been something since then, but I believe that that was true at the time. We had literally thousands and thousands of installations, and a whole generation of computer science students all around the world grew up on Berkeley Unix. It set a standard for operating systems that people are still having trouble doing better than. It was also the first efficient networking solution; for years it was the only game in town, the basis of Internet development. It really was one of the things that the people who made the Internet what it is today built on. There were battles that had been solved that didn’t have to be solved again in order to do whatever new part that they wanted to do.”
“Bill codes like a demon.” — John Gage
“His code was ugly.” — Kirk McKusick
“Bill Joy was a fabulous marketer.” — Eric Allman
“Bill was superb. He was the epitome of what one would like to see in a graduate student. — Bob Fabry
“He had an infectious enthusiasm about him, where he would just get the people around him to do stuff. And he had an incredible drive to get his software out so that other people would use it.” — Kirk McKusick
“Berkeley Unix was the work of many talented developers. Bill Joy’s particular genius was in integrating the work of these many contributors from many different organizations.” — Rob Gurwitz
“Bill’s really smart.” — Sam Leffler
Gage’s eyes twinkle as he recalls one of his favorite Bill Joy stories. The scene, he says, is in a boardroom high up in a building overlooking Washington, D.C. The time is the early 1980s. In attendance are some representatives from DARPA, some employees of BBN* (a Boston company that received the original DARPA contract to build the Arpanet) and a few Berkeley hackers, including Joy.
At issue was an annoying problem that had been bothering DARPA. DARPA had given Berkeley a major contract to enhance Unix so that it would be suitable for DARPA’s network of research sites. It had also decided that Berkeley Unix should incorporate TCP/IP,* a specification for how Arpanet machines would interconnect. Devised by Vinton Cerf and Bob Kahn, TCP/IP (Transmission Control Protocol/Internet Protocol) was — and still is today — the basic method by which computers talk to each other across the Internet.
But DARPA had given BBN the contract to implement the TCP/IP protocol,* to write the all-important TCP/IP “stack.”* Joy had been instructed to plug BBN’s stack into Berkeley Unix. But Joy refused to do so. In his opinion, BBN’s TCP/IP wasn’t good enough. So he wrote his own high-performance TCP/IP stack.
As Gage tells it, “BBN had a big contract to implement TCP/IP, but their stuff didn’t work, and Joy’s grad student stuff worked. So they had this big meeting and this grad student in a T-shirt shows up, and they said, ‘How did you do this?’ And Bill said, ‘It’s very simple — you read the protocol and write the code.’”
“That really frosted the BBN guys.”
Well, sure. In programming lingo, a flat statement like “Read the protocol and write the code” is, to borrow some modern slang, a major dis. But did Joy really say the words? And did BBN’s code really not work?
No and no, says Rob Gurwitz, the BBN programmer who wrote BBN’s implementation of TCP/IP. Gurwitz says he was at all the DARPA steering committee meetings that handled TCP/IP matters during that era, and he doesn’t remember ever hearing Joy make such a statement. Gurwitz, who says he worked closely with Joy and Sam Leffler on the integration of TCP/IP into Berkeley Unix, also says Joy’s version of TCP/IP was not a direct replacement for BBN’s code. Joy’s stack was designed to maximize performance over local area networks that had wide bandwidth connectivity — like an Ethernet* network designed to serve an entire university campus, for example. Gurwitz’s version was built to operate on the much narrower 56Kbps telecommunication lines that made up the Arpanet’s backbone.
Nonetheless, the Berkeley hackers were (and are) convinced that their implementation was superior, and they continued to resist all attempts by DARPA to force them to include the BBN version in Berkeley Unix, even though, according to Gurwitz, several DARPA sites continued to use BBN’s version for years. As is the case with most programming disputes, the deeper one delves into the TCP/IP spat, the more “Rashomon”-like the search for truth becomes.
So why is the squabble important? For at least three reasons.
First, the incorporation of TCP/IP into Berkeley Unix can be, and often is, singled out as the most important innovation that made the Internet function efficiently. Joy and other Computer Science Research Group* veterans argue that their version of TCP/IP was crucial, because only it was technically good enough to satisfy researchers who wanted to communicate with each other and get work done on their local networks as well as on the Internet.
The TCP/IP stack, one could argue, was the original Promethean gift of fire to the mortals of the Net. And when the Internet suddenly boomed in the ’90s, Berkeley Unix scaled up right along with it — a testament, says Kirk McKusick, to the quality of its design. To this day, BSD advocates contend that the networking performance of BSD, which can still be traced all the way back to Joy’s TCP/IP code, outclasses the best that Linux-based operating systems can do.
Second, the TCP/IP stack played a deciding role in settling the legal battle between AT&T and the University of California. The breakup of the AT&T monopoly in 1984 finally permitted AT&T to commercialize Unix. For years, as Unix became the preferred language of the Net and academia, AT&T had steadily increased licensing fees from the original $99 all the way up to $250,000. But AT&T wasn’t the only interested party. In the early ’90s, BSDi,* a spinoff of Berkeley’s CSRG, started selling its own version of Berkeley Unix, and the University of California had been selling its version for years. In 1992 AT&T sued both the University of California and BSDi, claiming that BSD Unix included proprietary AT&T code.
Unfortunately for AT&T, the version of Unix that the company was then pushing, System 5, turned out to incorporate large chunks of code originally written by BSD hackers — including the TCP/IP stack. Berkeley released all its code under an extraordinarily liberal license — basically, users could do anything they wanted with BSD code as long as they retained the University of California copyright. But AT&T had stripped the UC copyrights and begun marketing the software as its own. Hackers like McKusick were peeved.
“We had written this code and they were claiming it was theirs,” says McKusick, “and that they had the rights to it, and we just flat out didn’t believe it. And it pissed us off that they were basically taking our work, that they hadn’t paid a penny for, but that they had made money off of because of all the damn licenses that they had sold, and they were now trying to claim it was theirs.”
Fuente: https://www.salon.com/2000/05/16/chapter_2_part_one/