MICHAEL POWELL WASHINGTON POST STAFF WRITER Sunday, April 16, 2000 ;

SAN FRANCISCO -- A WHITEWASHED SPANISH CHURCH, and bay window rowhouses rising higgledy-piggledy up a hill toward an azure sky. A hundred Chinese ladies raising silken knees in balletic tai chi. Morning's cool shadow slanting across the park.

On a green bench a high priest of silicon sits and talks heresy.

Heresy about a golden Silicon Valley and the scientific Darwinism that runs amok there. About the technocratic culture he helped birth, and how it just might be the death of us.

"We are dealing now with technologies that are so transformatively powerful that they threaten our species," he says. "Where do we stop, by becoming robots or going extinct?"

His name is Bill Joy, a tall, 45-year-old man with the dreamy intensity of a scientist. He is a co-founder of Sun Microsystems, one of the world's most profitable companies. A widely respected computer expert who one night sat in an Aspen diner and on a place mat designed a software language that allows your refrigerator to talk to your car.

Now he watches as three fingers of doom arise from the world he helped shape. He looks at modern genetics, robotics and the ability to build molecule-size machines and sees a technology exploding beyond our ability to control it. He sees man-made plagues and malevolent robots overshadowing even the nuclear era in the capacity for utter destruction.,p> He warns that our scientists, his friends, must relinquish their Promethean fire.,p> "I'm sorry but there are certain technologies so terrible that you must say no." He leans forward on the bench. "We have to stop some research. It's one strike and you're out." The broader crusade he and a handful of influential scientists are embarked on is no less imposing: to challenge a scientific culture soaring too close to the sun on wings of wax. It's a culture where pursuit of "truth" is all, where researching the next hot thing is what matters.

"For Aristotle, an argument based on a poem was as valid as one based on science. We've lost that. I don't sense in this community that an ethical, spiritual-based argument carries nearly as much weight as a capitalist imperative or the notion that progress is the ultimate. That whatever happens happens. "It's scientific fatalism and it could be fatal for us." Apocalypse now? The mind manufactures caveats and yes buts and well, whatevers. The stock bull runs the wired decade, dot.com-ism multiplies, the Cold War has passed . . . and Joy's talking death by technology? The temptation is to say: Cut the dark talk and pass the IPO.

But that would reckon without our cultural schizophrenia. Popular culture celebrates its bigger, faster, 999,000-megabyte future even as it devours stories of cyber-terror and robo-cops, dystopian dreams and fears that chill.

American scientific culture pays less heed to its shadows. In the stock-optioned ecosystem of Silicon Valley, where scientific brilliance and venture capital leapfrog toward the horizon, the talk is of satisfying our deepest hungers. Scientists would stamp out disease, use computers to make us hear and see and feel better, maybe even beat back death itself. And they recoil when Joy, who was co-chairman of a presidential commission on information technology, talks of barring some research.

"The evolution of technology just continues the . . . explosion of biological evolution," says Ray Kurzweil, a cutting-edge scientist and author of "The Age of Spiritual Machines." "In the 21st century we will make 20,000 years of progress." Perhaps. But maybe the four-century-long process that freed the modern scientific mind from the bonds of religion, a philosophical string that runs from Copernicus to Descartes, has had a curious effect. Maybe our culture has made science so powerful that it's replaced God at the center of our universe.

"The technological impulse to break free from the constraints of nature and emancipate the human being is noble and a central archetype of mankind," says Richard Tarnas, the philosopher and author of "The Passion of the Western Mind." "The great danger of our time is that the quest hasn't been matched by a moral and psychological awareness of our limits." Joy spoke at Stanford University a few weeks back, at a symposium asking: "Will Spiritual Machines Replace Humanity by 2100?" Midway through, Joy peered quizzically at scientist Hans Moravec, a husky man who has established the world's largest robotics research program at Carnegie Mellon. Moravec is convinced he can create a conscious robot in 30 years, and is quite enthused by the prospect.

"Isn't there a point at which you worry?" asks Joy.

"Well," Moravec allows, "I do worry about intelligent robots getting strange ideas . . ." Gray Goo

Joy traces the kernel of his discontent deep into the technological cocoon, of which he is a creature. He recalled talking with Kurzweil about computers in the autumn of 1998. It is theory and faith among denizens of the digital world that computer chip speed doubles every 18 months. Joy always believed this rate would slow down as science rubs against the boundary of the physically possible. And he drew comfort from knowing there's a limit to the power of computers. Now Kurzweil told him that molecular-level advances would allow for computers a million times faster and smarter by 2030. So a world of possibilities revealed itself, and cast Joy into darkness. He's married, has two children and a beautiful home in Aspen; he saw the clouds moving in.

He wrote of his epiphany in a much-noticed essay for Wired magazine: "It was only [then] that I became anxiously aware of how great are the dangers facing us in the 21st century. . . . We have yet to come to terms with the fact that the most compelling 21st century technologies--robotics, genetic engineering and nanotechnology"--pose a more dangerous threat than any past technologies. These computers and genes and micro machines, Joy writes, "share a dangerous amplifying factor: They can self replicate: A bomb is blown up only once, but one bot can become many, and quickly get out of control."

Genetic research is the most advanced. Already scientists splice the fish cell with the strawberry, the salmonella bacterium with the sperm cell. (What is not known is how these new cells and bacteria will behave once released outside the lab; the only certainty is that cells and bacteria replicate and mutate.) A bit further into the future, scientists hope to build sub-microscopic nanotech machines so small they can fly by the billions into the atmosphere or cruise the bloodstream like galleons. Scientists envision computer robots so smart that, decades or centuries into the future, a metallic eye might blink and a silicon mind awaken.

The research fueling this micro industrial age is hurtling forward. It is not taking place in the crucibles of the nuclear age, the government-funded labs where scientists held information confidential on pain of imprisonment. That's the dinosaur age. The market rules now. The venture boys erect a thousand elegantly outfitted labs, and the drive for riches and pure research attracts scientists of the highest caliber. Information is swapped on the Web and the ethos is libertarian and triumphal: Leave us alone and we will transform your world.

To its devotees, this is the technological great leap forward. Joy divines a darker shadow. "Silicon Valley is like the Burgess Shale of the Darwinian experiment," he says. "Every scientist has become a new phylum and everyone is running around thinking they'll be the biggest mammal. They all want to be billionaires and no one wants to think about implications of their research." Joy acknowledges that new technologies might cure diseases and increase crop yields, and bring great surcease from suffering. But at the crossroads of knowledge and the individual genius, the threat to humanity metastasizes.

Ruin might come from a laboratory experiment gone awry. Or a college student reading a Web recipe and mixing test tube viruses. Or the genius psychopath, shooing a boutique virus or self-replicating robots out the door. It is what Eric Drexler, the father of nanotechnology, calls the gray goo problem: that someone might, inadvertently or otherwise, manufacture a ravenous and invisible man-made machine or bacterium that would outstrip natural competitors and, propelled by wind and rain, turn life to dust in a matter of days.

"There is a tremendous, market-driven haste to get into invisible technologies that are unstable and dangerous to life," says Tom Valovic, a research manager for International Data Corp. and author of "Digital Mythologies." "It requires this religious leap of faith that science knows best." Joy has renounced work on nanotechnology, seeing it as too perilous. But he acknowledges that software he invented now enables one computer to talk with another, and so might help another scientist in another lab create a menace.

"I may be working to create tools that will enable the construction of the technology to replace our species. How do I feel about this? Very uncomfortable."

I Am Become Death,b>

Men sometimes speak as though the progress of science must necessarily be a boon to mankind, but that, I fear, is one of the comfortable nineteenth-century delusions which our more disillusioned age must discard. --Bertrand Russell in 1924

Many scientists respect Joy, even as they dismiss his fears as primitivist. From Faust to Mary Shelly to Einstein, they say, man has recoiled from the shadow of the new. And as often--refrigeration, telephones and penicillin come to mind--our technological and chemical handiwork makes life immeasurably healthier and easier. It was once said labor-saving machines would cause mass unemployment, that recombinant DNA would be the end of us. Haven't we heard a thousand siren calls of doom?

"The fact is that he is right; this could happen," says John Perry Barlow, founder of the Electronic Frontier Foundation. "By the same token an asteroid could hit the planet Earth and have an even worse effect."

To which Freeman Dyson, the Princeton University physicist, adds: "There is a hidden cost of saying no. And that's the advances in health and food production that we will forgo."

Against such optimism lies the weight of homo sapiens history. So many technologies, from chariots to flying machines, become killing tools in our hands. And the bacteriological world spawns as many surprises: the Spanish flu, Ebola, AIDS. The grandfather of these 20th century disasters was the medieval plague. Arriving in so many guises that 14th Century doctors termed it the "strange chameleon," the plague ravaged Europe, killing one-quarter of the population and causing bailiffs in London and Venice to nail shut houses of the dead and dying. Today, scientists can identify but a few plague strains. Many more strains have disappeared, or lie dormant.

We tend, also, to underrate our scientific ability, and our capacity for mayhem. Scientists in the 1930s laughed at the suggestion that they might soon explode "dangerous quantities of available subatomic energies." The nuclear bomb, a renowned physicist argued in 1930, was "a hobgoblin." Fifteen years later, Edward Teller oversaw the construction of such a bomb. Warned that it might ignite the Earth's atmosphere, he endorsed an explosion anyway. And his colleague, Robert Oppenheimer, stared at the nuclear dawn and quoted the Hindu God Vishnu: "I am become Death."

The point is not that technology is evil. It is amoral. Human nature is the wild card. Joy has been talking for hours this morning. The white around his blue eyes is burnt red from lack of sleep. We leave a restaurant and walk across the street in San Francisco. A brilliant western sun beats down, a dog barks, an old man sips an espresso under a palm tree. Joy talks of the golden valley of silicon an hour's drive to the south. He discerns a thirst for scientific knowledge--and the money to underwrite it--that frankly scares him. "In the endless pursuit of truth, we're about to bring back the notion of fate. By empowering everyone to hold the powers of destruction, everyone controls our fate as a species."

His eyes catch yours. "That's not a pleasant thought, is it?"

Upload Your Brain

"We need to distinguish between artificial intelligence and downloading your brain into a computer, which is controversial," says Ray Kurzweil, on the coming age of spiritual robots. We're in a lecture hall at Stanford University, jammed with scientists and aficionados of technology, listening to a panel of geneticists and experts on robotics and nanotechnology discuss how and when man might become silicon. Bill Joy is on the panel, too, in his new role as house skeptic. To listen to these scientists, to swim in their virtual realities, is to understand how profoundly Joy's message disquiets them. This is a world where evolution equals technology equals progress. Where scientists talk about what technology "wants" and "needs" and "demands."

The passing of our carbon-based bodies is seen as inevitable, like a butterfly shedding its chrysalis. It's no insult to call this science fiction; science and fiction are tectonic plates that long ago converged in Silicon Valley. "We're in for a 100-year identity crisis as human beings, and we're launching right now," says Kevin Kelly, an influential writer and member of the high-tech digerati. "We should figure out what technology wants and train it to be a good citizen."

A few of the scientists see intelligent robotic life as centuries away. Others discern it around the bend of the next decade. Some believe that microscopic nanotech centurions will guard our bloodstreams against man-made viruses. Others argue that we may have to manufacture pathogens to combat other manufactured pathogens. Uploading human brains into computers, blending the biological with the silicon . . . the point of agreement is that swift, irrevocable change is inevitable. Technology will be our savior. What's striking is the bedrock faith in the church of science as the ultimate truth.

"Progress and technology are inevitable," Kurzweil says. "We are talking about the next evolutionary step. In virtual reality, we will have bodies and riches. And we will take a walk on a virtual beach in a virtual Cancun." Hans Moravec, the computer scientist with the blue T-shirt and the blue work shirt with the collar pulled up clerical style, is the least sentimental of the panel. In his view, sensory pleasure, ethics, morality, all of these are evolutionary outcroppings that could shear away as we evolve. "One way to evade the biological threat is just to become non-biological." Moravec plays with a smile that suggests he's aware that the homo sapiens in the audience might find his proposal discomfiting.

"The evolution of our descendants will push them into entirely different realms." He's still smiling. "They will become something else entirely. I don't know why you are disturbed by that." On and on the panelists talk, a wizard's brew of possibilities. Joy pulls at his upper lip and listens with the detachment of an apostle strayed. Their assumptions about the world and technology are no longer his. "To use the word 'want' about technology, to talk about technology as our 'children'--" Joy shakes his head, disbelieving. "We are using an incredibly misleading vocabulary to try to get around vast ethical questions." ,p> And he's not comforted by the notion of uploading the human brain into the computer maw. Hit the delete button on that one. "Robot existence will never be a human existence," Joy says. "It is not three-dimensional. We're not a native form in cyberspace. If we allow evolution to go on in there, what results will not resemble us." Roam the world of Web thinkers and scientists, in California and elsewhere, and it's not hard to find people who share Joy's fears. But the prevailing ethos, that scientific research cannot be constrained by man, is so strong that few even talk of reining it in.

Christine Peterson of the Foresight Institute, a California new-technology think tank, counsels fear followed by acceptance. "Bill's correct to be nervous and seriously scared," Peterson says. "But research is not going to stop, that's not an option. If you try to stop people, they'll go underground." Peterson would ask research labs to agree to outside surveillance. Other Foresight board members are working on legal and ethical protocols for nanotech research.

As for an age of intelligent computers? Peterson suggests we might want to be careful. It will be hard to know what annoys them. (When you thumb your nose at a conscious computer, will it know it's being dissed?) We'll want to keep the computers' competitive impulses in check. "We want a system where computers have no need to wipe us out. We want to make sure that physical property rights [translation: Our right to breathe and live] is protected." She likens the human race to the Amish. Our legal and ethical system ensures that the Amish can live happily insulated from a technological world.

So one day, by implication . . . ?

"Right. When computers rule, we're going to be the Amish."

Crash the System

"A Joyless Future" . . . "No Joy Here" . . . "Silicon Valley Killjoy"

So the headlines read in the days after Joy's essay appeared, the heretic nailing his manifesto to the door of the high-tech temple. National and international media interviews, scientists reacting, columns mixing praise with dismissive shrugs. He e-mailed copies to lawmakers, to the Supreme Court, the president and his Cabinet secretaries. He's yet to hear back from any politician of note. The Republicans and Democrats are clamoring to claim credit for the technological dawn. Who wants to talk about the night? Nor do many of his friends in Silicon Valley want to hear these arguments. In a way, it goes to why he decamped several years ago, left the Valley and its hungry young men and women sleeping on lab floors. He left for the vertical beauty of Aspen. Different time, different culture. "The Valley culture has kind of gotten away from me," Joy says. "They're in this Darwinian mind-set. They can't imagine not doing it."

So he hopes to take the conversation outside the church door, to like-minded scientists and activists. Several high-tech thinkers, from Valovic at Intel to physicist Amory Lovins to John Seely Brown, director of the Xerox Palo Alto Research Center, now argue that our high-tech culture suffers from technological tunnel vision, without concern for protocols and the social good. And the Clinton administration has taken halting steps, such as declaring emerging diseases a national security threat. ,p> Joy would go much further. Prohibit research on dangerous technologies and put stringent international controls on sensitive equipment, much as nations do with plutonium and biological war agents. Create an international whistleblowers' fund and pay vast sums to anyone who exposes life-threatening research work. And unleash the lawyers. Demand that private labs take out insurance against terrible accidents, and let the raptor tort types have at them. Death by lawsuit, a strategy immediately recognizable to any anti-tobacco activist.

"We need to reset the social contract," he says. "Just as we did in response to the technologies of the first industrial age."

In the end, perhaps, it comes back to faith. A former colleague or two has suggested that Joy has gone round the bend, become a primitive spiritualist in a rationalist world. But talk to the digerati long enough and you hear a sometimes unspoken belief in a "hidden hand" that shields us from the most terrifying use of our inventions. We've had the bomb for half a century, the argument goes, and we haven't destroyed ourselves. AIDS rose up along with the technology emerged to keep it at bay. We have survived so far, therefore . . . .

"There is a co-evolution between society and technology," argues Barlow of the Electronic Frontier Society. What keeps us in balance? "At a certain stage you just have to go theological," he says. ". . . I do believe that there are larger forces than ourselves at work."

Joy comes to quite the opposite conclusion. He is no atheist, he left that certainty behind in college. But the world, he believes, is in our hands. It is the price of our maturity as a species. He walks through the park toward his car. He stops and watches the Chinese ladies do a slow tai chi dip and rise. There's not a cloud in the sky. "If you don't believe extinction can happen," he says, "check the fossil record."