Dawn Levy, News Service (650) 725-1944; e-mail: firstname.lastname@example.org
The Prisoner's Dilemma: Does industry demand for software virtuosos siphon talent from computer science education?
"You want leather pants -- we'll make you leather pants." You want more, maybe we'll even build a factory.
That's how markets work, or are supposed to. Demand eventually begets supply. If customers are buying more rice one year, farmers plant more of it the next. Factories, responding to the signal of higher prices for their products, ramp up output or even expand capacity.
Yet when it comes to the market for software developers, the current industrial-strength demand for their services is actually causing a chronic shortage of those professionals. That was the counterintuitive message delivered at this year's annual meeting of the Stanford Computer Forum (SCF) in a keynote speech by Eric Roberts, the Charles Simonyi Professor in the School of Engineering. The SCF is a joint venture of Stanford's Department of Computer Science, the school's Computer Systems Laboratory and many industrial affiliates.
The two-day event was organized by SCF director Professor Fouad Tobagi and assistant director Suzanne Bentley. Held March 21-22 on campus, it featured a dozen talks and some 50 poster sessions on computing topics of interest to a business audience. There were also such whimsical digressions as the opportunity to virtually "conduct" the Vienna Symphonic Orchestra. Designed by Assistant Professor Jan Borchers, the exhibit allows visitors to wave an electronic baton to which the orchestra, appearing on a large wall screen, responds. Point the baton at a particular section, and those instruments play louder; wave it faster, and the tempo picks up although if you wave it too fast, the entire orchestra stands up and starts shouting curses at you.
Roberts' presentation was especially relevant to the attendees, most of them representing companies for whom software is mission-critical. "There's an enormous demand from employers for our students," said Roberts, who as associate chair of the Department of Computer Science is responsible for its undergraduate programs. "That's true even with the direction Nasdaq has taken in recent months. We need to produce many more than we are producing today."
Roberts said that skeptics, noting that Microsoft hires only 2 percent of all applicants for its software positions, ask, "How can you say there's a shortage?" To these analysts, it just looks as if older programmers are getting passed over for younger, cheaper workers.
This demonstrates a fundamental misunderstanding of the problem, Roberts said. "Those young workers are not cheap!" he said, adding that many senior workers would jump at the salaries offered to younger hires. "Rather, there's a huge variability in the productivity of software people. And we're not just talking about one bricklayer being able to do 15 percent more than the next." A classic 1968 study, for example, showed a greater than 20-fold variation, even among programmers with the same nominal backgrounds, in the time required to solve a particular programming problem. Moreover, "programmers who produced code the fastest also produced the smallest code, with the fewest bugs," Roberts said. "If anything, code is much more complex today, further exacerbating this skill variation. The spread is probably more like 100-fold now. I'm convinced most new hires don't pay their costs. The argument is made that 'if the market's tight, you can't be picky.' But if there's a huge variation in the quality of people, maybe you'd better be picky."
Although there's a tremendous demand for "Picassos" (the highest-quality software developers), telling a Picasso from a decent house painter is harder than it sounds. Factors behind superior performance -- they include both cognitive skills (the mastery of detail and "ability to keep lots of things in your head at once") and temperament ("a competitive urge not to be beaten by a machine") -- remain mysterious. Some programming virtuosos are born that way, Roberts said, while others are beneficiaries of a superior education.
"There's no good predictive test that ensures you'll get one," Roberts continued. "But the industry itself serves as a proving ground of sorts. No company -- unless it's really closing down -- lays off people who are 100 times more productive than the next person. There's a better chance of finding winners those few who will, by dint of their amazing productivity, ultimately subsidize the salaries of their more mediocre peers among a pool of graduates from elite universities than among those recently laid-off workers."
Companies, especially dynamic new ones whose success absolutely depends on hiring the best talent, face what game theorists call the Prisoner's Dilemma, whereby a particular choice benefits a decision-maker in the short run but penalizes the entire group -- including even the short-term beneficiary -- over the long haul: "You see a very bright CS undergraduate who's trying to decide whether to go for a Ph.D. or come and work for you. Should you hire them, or let them get the Ph.D. and become a professor? It's bad enough to lose them to a Ph.D. program, but far worse to see them hired by a competitor. From a market perspective, you have to try to recruit such people."
And why stop there? "One place you can always mine with a high success rate," Roberts noted, "is computer science faculty. We're eating our seed corn."
In the mid-1980s, he recalled, the computer market was sizzling, and corporations were siphoning applicants away from academic careers. "There was one applicant for every seven positions." With faculty attrition and fewer doctoral candidates came heavier workloads for remaining faculty members to compensate for the shortfall in head count and tighter entry requirements for undergraduates. "Universities imposed very stringent limitations. At UC-Berkeley, a minimum requirement for a CS major was a 4.0 grade-point average." Computer science degrees in the United States peaked in 1986. The subsequent decline was due not to an industry slump or diminished demand for courses, Roberts said, but to a diminished capacity to provide teaching resources in the face of a booming computer market.
The situation is repeating itself. "This year, there will be at least three CS faculty openings for every candidate," whereas most other fields will see hundreds of candidates for each position. The rapid evolution of the discipline, forcing nearly constant curricular re-evaluation and change, merely exacerbates the problem, Roberts said. "The number of people who will come out of the pipeline is certain to decline at a time when every serious labor economist believes that it needs to increase. I believe we'll see, within the next two years, a diminution of between 20 and 40 percent in the capacity of this nation to generate CS graduates."
Academia, industry and government need to work together to find extra-market ways to increase rather than decrease the supply of computer science undergraduates, concluded Roberts. He offered several suggestions for doing so: more financial aid to young faculty and graduate students; recruiting of additional computer science faculty with strengths in teaching rather than research; greater use by schools of industry-based adjunct faculty; and continued and enhanced financial support by industry for academic computer science departments, joint ventures such as the Stanford Computer Forum and internships.
Stanford, for its part, is doing its best to stem the reduction in computer scientists, Roberts said: Computer science is the largest undergraduate major in the School of Engineering, and one of the largest majors at Stanford.
By Bruce Goldman