Interview with Professor Sidney Dijkstra

(c) Copyright 1998, Wayland Systems Inc. All rights reserved.

Professor Sidney Dijkstra is a complete luminary, whose overall contribution to software engineering is impossible to measure. He has been outstanding in his field for several decades, having won the coveted 1955 Univac Prize for Excellence in Assembler and having invented in 1971 the Dijkstra Frame, an invaluable support mechanism for programmers and analysts walking through their deliverables.

In 1994, following his work on higher-order algorithms using n-dimensional ellipsoids to yield bounded approximations for multi-variate correlations in finite computational time, he was awarded the Freedom of the City of Salford in England. I was lucky enough to have the chance to interview Professor Dijkstra at San Francisco International Airport, as he was standing by for a flight to Guam, where he was to give a lecture entitled Software in the Twenty-First Century: More of the Same?

-Meilir Page-Jones

[With the permission of Professor Dijkstra, the text of the following interview has been edited and slightly abridged.]

MPJ: Good morning, Professor Dijkstra. Or, if I may venture a couple of words of Bulgarian, Dobro utro! Let me first ask you the question that everyone asks: Are you related to the other Professor Dijkstra?

SD: Dobro utro! And let me give you the answer that I always give: To my knowledge, no, I’m not. I’m a graduate and former computer-science professor of the University of Veliko Turnovo and I now teach software engineering at the prestigious Tukwila Community College in Washington State.

MPJ: Professor Dijkstra —

SD: Call me Sid.

MPJ: — what do you see as the greatest single problem currently facing US software-development shops?

SD: An organizational problem that I encounter in shop after shop is the staffing of entire teams with programmers. In other words, the positions of analyst, designer, tester, manager, librarian, documenter, ... are all occupied by programmers.

MPJ: But surely most shops don’t label everyone "Programmer". Just a glance at employees’ business cards reveals Analysts, Software Engineers and so on.

SD: Of course they don’t; this phenomenon is rarely blatant. Usually, everyone is issued with the correct business card, with "Manager" or "Analyst" or whatever on it. However, when I scratch the surface of such an "Analyst" I often find a hastily promoted and retrained programmer underneath. Yet, this retraining — usually minimal at best — doesn’t help much. Someone I met recently at a conference did a study of this issue. He said that, based on his experience at a large corporation, "teaching a programmer to analyze is not only a waste of time but it also annoys the programmer."

MPJ: I have to say that this runs against my experience. I’ve known several people who were both excellent analysts and excellent programmers.

SD: Yes, the issue is complicated by the fact that even programmers are people. A given programming person may have skills beyond those of programming. But skilled programmers who are also skilled in (say) analysis are, in my experience, in the minority. The arbitrary promotion of a programmer to analyst — often as a reward for good programming service — may therefore rob the shop of a good programmer and create an analyst incapable of appreciating business needs. As one of Tom’s colleagues remarked recently, "moving Tom from Programming to Analysis will reduce the average competence of both groups."

MPJ: So you’re saying that an accomplished programmer could also be a good analyst, just as an accomplished pianist could also be a good water-skier. But one skill isn’t necessarily a predictor of the other, and, in any case, a great deal of retraining would be vital for someone to transfer from one activity to the other.

SD: Well I hadn’t thought of it quite like that before, but I suppose that’s an adequate metaphor. And don’t get me wrong. Programmers are wonderful. I experience waves of joy whenever I see a programmer. Indeed, one of my best friends used to be a programmer. But programmers’ skills are simply not the necessary skills for library management, analysis, project management, user-interface design and all those other tasks necessary to a modern shop.

MPJ: Perhaps, then, we could look at each of these roles in a shop and you could explain why you feel that programming skills alone are insufficient to fulfill that role adequately on a project.

SD: Sure! Where shall I begin?

MPJ: How about with the project charter?

SD: I presume you mean by "project charter" the "why" of the project, its business rationale, goals and expected benefits.

MPJ: Yes.

SD: It’s strange you should mention this, because last month I was at a company where the business people had completely abandoned their responsibility for the project charter to the IS Dept. The IS people hadn’t the information, the perspective or — frankly — the expertise to put together a charter. They should have just said "no," but they didn’t. In a frenzy of excitement they proposed a whizz-bang client/server system with distributed object-oriented databases and the capability for transmitting video clips over the network. All this for an apparently straightforward order-entry and invoicing system.

I suggested that choosing such a technology solution was somewhat premature, given that no-one had yet studied the specific business needs being addressed. From then on, I was intently ignored by the IS people on the project. I guess they considered me what you Americans call a pooper-scooper.

MPJ: Party pooper.

SD: Quite. Anyway, I noticed that the project was being driven by several programmers’ desire to use approaches such as C++ that would breathe life into their resumes. The project is now proceeding on a plane of reality entirely apart from the corporation’s business. The various business users are, respectively, captivated, fascinated or intimidated by the new technologies. They’re therefore not in a good position to apply the brakes of sanity to this run-away project, either.

Well, we’ll see how the project turns out. My guess is that it will yield a Taj Mahal system, a monumental system that’s a wonder to behold, but absolutely useless for day-to-day business.

MPJ: I can see why programmers might not be best suited to assessing a project’s best business mission. But let’s turn instead to detailed systems analysis, requirements analysis. Don’t you think that here programmers can really shine?

SD: Not necessarily. Analyzing requirements and cutting code are different worlds. The skills that reign in each world are different — they’re almost orthogonal to each other.

MPJ: What do you mean — "orthogonal"?

SD: Well think about the two worlds. What abilities do you need to be a requirements analyst in a typical business or engineering corporation?

MPJ: Off the top of my head, I’d say that analysts need to be good at communicating with the business folks, to have an understanding of the business itself, to be able to separate out various concepts and to be able to model business abstractions in a verifiable form.

SD: Yes. And add to that the ability to cope with vagueness, misunderstandings, arguments, conflicting requirements and a scope that seems always up for discussion. One analyst that I know likens analysis to "trying to get fog into a bottle." Many programmers don’t even have the patience or temperament to get on in a world like that. I’m not surprised those IS shops that simply send unprepared programmers over to the business people usually get themselves into political hot soup.

Contrast the analyst’s world I just described with the programmer’s world, where the "big picture" has already been focused during analysis and then chopped up into smaller pictures. Thus — for an individual programmer at a given time, at least — the scope of each problem is constrained and that problem is relatively well defined.

MPJ: You know, this portrayal doesn’t tally well with some of the programming assignments I’ve been handed.

SD: No doubt it was because analysis wasn’t done, or wasn’t done properly. Therefore, some of the undone analysis work spilled over into your job, where you probably had neither the time nor the inclination to deal with it.

MPJ: That’s true. I didn’t have easy access to the business end-users, either. So, assuming that analysis has been done upstream, what skills should a programmer have?

SD: In short, the ability to communicate with a computer. For example, a programmer should be up to date on software technology, and even some hardware technology. S/he should be adept in the programming languages s/he is using and s/he should be able to model ones and zeroes at a higher level than lines of code.

MPJ: But where’s the creativity in that?

SD: Whenever I hear a programmer mention the word "creativity", I reach for my revolver. All too often, the term "creativity in programming" is a euphemism for anarchy or time-wasting. Much of the creative work in algorithmic programming has already been done, but yet I find some programmers have to reinvent everything from first principles all over again for themselves. Recently, I had a guy who spent weeks dreaming up a new way to code a list-merging algorithm. Then, after all that waiting, I got a "creative" solution that I recognized from [Professor Donald] Knuth’s work in nineteen sixty-something.

MPJ: So, you’re saying that there’s no creativity in programming? All the creativity is in analysis?

SD: No, there’s still creativity in programming. For example, it takes intelligence to look at a given problem and a given hardware/software platform and choose the best strategy for solving the problem — the most appropriate data structures, algorithms and so on. When I say "best strategy", I sometimes mean "least worst", of course!

MPJ: Am I right in summing up the difference between analyst and programmer by saying this: The analyst is the problem definer, while the programmer is the problem solver?

SD: Yes, very good. I see you’ve read your Phil Metzger, who’s very sound on this kind of thing.

MPJ: But there still seems to be one glaring flaw in separating the role of analysis from that of programming: An analyst who doesn’t know how to program may specify a requirement that will take 100 trillion machine cycles to run, or which isn’t even algorithmically possible at all.

SD: That statement really misses the point. I don’t say that it’s bad for an analyst to know how to program; it’s simply that the coincidence of both sets of skills in the same person is rare. I’m also saying that a lot of havoc is caused by managers’ believing that, rather than its being rare, the coincidence is common. But, in answer to your statement, obviously someone with knowledge of technology — of feasible implementations — should review the spec. before any commitment. However, that person may not be the author.

MPJ: Let’s turn from analysis to design. Isn’t the situation different there?

SD: Design of procedural code, you mean?

MPJ: Yes, we’ll take that for a start, anyway. Presumably, the skill of designing code and programming code are closely related.

SD: OK, I’ll concede that almost entirely. Most good programmers are also good designers, and vice versa. However, even that correlation may break down. I’ve met some programmers who can write 100 lines of exquisite code. But give them 10,000 lines to write and the result is a jungle of unnecessary complexity.

MPJ: Ah, but here the programmer simply isn’t doing design.

SD: That may be true enough; the programmer is just hacking, allowing some sort of design to emerge from the code. But my point is that a programmer who doesn’t actually do explicit design can hardly claim to be a skilled designer in any useful sense. Unfortunately, there are plenty of such programmers around. There’s also plenty of gobbledygook around to "legitimize" what they do. Some claim that they’re following the RAD [Rapid Application Development — Ed.] approach. I met a hacker last year who claimed that he was following a so-called "spiral methodology". I quizzed him on what he’d read of the spiral approach. Not a syllable! He’d picked up the term from another hacker, who’d picked up the term at a conference, perhaps from another hacker — who knows!

*Footnote added later by Professor Dijkstra: The true spiral model of a software project is a legitimate one, excellently expounded by authors such as Boehm. [See Boehm’s original paper in Software Engineering Notes, 11(4), p.22 - Ed.]. The spiral model represents a formal approach to project management, crafted to reduce risk of project failures, especially on large or novel projects. It emphatically does not equate to “just hacking”!

MPJ: We’ve just discussed procedural design. What about data design, database design for example?

SD: Normally I consider data design, alongside procedural design, to be an integral part of software design. Therefore, what I said above about procedure should, in theory, apply equally to data.

MPJ: Why do you say "in theory"?

SD: Because it doesn’t always work out like that in practice. For some reason — maybe it’s poor training or skewed experience — when some programmers turn their hand to database design, they really hose it up. Is that the right American term, "hose it up"?

MPJ: As in "hose up the database design"? Yes, exactly.

SD: I’ll give you an example from a real project, which collapsed only last month. I’ll have to simplify the details a little, or we’ll be here all day. Diggory, the database "designer", created four relational tables to handle ten distinct entity types.

MPJ: In other words, he set up far fewer tables than there were entity types.

SD: Yes. And this was the first sign of trouble. But the saga continues. The requirements analyst had assigned a total of 80 attributes to the ten entity types. (Each entity-type had about 8 attributes.) Diggory’s four tables, however, had 56 columns in toto.

MPJ: I get it. There were far fewer columns than attributes and I bet you’re going to say: "This was the second sign of trouble."

SD: Precisely. Because not only had Diggory packed more than one entity type into a table, he’d also packed more than one attribute into the same column. So he’d overloaded some columns with several meanings. For instance, one column of the CUSTOMER table would mean either customer-credit-limit or annual-dollar-sales, depending on the value of a flag in another column of the respective row. And there were dozens of similar abominations. To make matters worse, because each table had many "flag columns", Diggory usually decided to bit-pack various flags together into a single column.

MPJ: Arghh! No wonder the project was a disaster.

SD: But it wasn’t ... not yet. You see, Diggory was an accomplished procedural programmer. He handled his tricky data design through some even-trickier procedural code — and, believe it or not, the whole thing worked! Sort of. For a while, at least. But no-one could issue simple ad hoc queries against the database and maintenance proved to be a costly nightmare. Changes to database design are always problematic, but in this case they were catastrophic. The system soon became so defect-ridden that the users began to pour scorn on it from a great height.

Diggory then went to still greater heights to pour scorn on the intelligence of the users. This caused him to fall into a political casserole. Soon he could be seen through the glass corporate front-doors as a shrinking silhouette in the parking lot. After Diggory’s departure from the company, there was no-one left who really understood either the database or the code. The project unconditionally capitulated.

MPJ: A cautionary tale, indeed! But, I trust, not a typical one. If I may suggest a more typical design area in which programmers perform inconsistently, it’s GUI [graphical user interface — Ed.] design.

SD: Oh, absolutely. I’d estimate that of every 100 programmers asked to turn their talents to GUI design, 20 "get it" right away, 40 can be made to "get it" with suitable prodding and 40 never "get it" even if the prods pack several kilovolts.

MPJ: Why do you think this is? What’s the factor that determines who will be good at GUI design?

SD: I’m not absolutely certain. I’ve given it a lot of thought and I’ve decided that there isn’t just one factor. There are several factors involved, some mental, some environmental. A good GUI designer must have a sense of esthetics from the large to the small, from overall window layout to individual field placement. S/he must also be prepared to follow standards, some of which — such as the placement of a colon — may be minuscule. But the biggest differentiator that I’ve found between good and bad GUI designers is simply this: empathy with the users.

MPJ: You mean really understanding what the users want to do with the system?

SD: Yes, and how they want to do it — again, at several levels of detail. I’ve noticed that the best GUI designers really "think and feel user". Some GUI books talk about understanding the users’ units of work, but that’s a sterile way of putting it. In reality, good designers have a deeply intuitive grasp of the users’ jobs. Part of this comes from natural empathy and part of comes from simply spending time with the users in their day-to-day work. Programmers who stay closeted up in their cubicles with their workstations will never make it as top-notch interface designers.

MPJ: So, most education for GUI designers comprises hanging around users until you’ve caught on.

SD: Well, I don’t know about "most", but I’d certainly say "much". In any case, mixing with users serves as a safety net even for the worst GUI miscreants.

MPJ: How so?

SD: I’ll give you one example. Rufus was a very serious, ultra-logical programmer, who had turned to designing GUIs for the first time. He came up with an idea for command buttons that had him laughed out of the room the whenever the users saw his windows. He figured, logically enough I suppose, that a button’s size should be proportional to the frequency with which the button would be clicked. So, he made buttons like Ok and Save very large, while he made ones like Special Options ... very small. In order to fit the large buttons on a window, he would use any available clear real-estate, whether it was in the NW, SW, S, NE — or wherever — of the window.

MPJ: At least he didn’t use the middle.

SD: No, but the effect was still ludicrous. There was no way to predict where the Save button might appear from window to window, or how big it would be. Add to this a color-code of Rufus’ own devising ("safe" actions had greenish buttons, "dangerous" ones had reddish buttons) and you can imagine why the business people would ask to see Rufus’ windows again and again. They took endless delight in startling uninitiated users with the "Bozo the Clown" windows.

Needless to say, Rufus’ interesting GUI design didn’t make it into the final system. However, if Rufus hadn’t gone out to the users early, perhaps his design would have been delivered, because none of his fellow designers called him on it. In fact, some of them had begun to imitate it!

MPJ: Rufus must have been very demoralized by the whole experience.

SD: No, Rufus was always professional enough never to let his ego interfere with his education. He learned a lot from the experience and later set up a small interface lab for the shop.

[Mr. Dijkstra. Mr. S. Dijkstra. Please come to the ticket counter.]

MPJ: I hear your name being called. It looks as if you made it on to the Guam flight. Since we’ll need to keep this short, let’s skip library management and move on to testing.

SD: Before we do, I’d just like to say this: Most programmers hate document management. Some reusability projects that I know have failed simply because no-one could be bothered to track the inventory of reusable components. In fact, the best object-class librarian that I ever came across was by no means a programmer. She had — no surprise, here — a degree in library science.

MPJ: OK, now for testing. I have to say that one of the best testers I ever met was formerly a programmer. To be honest, though, he was rather a poor programmer and he almost got fired before the shop found him a niche in the testing group.

SD: Sounds to me as if he was a great tester because he’d had so much experience at it. Being such a poor programmer, he’d probably spent far more time over the years on testing and debugging than he had on coding.

MPJ: Yes, you’re probably right. I’d never looked at it that way before. In general, though, what do you think of programmers as testers?

SD: I have no problem with programmers’ doing testing. Some of the perverse attention to detail that makes a good programmer also makes a good tester. However, there are several aspects of testing that require little programming ability.

MPJ: For example?

SD: These are some of the aspects: interface testing, testing to see whether the software meets the business need, and even black-box testing. Remember that the crux of testing is coming up with the test cases.

MPJ: Of course, a tester needs some computer skills to run tests.

SD: Yes, but not specifically programming skills. Anyway, with today’s testing tools you don’t need very much in the way of computer skill to run tests. I know several people in SQC [Software Quality Control — Ed.] who are by no means programmers.

And that reminds me, we’re missing the most important point here: It really is bad news for a programmer to test his/her own code. A shop really needs a separate group — call it QA, QC, whatever — to check the quality of software before delivery. It’s perfectly fine to rotate people through this group, people such as users, analysts, certainly programmers. The only rule is that no-one should develop test cases for his/her own deliverable; a person who unconsciously creates an error is unlikely to consciously craft a test case to expose that error.

MPJ: Now the big one: management. What do you think of programmers who move into management?

SD: I’m sure you’re expecting me to say: "Not much!" But that would be an over-generalization. You see, everyone has to begin his career as something other than a manager.

MPJ: So you don’t hold much stock in those people who graduate in something called "management"?

SD: No, although it’s important to learn generic management skills, project managers should have strong feelings for — preferably experience of — the trades that they’re managing. You can’t manage thin air. So I’m happy that most project managers in the software business do indeed arise from some software trade. But I’m a little less happy that that trade is almost always programming; in some shops, it biases first-line management towards coding activities and away from solid analysis and design modeling.

MPJ: I recall reading that you distinguish two project-management roles: the director, the inwardly focused role that concentrates on the technical side of the project; and the producer, the outwardly focused role that concentrates on the financial and political aspects of the project, especially vis à vis the corporation as a whole.

SD: Actually, that was Fred Brooks in The Mythical Man-Month. But I certainly go along with his distinction.

MPJ: How do programmers fare when they’re promoted to project producers, as opposed to project directors?

SD: Not always very well. As usual, I have an example. Judson was a programmer who got into the software business to program. He loved to code. A workstation with compiler was Jud’s idea of heaven. He was a good programmer, too. All his colleagues respected him and often turned to him for advice. In recognition of his competence, he was promoted to project manager and took charge of the next major development project.

He managed the project well. He knew just what to do and when. He trusted and understood his team and his team trusted and understood him. But where Jud really came apart was in managing the relationship between his project and the business users. As my friend from Boston likes to say, there Judson really got scrod.

MPJ: Hmm, I think I can see what’s coming ...

SD: Once the users had signed off on the scope, schedule and budget for the project, Jud thought that all was well. The schedule and budget seemed tight, but not absurdly so. However, even while the ink of the signatures was still wet, the users began to pile on demands for more and more functionality. At every demand, Jud caved in. To make matters worse, halfway through the project upper management cut back the budget by 15%, as part of a corporate cash crunch. In every negotiation, Jud was politically outmaneuvered, until soon the schedule and budget really did become absurd.

Unable to reflect the pressure back to upper management, Jud transmitted the pressure down on to his team, exhorting them to "work harder". The previously good relationship between Jud and his team began to sour and then turned downright acrimonious. A key team-member quit, while the rest of the team staged an open revolt. Much time and energy was wasted and the project exceeded its budget without delivering anything. Jud’s career at that company was over.

MPJ: So it doesn’t work to promote a programmer to project manager?

SD: Yes, it may work — so long as you supplement the technical manager, the director, with an expert project producer. Contrast Jud’s story with — I’ll make up a name — Judy’s. Judy was a programmer promoted to technical project manager, who had a corporate project manger — a producer, in other words — to handle politics and finance. He was an ex-marine, a battle-hardened character that people didn’t mess with. He negotiated any increase in functionality by demanding, for instance, a larger budget. Either he got the larger budget, or the users thought twice about the scope creep.

[This is the final call for flight 4003. Would passenger Mr. S. Dijkstra please come to Gate 61 immediately.]

MPJ: I see that you’ve got to get going. Could I just get a quick summary from you, please?

SD: Yes. A good software team should have the hybrid strength of many different skills and personalities. The team should be an interlocking set of specialists held, together by a few generalists. It should never be made up solely of programmers. Or, to reduce this idea to a bumper sticker, as you American folks seem to like to do:

SOFTWARE IS TOO IMPORTANT TO BE LEFT TO PROGRAMMERS.

MPJ: Thank you very much, Professor Dijkstra. Have a good flight.

SD: Molya. Dovizhdanye!

Biographical note: Professor Sidney Dijkstra’s father was the Dutch ambassador to Latland when the Second World War broke out. The Dijkstra family was trapped in Latograd for the duration of the war. After hostilities ceased, Mr. Dijkstra Sr. was posted to Veliko Turnovo, Bulgaria, at whose university Professor Dijkstra was later awarded the Vlad Tepes Chair of Computer Science.