Let's get right to it. The main reason that DP projects are late and over budget is that they were underestimated and underbudgeted right at the beginning. It's true that there are other contributing factors; many projects are mismanaged (The Data Processing Project Seen as a Summer's Day) and a few are just plain unlucky, but most were way behind schedule and over budget on their first working day. Why?
Many are late because the dates aren't set on the basis of the amount of work to be done and the resources available. They're set on the basis of things like, for instance, the comment that your CEO made to the CEO of that other company on the 7th green of the local golf course.
"All of our transactions will be entered through the Internet within six months," brags our guy.
"Really, that's mighty ambitious," says the enemy CEO, "Tell you what, I'll bet you a dozen golf balls that you don't make it."
Why did our guy say that? Was it based on a careful analysis of equipment to be installed, programs to be written, other projects currently in progress, and etc? Not even close! It was probably a line item on a report called Future Projects that happened to cross his desk a few days before. He probably said it because on the 6th green the enemy CEO had mentioned his new email-based marketing system "Spam 'R' Us", and our guy felt that he had to counter. And so the offhand, slightly blustering comment becomes a target date, chipped in stone and sitting in the lobby of the building for all to genuflect to as they pass.
A multitude of other should-be unrelated factors preset target dates and budgets. Sometimes greed sticks its little green whatchamacallit into the proceedings. Making an end of the year target date might mean bonus bucks for a manager in an incentive program like Management by Objectives (Magic Potions). Sometimes the target date is derived from other factors that should be derived from it. Examples of this "space shuttle pulling the booster rocket" scenario ("cart before the horse" seems so old fashioned) are: setting target dates based on the amount of money available, or setting them based on the Gant chart for other projects ("Project A has to be done in February because Project B will need its output then."). Finally, there is a tendency to estimate projects low just to get them started. Upper management might go into serious "sticker shock" at your actual one year, one million dollar estimate for the project, but they'll go for 750 thousand and nine months and then they'll go for an extra 250 thousand and a three-month extension so as not to waste their initial investment. This technique is used consciously and shamelessly by some software salesmen and consultants, but it's also used subconsciously and shamefully by middle managers who are simply afraid to go to their bosses with a big number.
But let's be honest, even when we set the target date ourselves and there are no external pressures forcing the date lower than it should be, the *#^@% project is still late and over budget. Why?
I had noticed for some time that when I made an offhand guess at how long a project would take ("Oh, sounds like about a man-year to me"), and then later came up with a carefully studied and analyzed estimate ("8.375 man-months"), that the carefully analyzed estimate was always lower than the offhand one and that, in the end, the offhand one was closer to being right! The reason for that is that offhand estimates and carefully studied ones use different methods. The offhand estimate is done by comparing what you know about the new project to similar projects you've worked on in the past and using the amount of time they actually took. Someone gives you a little information about a new project and asks you for a time guess. "Well," you think, "This one is a little like the XYZ project I worked on two years ago, and a little like the ABC project my friend Joe just finished." So you throw those two projects into the mental blender for a few seconds and come out with, "Sounds like about a man-year."
The careful analysis method usually begins by breaking the project down into smaller pieces. It then estimates each piece and adds up the estimates to get a grand total. This method is almost guaranteed to produce an estimate that is too low, first because you never can identify all the pieces before you start, and second, because within each piece, you never really know what's involved. The silly, offhand method actually takes the missing pieces and the extra difficult pieces into account because that kind of thing happened in the real projects you're using as a measuring tool.
So what's to be done? If the target date was established on the golf course, you're stuck. Well, pretty much stuck. The best bet there is to redefine the question you're being asked. The usual question is, "How long will it take and how much will it cost to do this project?" If the time and costs are already established by some outside agency, you can try changing the question into, "What subset of the project can you get done in this time frame with this amount of money?" If your CEO promised that all company transactions would be on the Internet by the end of the year, he didn't really mean ALL, he meant more like...some, or possibly...a few. And what does it mean to be "on the Internet?" It doesn't necessarily mean that everything is being processed on line; maybe we'll collect data on the Internet and then do some batch processing in the background, maybe a modest manual effort... The point is that the content of the project is itself a variable. It's less and less of a variable as the project progresses and it finally turns into a constant, but right at the beginning, it's very changeable. So if you're stuck with a fixed time frame and a preset budget, modify the project to fit it.
But let's say they ask you for an estimate. The first thing to shoot for is breaking the estimate down into an analysis phase and a programming/installation phase, and then giving an estimate for only the analysis part. This is actually the only sensible way to do it and the only way your estimate is going to be even halfway reliable. What you're saying is, "Maybe we should find out what we have to do before we estimate how much it's going to cost and long it's going to take." But that very reasonable idea is often countered by a less reasonable but often more compelling thought, "Sorry, but I have to have a number for the whole project so I can get it into this year's budget."
So now you're in the position of having to come up with an estimate on the basis of woefully inadequate information. This is the normal case. You could try using the offhand estimate method, but the more scientific appearing method gives everyone a feeling of confidence and, anyway, there aren't always similar past projects you can use for comparison. People say things like, "Do your estimate, then double it, add 50%, and then triple it and you'll be about right!", but that also seems to lack that desirable scientific luster. Following are some suggestions that may help a bit; at least, they represent the areas where most estimates are furthest off the mark.
For every outside interface add 50% for coding and 25% for testing. Your project needs an interface to an outside data source? Estimate that piece honestly on the basis of everything you know about it and then add 50%. If the outside data source gives you an estimate for work on their side, say "Thank you very much," and then add 50% to it for your own purposes. This applies to other departments within your company as well as sources completely outside your company. Your own computer center needs to buy equipment for your project and install it. The computer center guy says, "Well, it usually takes about three weeks to get the hardware and then we'll need another three weeks to get it installed and tested." Three weeks + three weeks = nine weeks.
Outside interfaces are a problem for testing as well, particularly for that big dress rehearsal where everyone has to come in on Saturday and run a day's work. If one of your outside interfaces can't make it, you've lost a week, and if you have four or five outside interfaces, the chances of one of them having a problem are high.
For every part of the project you don't entirely understand, add 50%. Like outside interfaces, this is a form of lack of control, but here the problem is that you're dealing with an area that you haven't had time to study. Perhaps your project is replacing that old legacy system that was written in 1980 by Joe and Mary. Unfortunately, Joe got tired of data processing and entered a monastery in 1990 where he has taken a vow of silence, and Mary moved to Silicon Valley and you lost track of her after her seventh job change. Neither of them were much for documentation. Hey, these higher level languages are all self-documenting, right? -- and the current maintenance guy is only familiar with three of the fourteen source code modules. Hmm, maybe you ought to add 100% for this one, but you get the idea.
Don't forget that testers find bugs. Thinking that they don't is a popular, but most peculiar oversight. People build time into their estimates for Unit Testing and QA, but don't build in any time for the testers to find anything wrong and the programers to fix it. It's as though they're saying, "Of course the system is perfect, but we still have to go through this silly, testing formality." The code won't be perfect and testing isn't a formality, so build in some time for the project to take a couple of backwards steps before finally marching forward to victory. How much time to add depends on the control you have over your testing environment. If you're blessed with your own standalone test system, you can turn around testing fairly quickly, but if your testing is squeezed into the cracks of a production system it could be a major consideration.
The result of these considerations will be a higher initial estimate and you'll have to deal with the sticker shock problem, but you may have a chance to make your target date.
If you do make it, make sure EVERYONE knows about it.