One Bad Bid
...the consequences of fat tails
The whole point of estimation is to support decisions. If an estimate doesn’t help resolve uncertainty about a decision we have to make, then it doesn’t have any value. It’s not helpful when pro forma estimates are taken as performance targets. In contrast, there is substantial value in the estimation process when it is understood as a quantitatively expressed reduction of uncertainty based on observations.
The Perils of Parametric Estimation
One of my earlier consulting gigs was doing estimation for an electrical contractor; My client taught me his process, which we modeled in Quattro Pro. Spreadsheets seemed like magic in those days. This construction guy told me that what appeared to be a question of sorting out time and materials, is really more like of a problem of topography, where we’re mapping time and materials to a schedule to reveal the landscape where the project will play out.
There was a story in the news at that time that made a big impression on me: the bankruptcy of the largest construction firm in Vermont. I knew some of these guys, a family business that went up in smoke over a single bid. It was the Northfield Elementary School re-roofing project.
The lucky man has nothing to worry about.
— Sancho Panza
The estimator was an expert; he’d done a lot of these projects. He walked the roof for inspection before writing up his bid. What he missed was just how many times the roof had been resurfaced over the years with layers of tar and gravel, all hidden below the surface.
The removal of the old roof was included in the fixed bid, and, as it turned out, the cost of tearing up and disposing of the tons of old, hardened tar ran into the millions. To have gotten that bid right, he would have had to have core drilled the roof; then there would have been no doubt. Instead, he made an assumption that turned out to be very wrong.
Ever since then, whenever I’ve been asked for an estimate, I can’t help but to think about that poor guy standing up on the rooftop, about to bring his company down. It brings to mind that old line:
It’s not what you don’t know that’ll hurt you, but what you know that ain’t so.
— Mark Twain
That is to say, expressing certainty about something you have no rights to be certain about.
A measurement is a quantitatively expressed reduction of uncertainty based on observations.
— Douglas Hubbard
So I’ve been thinking about this for a long time, but it was only recently that I discovered how to leverage uncertainty in estimation, and that’s what I want to share with you.
Let's agree to define productivity in terms of throughput. We can debate the meaning of productivity in terms of additional measurements of the business value of delivered work, but as Eliyahu Goldratt pointed out in his critique of the Balanced Scorecard, there is a virtue in simplicity. Throughput doesn’t answer all our questions about business value, but it is a sufficient metric for the context of evaluating the relationship of practices with productivity.