Solving shtoopid problems

By Eric Raymond

There is a kind of programming trap I occasionally fall into that is so damn irritating that it needs a name.

The task is easy to specify and apparently easy to write tests for. The code can be instrumented so that you can see exactly what is going on during every run. You think you have a complete grasp on the theory. It’s the kind of thing you think you’re normally good at, and ought to be able to polish off in 20 LOC and 45 minutes.

And yet, success eludes you for an insanely long time. Edge cases spring up out of nowhere to mug you. Every fix you try drags you further off into the weeds. You stare at dumps from the instrumentation until you’re dizzy and numb, and no enlightenment occurs. Even as you are bashing your head against a wall of incomprehension, consciousness grows that when you find the solution, it will be damningly simple and you will feel utterly moronic, like you should have gotten there days ago.

Welcome to programmer hell. This is your shtoopid problem.

Yes, I have a real example. I just spent the better part of three days debugging code to close block scopes in Python, for a little tool that translates Python into a different language with explicit end brackets. Three days! Of feeling shtoooopid.

It left me wondering what it is about some apparently simple conundrums like this that repels solution. I can say that there are certain recurring patterns in shtoopid problems – when you’re experienced enough, you will notice them partway in and get that sinking oh-no-not-again feeling.

A big one is when your algorithm is such that off-by-one and fencepost errors that are easy to make and hard to spot. The defects that go with this are reversed sign on an increment, or trying to step through an array or list or code loop by bumping the wrong counter.

Another recurring shtoopidness trap is near data sentinels – that mysteriously don’t. And a third is any situation where you are mutating (especially deleting) parts of a serial data structure while iterating through it. (Some languages have mutability restrictions in loops to try to prevent this one. The Dread God Finagle can easily contrive a way for you to screw yourself despite them,)

Copy-paste code that is just slightly wrong in a way that is difficult to spot is a common source of shtoopidity. So are undetected namespace collisions. And variable shadowing. And reversed test in conditional guards.

If you ever find yourself staring at your instrumentation results and thinking “It…can’t…possibly…be…doing…that”, welcome to shtoopidland. Here’s your mallet, have fun pounding your own head. (Cue cartoon sound effects.)

Solutions to shtoopidity traps are not conceptually hard, but they’re slippery and evasive. They’re difficult to climb out of even when you know you’re in one. You’re not defeated by what you don’t know so much as by what you think you do know. It follows that the most effective way to get out of your trap is…

…instrument everything. I mean EVERYTHING, especially the places where you think you are sure what is going on. Your assumptions are your enemy; printf-equivalents are your friend. If you track every state change in the your code down to a sufficient level of detail, you will eventually have that forehead-slapping moment of why didn’t-I-see-this-sooner that is the terminal characteristic of a shtoopid problem.