Tacit Knowledge


In my previous post I sketched out a solution to a complicated financial analysis problem I worked on a while back. It occured to me that I may have left a wrong impression with that post: namely that I know a lot about financial analysis, quadratic optimization and Fortran. The truth is I know very little about these subjects. I barely got through linear algebra in college, my financial knowledge consists of a rudimentary understanding of stock investing and I've never even written a Fortran program.

So the question is: just how is it that I was able to develop a solution to a problem using a set of technologies I only barely understood? Is such an ability unusual?

No. I don't think it is very unusual. In fact, I think that just about any competent software developer could've come up with the same solution (or perhaps a better one). And because of this I came to this conclusion:

The software industry overvalues comprehension and explanation.

We suppose that the best way to guage someone's abilities as a software developer would be to guage their ability to answer technical questions and to supply white-board solutions to programming problems. If someone can't give you an example of any sorting algorithms or explain how such algorithms work, then how can you possibly expect them to implement a program which can sort a list of numbers? And isn't the reverse true as well? If someone successfully produces a program which can sort a list of numbers, doesn't that imply that they know a sorting algorithm and can explain how their code works?

But, as unintuitive as it may seem, these are in fact very different skills. Knowing how to program is very different from explaining your program, or even understanding the program you just finished writing.

Michael Polanyi famously observed that: "We know more than we can tell". This kind of knowledge, which is difficult to transfer, is called "tacit knowledge" and examples of it are abundant in everyday life.

For example, take the knowledge of how to play an instrument. We might say that the formal knowledge of how to play a guitar involves positioning your fingers properly, with the appropriate amount of pressure and for the appropriate duration, in order to produce the correct set of notes, to play a song.

But, crucially, such knowledge is nothing like what it means to play a guitar at all. This is indeed, how one begins to learn to play the guitar: by attending to such minute details. But once you've gained any proficiency with the instrument you no longer do so. Your knowledge of how to play the guitar is tacit, no longer formal, and the truth is you probably can't explain what you're doing. You don't even "know" (formally) how much pressure you are applying, or how long you are holding notes down. And yet, you do "know", because the music you produce shows that you know how to play the guitar.

Most people have a category for such knowledge when it has to do with mechanical skills. (playing an instrument, riding a bicycle, throwing a football) We call it muscle memory. But it's not just physical activities which are tacit. Playing an instrument can involve sophisticated mental activities. For example you might be in a jazz band, and you're not merely reading from a sheet of music (as if interpreting sheet music itself weren't a sophisticated mental activity).

Perhaps you've had this experience: you were driving somewhere a few hours away, you put some music on and at some point you look down at the clock and notice that over an hour has passed. And it dawns on you that you can't remember a single thing you've seen on the road in all that time. And yet you were able to drive for an hour without thinking about it. That's because the knowledge of how to drive is tacit, not explicit or formal.

Programming is no different. Competent software engineers are capable of breaking up a very large, complex problem into component pieces, implementing solutions for each of those pieces, and then combining those pieces into an aggregate software application. They do this by (along with a myriad of other skills) black-boxing systems, following best practices, ignoring unimportant details, and implementing a nearly robotic validation process (quality assurance, unit testing, ...). But they rarely consider what they are doing.

Speaking personally, I've implemented entire features one week and been completely unable to remember how they work the next. I'll dive in on a bug, do a bit of googling, fix it and then I move on. Forgotten. Just as if I were cooking a meal I cook all the time, or going through my morning routine. These are the habits of life and since programming is something I do (almost) everyday it shouldn't be surprising that it ends up working the same way.

And yet our entire evaluation system fails to account for this. We ask interview candidates to whiteboard software solutions, which, if you think about it, is roughly akin to asking a painter to paint without his brush or a carpenter to make a chair without using nails. We evaluate skills used in school (answering questions on a test, verbally explaining to prove comprehension) but which have almost nothing to do with the actual production of real software.

Suppose you spent 6 months learning vi keyboard commands, then spent another 5 years using them every day. We can say that you know how to use a text editor. But you're probably no different from someone who knows nothing about text editors when handed an entirely different text editor (like emacs). Your knowledge of text editing adheres in the very tools you use.

Now imagine someone comes up to you and asks you how to do something using vi. You might be able to explain this on the spot, but more likely you need the actual editor & keyboard in front of you to think about it.

And what's sad is in an interview you'd get rejected by your inability to deliver an answer to that technical question about vi.

Actually it's worse than that: formal knowledge is often evidence of exactly the opposite of what you're looking for. That guy who can actually answer your technical questions? He probably read them in a book yesterday. It's as if you were trying to hire a guitarist, and rejected Paco de Lucia because he didn't know how to read music.

Now I don't have a better solution for interviewing; tacit knowledge is a hard philosophical problem and it often looks like incompetence. But perhaps the next time you evaluate a candidate you should come to the table with a bit more humility about your own ability to assess his or her skills.

I'll finish with this: Programming is an amazing thing. We make computers do things. And though from the outside it looks like we must have a total grasp of everything that that computer ends up doing, this is an illusion. Like the production of a pencil, the part I play in the production of software is but a small cog in an incredible, emergent order. It's amazing that the programs I get to write actually work in the end. But what's perhaps more amazing is that I don't even really understand how it is that I wrote them.