Introduction
A Discovery Process is a piece of work that takes approximately zero time to do the second time one has to do it. In other words: something that is extremely easy once you know how to do it. Modern mental work abounds with discovery processes, such as finding specific information, finding out what is causing a problem, and solving abstract problems. It is a natural consequence of the computer and the internet: if the result of one's work can be copied without cost, then it only needs to be done once, ever. Since it is only useful to know how long a specific Discovery Process takes to complete when we have exactly zero data on it, it is inherently difficult to estimate the time it will take. This creates problems for those of us who want to produce good work reliably, as well as for those who are dependent on it. In this post, I will analyse some consequences of the abundance of discovery processes for modern organisations, and for individuals.
Examples
Let's first hone in on the definition so that we know what we are talking about. A simple test as to whether a given task is a discovery process or not, is to ask what would happen if the work's deliverable was lost. Suppose a craftsman makes a beautiful table. The table itself is the deliverable of the work. If the table is stolen or destroyed, replacing the table will take almost as much time as making the first table. I say almost as much time, because I'm counting on some learning from the craftsman [1]. The bigger the "design" part is of the tablemaking, the less it matters that a single table is lost. Compare with a program: most of the work in programming is finding out the right way to do something. If the code is lost (immediately after the coding is complete), it will not take nearly as much time to rewrite it. This applies more the more high-level the programming language is. Somewhere in between we find classic engineering, where a prototype or production process is developed and set up. Having to start over that work would be frustrating, but it would probably take at most 50% of the time the second time around. This pattern (carpenter, industrial engineer, programmer) gives us a clue for how work will develop in the future: all work, that is not about maintaining relationships or signalling, will become more and more like an ideal discovery process. If this is really true, it seems extremely important for us to start making sense of the properties of discovery processes.
Estimating remaining time
I propose a simple model for estimating the remaining time of a discovery process: if one has worked X hours on it, then on average it will take another X hours to finish. A bug that has evaded the programmer's attacks for 1 day, will on average take another day to resolve. A conjecture that remains unproven despite 10,000 mathematician-years will take on average 10,000 more mathematician-years to finally crack (though I expect that many years from a single mathematician will pay off much better than a few years from a great many, who will have to redo a lot of thought-work between each other).
Caveats
When does this not apply? It does not apply if we know that the right answer hides in one of a finite amount of places, and we have to do some nonzero amount of work for each place (such as compiling, restarting, or moving a lot of data). In those rare cases do we ever find ourselves saying something like: I have worked on this problem for 4 hours, but now I am certain that it will not take more than another 2 hours. These are really rare cases, and they're actually not good signs. A team lead who hears that a piece of work is certain to finish in a known amount of time is probably happy to have a solid figure for once, but they shouldn't be! The fact that one's organisation can know how exactly to do something but not actually be able to do it (yet) means that the work is not progressing at the speed of thought! It means that the bottleneck in development is not the most expensive part (developers' brains) but a relatively cheap part (computers) [2]. When I hear that a software project will take another 2 years, it makes me think that either their organisation is bottlenecked by the wrong things, or that they cannot be certain that the project is even possible.
Another example when we need to locate a specific piece of information in a book. After the "shortcut" approaches have been tried, such as grepping for keywords, googling for a quote, or trying to narrow down the search by looking at the table of contents, we know that finding the information (or realizing that the information is not in the book) can take no longer than it takes to read the entire book from cover to cover. However, this is significantly longer time than it takes to try all the shortcut approaches, so we're better off starting with them.
Consequences of having a job dominated by Discovery Processes
- Constant uncertainty about when tasks will be finished. This is actually a good thing for the slacking employee: uncertainty of hardness of tasks makes it difficult for the employer to know how to price it. In the other extreme, highly repetitive work, the employer has very good information about how long tasks take and can therefore set a price just at market price for the class of workers who can perform the task. The information asymmetry should make us expect that companies are very hesitant to start paying for new software projects, especially ones that sound a bit weird. The result is largely the state of software consulting that we see today: well-paid consultants working less-than-optimally on software projects that seem safe and doable to a big organization.
- Divide and conquer is golden. Any insight about how a discovery process may be divided into independently solvable parts can drastically reduce the time is takes to solve it.
- Knowledge is valuable. Having a large toolbox of general solution methods to be thrown at any new discovery process can make a developer much more productive.
- Experience is valuable. Having heuristics for discovery process in one's field can help with prioritizing solution methods.
- Curiosity is valuable. Being mentally malleable enough to reshape one's mental models on the fly when working on a new discovery process can make up for lack of experience.
[1] This gives us an alternative definition for an (ideal) discovery process: it is work that has an infinitely steep learning curve.
[2] This does not apply for industries that are limited by their tools, such as large-scale machine learning, super computing, or astronomy.