Tuesday, August 18, 2020

Bottom-up and Top-down

The great insight in the theory of natural selection is that nature is bottom-up optimized rather than top-down optimized. 

A bottom-up optimizing system is characterized by local equilibrium equations. A top-down optimizing system is characterized by a global objective function. 

Top-down paradigms are popular among both scientists and mythologists. Here is an anecdote told by my astronomy professor. Why does Jupiter have so many moons? Top-down philosophers in the 1600s said that the reason is that the people on Jupiter are great scholars who like to read at night, and the many moons are there to provide them with plenty of reading light around the clock!

It takes a great effort to wean oneself from the impulse to apply top-down optimizing models to systems that are actually bottom-up. Genetic populations are bottom-up optimized. A genetic population stops evolving when there is no variation in the population that has a higher or lower relative fitness. This is not the same as the population having achieved perfection in any sense. Likewise, a variation that has a high relative fitness in the population, at the expense of decreasing the fitness of the species as a whole, can still prosper. 

Free markets are also bottom-up optimized. A market stops developing when there is no way to get an investment to create a company (or modify an existing one) that will produce the product with a margin that will yield a higher return on investment than the current interest rate. This is not the same as optimally converting given limited resources into the product. 

Now we get to looser ground. Something I've learned in my working life is that large organizations are also bottom-up optimized. A large organization stops changing when there are no changes that can be done within any manager's budget that will increase that manager's power. This is not the same as the company producing its product optimally. It's not entirely true that large corps are bottom-up. There can still be heroic figures who do things that help the company that get ignored, or worse. Small companies have a coordination boost relative to large companies because "stuff that increases an employee's status" and "stuff that is good for the company" are much easier to align in small companies than in large ones. 

So, can markets and large organizations be made to behave in a more top-down manner? Yes, if they are in the presence of a large source of energy. The word energy will have to do here, for lack of a better word.  For the market and the organization, the best example of an energy source is an unexploited innovation. In the presence of many great unexploited innovations, bottom-up systems can behave in a very harmonious and coordinated way. When the innovations run out, chaos, inefficiency, and stasis returns.

Something that should always be asked: what does the above model of reality predict and, more importantly, what does it prohibit? First of all, this should reduce our trust in institutions that were put in place during great ages of innovation. It's possible that they were always broken, but that their brokenness wasn't noticed because everyone was high on innovation-dope. 


The most important use of the bottom-up view is as a tool to analyse possibly broken organisations. When looking at an organisation we can ask "is this organisation behaving as if it had a single goal, or is it behaving as if it consisted of several semi-independent parts with possibly contradictory objectives?". The most apparent red flag is that one part of the organisation is working against another part. Another red flag is that one part is impeded by another part for a long period of time.

No comments:

Post a Comment