Competition Like Rivers
This was originally titled ‘a short digression’ in a more personal piece I’ve been writing. It was not short, so here it is.
In a world that is becoming increasingly more competitive, along virtually every dimension, the most successful entities in any space will be groups of highly specialized individuals. There will be no more Renaissance men, no more polymaths.
If a given characteristic (income, beauty, fast-twitch muscle fibers) follows a power law distribution within the population, then there is a long tail of people at the top that are dramatically better than the rest. If you've watched NBA pro-ams recently, this should be pretty obvious, the best amateurs are nowhere close to a journeyman NBA player. But even within the tails, there is greater and greater dispersion as you get farther out.
(an example 80/20 power law distribution)
Most people tend to compare themselves to people who are similar to themselves. To stick with the basketball analogy, if you are an average player, you might play a weekly pickup game with people of similar skill. The variance among your friends, then, is not enormous, some days you can win and some days they win. This is fun, people like to play challenging games, you can all say you tried hard, whatever.
If you are a 99th percentile player, say a D1 college player, your comparables are now much farther apart in terms of skill. You might play a game against a mediocre college player with no hopes of making the NBA one day, and the next day against the sure-thing number 1 pick. This can be fairly demoralizing, especially since you are used to playing more properly matched games when you were a lower skill level.
This is easily modeled mathematically - assume that in a population, true skill is distributed according to a Pareto distribution and that win probabilities follow a simple Bradley-Terry-Luce model. Then, we can simulate multiple round robin tournaments among players and see the distribution of wins per players. As expected, ‘average’ players (drawn from the 45th-55th percentile of skill) have very tight groupings of wins close to 50%, while ‘top’ players (above 90th percentile) are very skewed with outliers dominating their wins, with the top player winning almost 95% of their matches. This is despite everyone playing within a 10% band of their ‘true’ level!
I'd argue this is why "keeping up with the Joneses" maintains such a stranglehold on the yuppie class and why the hedonic treadmill is such a drag - as you get higher out on the skill distribution, there is a wider disparity among your comparables, and your chance of winning drops too.
A corollary here is that "first principles thinking" is kinda worthless. Maybe you use that to judge a problem at a high level or for truly de novo problems, but it’s quite arrogant to think that economically important problems have been approached totally wrong before. Exceptions to this rule tend to be along discriminatory lines - e.g. sports teams willing to accept black players or use “moneyball” statistics over biased scout judgements, or Alan Greenspan hiring female economists when others wouldn’t. Markets are efficient, except where humans are involved.
So why do we compare ourselves? Life is nothing without struggle. You struggle to secure food and shelter, then you struggle to secure a better car and better clothes, then you struggle to secure a bigger vacation home and boat, then you struggle to get to Mars first. The Buddhists say that nirvana is the extinguishment of desire, and are they wrong?
But back to the original point, athletics are a stark case where humans have gotten far better over time. We can attribute some of this gain to better training (NBA players used to smoke cigarettes at halftime) but I think it's mostly dominated by a growing world and growing access to the sports. NBA scouts will travel anywhere for 7' teenagers with decent footwork, HFT quant trading firms sponsor worldwide math competitions for high schoolers, tech companies will hire anyone who can do 3 LeetCode mediums in an hour, regardless of where they live. We are also a cultural species, who learn from previous generations - the entire work product of Renaissance mathematicians might be a sufficiently advanced high school class, and pre-Renaissance work might be a middle school class.
The title for this post comes from a Chinese idiom, 车水马龙, which is usually translated as an endless stream of cars, and refers to heavy traffic. But another way I translate it is that there are so many cars, or droplets of water, that they make up a raging river, flowing fluidly. Deleuze describes late capitalism as creating infinite smooth spaces, which are essential to the flow of capital (free trade good, global markets good, etc). In our interconnected world now, every important problem has such a fluid flow of humans, incentivized by capital, to solve them.
From Islands
(source)
One tradeoff of being the 'best' in a field is that you must sacrifice other things to get there. Michael Phelps' extraordinary wingspan might have been caused by Marfan syndrome, tech bros and doctors lose their money in crypto scams, Premier League soccer players flat out can't read. I was a good but not amazing badminton player as a kid and wound up with nearly 20% more muscle on my right side, which still causes lots of muscle balance problems.
Flora and fauna on islands similarly display hyperadaptive characteristics. The dodo bird, as much as it is a punchline today, was incredibly evolutionary successful: after arriving on the island, it lost its ability to fly, but flying is a waste of energy if there are no predators to escape from on an island, and the most energy-efficient animals tend to win. Of course, once the environment changed and predatory humans arrived, the dodo died out.
Most careers are also islands. You wouldn't trust a dentist to do hip surgery and you wouldn't trust an orthopedic surgeon to fix a root canal, even if they were the very best in their domains. Many domains understand the negative consequences of competition, and artificially limit the number of new entrants - the American Medical Association for doctors, the bar exam for lawyers, summers in Montauk for bankers.
Software engineering is a notable exception, and it's already become the norm for early stage startups with a non-/semi-technical founder to outsource their development to Eastern Europe. Global competition makes the market for SWE jobs more competitive, but simultaneously means that the variance in the tails is more pronounced. Forget the 10x engineer, the 100x engineer is what you're looking for at that point.
My sense is that median SWE compensation will decrease and that we have oversupplied bootcamp grads etc, but the high end will continue to be high and potentially grow even higher, as more businesses recognize that value. Facebook, for example, has a general hiring freeze but continues to hire generalist E7+ (the top 3% or so internally) and ML engineers.
As the variance gets higher, it's harder and harder to evaluate that true underlying skill level -- A players hire A players, B players hire C players. That's partially because of the higher variance overall, and partially because the measured skills are different and ability to measure skill in one dimension doesn't necessarily transfer to another dimension.
To Archipelagos
(source)
I propose that strong teams should be 'archipelagos' - whilst people of a similar background might be a single island, archipelagos are made up of related but distinct islands, each specializing in a different space. The islands are still connected to each other and share common elements
Archipelagos solve some key problems:
Clear delineations of scope minimize number of cooks in the kitchen and reduce chances for conflict. The biggest engineering trends of the last decade solve organizational problems that arise from scale - microservices beat monoliths when you need clear org divides, GraphQL and Protobufs help cross-language and cross-domain teams enforce transport-level unity,
Islands can enforce strong cohesion and cultural similarity, which likely will not scale to the entire archipelago. I've worked on data science teams made up of ex-social science PhDs, ex-stats undergrads, and ex-physics PhDs. As a stats/econ (kind of a social science) undergrad, I got along with the ex-stats undergrads > ex-social science PhDs > ex physics PhDs best. Cultures are enforced not just at the level of a discipline, but at the level of companies: ex-Bain folks (data driven) often clash with ex-McKinsey folks (stakeholder management focused). It's islands all the way down...
Archipelagos scale to new problem spaces. As companies scale and new problems get large enough, you can say "here's a bucket of complexity that we don't know how to best handle, find someone to do it" - and because people are specialized, there's probably someone out there who can solve your problem. The more economically important your problem, the more likely that other companies have worked on it, and that you can find someone with the right experience. The worst outcomes I've seen are where early employees are 'rewarded' with a leadership role completely out of their depth - sometimes this can work, but as noted elsewhere, the more economically important the problem, the more people with knowledge, and the wider your possible outcomes.
There are of course still problems - all of this is predicated on the founder(s) being able to hire good leaders for each of the islands first. If you have an ads problem, sure you could hire from the Google or Facebook ads teams, but how do you get the good ones? How do they tell that you are good? Exercise left to the reader.