Note: This was an independent thought piece I wrote based on the connections I saw between Iain McGilchrest's celebrated book on brain lateralization, "The Master and His Emmissary" and Palantir's software development approach. I am not nor have I ever been affiliated with Palantir!
One of the most thought provoking books I've read in recent years is "The Master and His Emissary", written by Iain McGilchrist, an eminent British psychiatrist1. McGilchrist makes the striking claim that over the past few hundred years, the left-hemisphere of the brain ("the trusted emissary") has usurped its natural "master", the right-hemisphere – and that this cosmic battle has had far reaching (and self-reinforcing) effects on the whole of human and technological development. I'd like to try and make some connections between his thesis, and the attributes which I believe make Palantir so special.
McGilchrist views the two hemispheres of the brain as having two highly distinct ways of attending to the world – a difference in "how-ness", more important but less understood than the standard and well- known differences in "what-ness". The left-hemisphere takes an abstracted, systematized, linear, and conceptualized view of the world. It "re-presents" reality symbolically, allows us to manipulate tokens, the physical items within our grasp, all with specific goals in mind. We can calculate and logically deduce, we can focus, we can plan and then execute on that plan. Our attention is narrowly focused and relies on what the left-hemisphere in some sense already "knows" about the world.
Orthogonal to this way of being is the right-hemisphere – McGilchrist's idealized "master". It is in the right-hemisphere where the spotlight of attention is broadened, left open to whatever might exist in lived reality. The world isn't defined in terms of concepts or abstractions, objects not in terms of attributes or categories, but everything exists concretely, singularly, uniquely, special in it's own right and embedded within a context, part of a web of interconnections. The right hemisphere sees things afresh, for the first time (all the time), and has no preconceived notions about what may turn up as it encounters the world. Its attitude is one of openness, relationship, "betweenness" – in opposition to the left-hemisphere's narrowness and tendency towards isolation. Its knowledge is implicit, experiential, and intuitive.
The author's primary thesis is that the "natural" relationship between the two hemispheres has been upturned over the last few hundred years via a self-reinforcing process by which the left-hemisphere's way of attending to the world has in fact created a world that requires and (over)values a left-hemisphere dominated mode of thinking. The trusted emissary has usurped its master, and the result is a world and a people now disconnected from the natural state of a right-hemisphere dominated mode of being. Of course, the abstracted, logical, mechanistic, symbolic approach to the world has enabled the scientific, technological, industrial, and economic revolutions of the past few centuries. But at what cost – and what if we don't even know what we don't know anymore?
Viewed under this left/right dialectic, I see Palantir as a fascinating technological anomaly. Its software written by engineers who by their very nature and nurture understand and create their world in terms of abstractions, symbols, logic, and mechanisms, it nevertheless has transcended the self-enclosed, self-referential world the left-hemisphere wants to create. Palantir, at the core of its being, accepts, respects, and enables the "master" – returns to it the brilliantly engineered fruits of the "emissary". This is very different from other approaches to analysis – "intelligent" (but naive) machine learning that assumes all meaning lies completely within the data and solution method itself, and that all that is required to bring it forth are better and better algorithms, applied on more and more data, converging on some optimal solution.
This strategy has fueled immense technological innovation in recent years, and yet, there are so many examples of the failure of this approach and the extreme dangers of removing external human insight and judgement from the equation. In finance: the "flash crash", recent trading losses at JP Morgan, the $400M loss-in-1-hour at Knight. These were "bugs" and "errors", yes, but they also reflect a presumptuous faith that Moore's law has made intuition and human judgement obsolete, or at the minimum, made it approximable by predefined (if even constantly-changing) algorithms. In the consumer internet: do a Google search on any health-related topic, and witness probably the most advanced machine learning algorithm in existence, applied on one of the largest datasets in the world, deliver pages of pseudo- spammy results of obviously questionable quality (at least in the right-hemisphere's immediate and holistic judgement). Even an algorithm that takes actual human behavior as one of its many signals fails in a world where the act of measurement itself (the crawl) in effect changes the data it's analyzing (the "adaptive adversary" SEO response of website owners).
But in Palantir, the tools of the left-hemisphere are used and applied by the right-hemisphere, as McGilchrist contends they were always meant to be. Accepting the primary importance of human-derived intuition and insight as both the starting point and the end goal for analysis, Palantir enables an open and exploratory method of learning about a problem. With a SQL query, you in some sense already have an idea of what you're going to get, or at the minimum, the topology of the answer space. But in Palantir, datasets unfold in terms of relationships between individual, singular entities, and in terms of patterns of commonalities across all entities. This "betweeneness" uniquely engages the right-hemisphere, and the speed and ease with which relations can unfold on the graph keeps the right-hemisphere's attention open and engaged – an important distinction between other forms of back-and-forth analysis, where the stream of conscious attention is broken up and fragmented, limiting the opportunities for natural insight to surface organically. I need not have any predefined notion of what I'm looking for, or how I must look for it when I use Palantir – this approach aligns with the right, not the left-hemisphere's mode of being, and is what enables intuition and human insight to guide the stepwise (linear, necessarily "left") decision making process that analysis involves. A hierarchical ontology –the very definition of the way the left-hemisphere understands the world – is only useful if it is consistently applied across all data (is in some sense "static"). But Palantir has made its ontology dynamic, designing it specifically for the right-hemisphere's end goals – non-trivially allowing the very architecture of knowledge itself to change in response to analysis and to a human's best judgement. In my mind, Palantir is unique among technology companies in that its approach unites extreme technical ambition with an extremely contrarian humility that believes there are deep fundamental limits to computation's ability to approximate human thought.
McGilchrist views the right-hemisphere's loss of primacy as a supreme danger, and the cause of much of what ails the modern world. While being aware of the risk of asserting a false dichotomy, in his view, the left-hemisphere is simply not equipped to know. All it can do is manipulate and process, but it is not the source of meaning – only the right-hemisphere understands. In technology, then, it is critical to make distinctions between those products and processes that put the left-hemisphere in its rightful place – in the service of ultimate meaning and insight that only the right-hemisphere is equipped to understand, appreciate, and create. In my mind, Palantir is singular – there is no technology company that better understands that "concepts without intuitions are empty, intuitions without concepts are blind."