Tokens of the future
"A CPU is a rock we’ve tricked into thinking."
Where we’ve been
It was interesting to me to observe the advancements of CPUs as a layman. I still remember reading tech magazines as a child and later teenager, always coveting that new CPU/GPU. Before multithreading, cores and caches became important; one of the main metrics was CPU frequency. It seemed like almost every month there was a new chip that could make games run faster, which was my main concern at the time. MHz at first, GHz soon thereafter - Moore’s “law” was in full effect. [1] A 2 GHz Pentium 4 CPU was my first not-handed-down chip, and my most prized possession at that time, one that required extensive bargaining, begging, and convincing my parents to acquire. But it ran Half-Life 2, so everything worked out.
Over the years, the pure GHz arms race slowed down. Sure, you could (and can) cool your CPU with dry ice and reach insane clock numbers, but frequencies practically stalled and still remain in roughly the same ranges today. [2]
If we skip over some very important concepts, such as instructions per clock, multi-core chips, multithreading, and much more, we arrive at what seems a significant shift in architecture that is ARM - a CPU isn’t a general CPU anymore. Instead, it houses high-performance cores, low-performance cores, neural processing units, image signal processors, secure enclaves, media encoders, all on the same die. Put simply, why use a general-purpose CPU core to decode a 4K video when a dedicated media engine can do it 100x faster using 1/10th the power? Apple M, Snapdragon, and other ARM chips suddenly pulled portable devices up a significant step.
Where we’ve been - even farther back
A processing unit that has specialized sectors optimized for various tasks it might encounter… Doesn’t that sound a lot like the structure of a brain?
Over millions and millions of years, trillions and trillions of iterations, the organ we call “a brain” developed (in tandem with lots of other components), to enable biological systems to navigate what is perceived as an environment where these systems coexist. A few levels downstream of elementary particles, atoms, molecules, cells, tissues, organs,… we encounter Animalia, but there is no specific reason to focus just on that level. Everything is a system of subsystems, and my reason for staying within this scope is that I (and you?) am also an Animalia, and so it is convenient to predominantly stay within the dimensions we inhabit and can somewhat influence. [3]
In terms of raw power, a human brain is considered to consume around 20W. For better or worse, it manages all the processes in the body - it takes data from built-in sensors, processes it, and sends out electrical commands to various subsystems. It takes care of subsystems like the circulatory system, the respiratory system, thermal regulation, it calculates the physics, timing, and spatial geometry required to execute movement,… On top of that, some people have enough processing power left over for abstract thought to emerge. Planning, communicating, “believing”,… All things considered, a lot is done with 20W. [4]
I always understood (or better: imagined) consciousness as something that emerges on sufficiently powerful hardware. I have a strong disdain for all the hand waving, god seeking, and mysticism that is attached to the experience of being a self-aware system. A giraffe has a long neck. This enables it to strip leaves from trees otherwise not reachable to other biological systems, this is its edge. Other animals have other edges, ours is the ability of abstract thought (and fine motor skills, and…). As systems evolved, they found their niches in the broader system they inhabit (for example: Earth) that ensured their survival and replication.
All designs also have negative externalities: a giraffe requires a circulatory system with a very high blood pressure. The very same ability that enables humans to cooperate en masse (alone I hunt rabbits, together we hunt mammoths; the concept of a limited liability company; etc.) is also the same system that in turn leads to tremendous horrors done to each other in the name of abstract concepts. Very quickly we seem to forget that we share 90% of our source code with cats; that we will all end up in the same exact spot (dead); that the sun will stop working, and so will the universe. We have an external enemy, in some sense, we just don’t see it.
Strip away the exact produced sounds and keep only the information; the messages shared between humans on a daily basis in Tokyo, Paris, and Mexico City aren’t that much different, even though they sound very different. But we all sleep, eat, go to work… Our actions are enumerable, and as Noam Chomsky researched, there are underlying patterns in human communication. Which brings us to:
Where we’re going (maybe)
Consider now the latest LLM models. They give out the appearance of thinking, understanding, communicating. Suddenly, we are feverishly debating “what is intelligence”; will we remain the most intelligent system on Earth; doomsday scenarios are peddled in the media, etc. etc. At the time of writing, words like “paradigm shift” and “industrial revolution” are being spouted everywhere you look; certain models are being treated as national security risks. And yet, underneath, they are statistical models, layers of matrix multiplications, linear algebra. [5]
At the same time, it takes immense amounts of energy in order to develop, train, and run all the broader technology necessary to reach even the current state of models. So much so, it seems it took a development like this for humans to start turning to nuclear energy more seriously again, as if it wasn’t really worth battling bad PR until now. This alone could indicate this particular technology tree branch looks promising enough to be vigorously pursued.
Finally, if at the moment it takes tremendous amounts of energy to get to current models, and processing units are becoming more and more specialized (with great results), and our brains still outpace our current best processing units Watt by Watt, then it might be an immense pleasure to wait for the capabilities of models to go up, while Watts per token comes down - not even considering that even current models are in several aspects truly impressive and more useful than a single human could ever be. At what point will our 20W brain meet processing units of equal output, and how much energy will that unit consume?
Would That It Were So Simple
As impressive as contemporary technology is, true simulation[6] still seems far away, with advancements in many fields required to happen before the concept as a whole can advance. C. elegans, a tiny worm, possesses only about 300 neurons and has been studied extensively over the last 30 years, yet a true “simulation” of its brain still eludes us. It is not enough to know how neurons connect to each other through electrical and chemical synapses; we also need to understand the biophysical parameters governing each neuron’s behavior. [7]
What is going on, then, when a freely accessible LLM can explain in astonishing detail everything about C. elegans, seemingly surpassing the capabilities of 300 neurons by an exponential margin, yet at the same time scientists show us we are clearly not there yet, or perhaps not even dealing within the same realm? Can we push this technology further into new paradigms by providing more power, more compute, and more data, perhaps a model that is self-correcting and self-developing? And what is an original thought, anyway, if everything, elementary particles, and neurons alike, exist in relation to each other?
Moore’s Law is the observation that the number of transistors on an integrated circuit will double every two years with minimal rise in cost. ↩︎
The reasons are unfortunately the laws of physics. ↩︎
There is no free will, but from my perspective, there is. Ants don’t care about our highways. ↩︎
I hope it’s not just me, but I distinctly felt “slower in the head” while at a hackathon in Athens. Air conditioning solves the problem, but the operating temperature window of a human is worryingly narrow. ↩︎
I am not in the slightest minimizing the importance of the technology. It is rather marvelous what layering does. ↩︎
“a good simulation of a nervous system is one that both accurately replicates its functionality and reliably predicts the future activity of a real system under the same initial conditions.”; Michael Skuhersky: We Can, Must, and Will Simulate Nematode Brains ↩︎
Michael Skuhersky: We Can, Must, and Will Simulate Nematode Brains ↩︎