Beyond the Transformer: After the LLM, Then What?

Written by Ben Esplin

The transformer architecture stands today where vacuum tubes stood in the late 1940s. Revolutionary? Absolutely. But also hot, power-hungry, and, as I argued in a previous post, fundamentally unsustainable at scale. Transistors didn't merely optimize vacuum tubes—they replaced them entirely. Hopefully, we approach a similar inflection point where fundamentally different architectures will achieve not 10x improvements but 1,000x gains in resource efficiency.

The Quadratic Problem

Transformer-based LLMs dominate contemporary AI, yet their architecture embeds profound inefficiency. Doubling a document's length quadruples the energy cost. This makes transformers extremely expensive for analyzing long sequences—full genomes, legal repositories, or comprehensive patent filings.

Optimizing transformers yields 10x efficiency gains, but replacing the architecture entirely promises 1,000x improvements. We face not a physics limitation but an architectural choice we can change.

Some Potential Hopes

Traditional neural networks fire every neuron every time. Spiking Neural Networks (SNNs) function differently—operating only when something changes. This approach can lead to efficiencies that are orders of magnitude, especially for analysis particularly well suited to event-driven architectures, such as computer vision systems that are responsive to perceived changes.

Developed at MIT's CSAIL, liquid neural networks change their underlying equations during inference, adapting in real-time without retraining. Sample efficiency appears to be remarkable. Complex control tasks, such as lane-keeping, have been demonstrated using as few as 19 neurons with liquid networks versus thousands for deep learning. These networks supposedly learn causality rather than mere correlation, enabling trustworthiness for safety-critical applications.

Contemporary LLMs perform "next token prediction"—sophisticated pattern matching. Neuro-symbolic AI pursues a different goal: building models that understand relationships and logical principles rather than memorizing correlations. Instead of learning that "1+1=2" from frequency in text, these systems learn the underlying rule, enabling generalization from first principles. This dramatically reduces training data requirements and energy consumption, as the system learns logical structures applicable across contexts.

The Innovator's Dilemma

Here lies the strategic tension: OpenAI, Google, Meta, and Microsoft have invested billions in GPU clusters designed specifically for transformer architecture. Meta, xAI, and OpenAI/Microsoft race to build clusters with over 100,000 GPUs, each costing more than $4 billion in server expenditure. A 100,000 GPU cluster requires over 150MW and consumes 1.59 terawatt hours annually.

These investments lock incumbents into the transformer paradigm. Just as successful companies fail to adapt to disruptive technologies, AI leaders face structural barriers to embracing architectures that render their GPU investments obsolete.

Yet transition to any new AI paradigm poses practical challenges. Neuromorphic computing requires new development tools and programming paradigms. Liquid networks remain relatively new. Neuro-symbolic systems demand formal knowledge representation alongside neural learning.

Existing datacenters optimize for GPU-accelerated deep learning. Neuromorphic chips require specialized support and integration. This creates a chicken-and-egg problem: limited hardware deployment constrains software development, while limited software availability slows hardware adoption.

A Path Forward

The most valuable near-term application of current LLMs may be designing their own replacements. Neural Architecture Search, enhanced by large language models, uses existing AI to discover novel architectures automatically.

LLM-guided NAS explores architectural possibilities humans might never consider, discovering unconventional activation functions, connection patterns, and structures achieving superior performance. MIT researchers proved neural networks can be designed optimally by selecting specific activation functions derived through theoretical analysis.

Current transformer-based models enable rapid exploration of alternative architectures—using expensive, inefficient scaffolding to design efficient permanent structures. This represents accepting temporary waste to achieve lasting conservation.

The transition ahead demands balancing immediate needs with long-term sustainability. Transformer-based models serve important functions today but cannot represent the endpoint of AI development. As jurisdictions increasingly mandate energy accounting, companies developing fundamentally more efficient AI will gain regulatory advantages.

The challenge lies in ensuring we dismantle the scaffolding before resource costs become unsustainable. Success will be measured not by how long we sustain the current paradigm but by how rapidly we design its superior replacements and how gracefully we transition to systems achieving more while consuming dramatically less.