Amid the heated debates about AI's rapid progress and its future impact, one fundamental truth is often overlooked: AI is, at its core, a computational model that operates on machines built from 0s and 1s. To seriously consider what AI can ultimately achieve and to what extent it might replace humans, we must return to the foundations of computation itself and ask whether the human mind can, in fact, be fully and precisely captured by computation.

What computation is, and isn’t

Computation is the systematic manipulation of symbols according to rules. Alan Turing and Alonzo Church formalized this idea in the 1930s, showing that any algorithmic process can be modeled by a Turing machine. This insight, known as the Church-Turing thesis, defines the scope of what machines can do.

But computation, while immensely powerful, has hard boundaries and cannot capture all the truth in the universe. A few most important theoretical limits:

• The Halting Problem: Turing proved that no program can decide, for every possible program-input pair, whether it will halt or run forever.
  
  
• Gödel’s Incompleteness Theorem: In any sufficiently expressive formal system, there are true statements that cannot be proven within the system itself.
  
  
• Chaitin’s Constant: Certain numbers that encode algorithmic randomness are uncomputable - no finite procedure can determine their digits.

Taken together, these results show that there are truths and structures forever beyond computation. When we talk about AI, we must remember that as a computational mechanism, AI inherits these very same limitations.

The unique problem of AI

Most sciences use the human mind to study external phenomena: physics investigates matter, economics analyzes markets, and biology explores life. Artificial intelligence, however, is different. It is the human mind attempting to model the mind itself. This creates a unique circularity: the brain becomes both the subject and the object of inquiry.

From this circularity arises a profound question: Is the human mind equivalent to a logical system and program, and can computation fully capture such a self-referential study?

• If the answer is yes, then in principle AI could evolve into an all-encompassing program capable of replicating every aspect of intelligence.
• If the answer is no, then AI encounters an inherent ceiling: as a program, it may approximate but can never fully embody the richness of human thought.

The paradox of invention

Let us imagine that human minds one day succeeded in inventing a program fully equivalent to the human mind itself, thereby solving the problem of artificial general intelligence in the most radical sense. Such a scenario would imply that intelligence is nothing more than a computable program.

Yet this thought experiment quickly encounters a paradox. If the mind were merely computation, then since the mind invented such a program, the program would be able to invent the same program of itself, autonomously recognizing itself as the ground of intelligence.

This is not to be confused with self-replication or self-optimization, which software systems can already achieve. Modern AI systems like large language models can generate traces of thoughts, then autoregressively predict future tokens based on previous generations. But this is not the same as reflecting on the mechanism that produces those very thoughts. A model can simulate the appearance of reflection, but it does not grasp its own generative process as a system. By contrast, for a program to invent the intelligent program itself requires something more radical: a higher-order act of self-realization and self-consciousness: the capacity to recognize the code itself as a system of intelligence, then intentionally generate the very notion of symbolic procedures equivalent to itself.

Yet computation, by its very nature, cannot transcend itself in this way. A Turing machine cannot generate the abstract concept of Turing machines. A formal system cannot, from within, prove all its own truths. Likewise, a program cannot invent the very idea of computation and realize that itself is a realization of intelligence ex nihilo. And yet, humans did. We conceived the very notion of computation, invented formal systems, and constructed machines that simulate thought. This fact suggests that the mind is not reducible to computation alone. It transcends it.

One layer above computation

The fact that humans can grasp and formalize the limits of computation, proving results like the halting problem that computation itself cannot generate from inside, reveals that the human mind operates on a meta-level beyond computation. While machines can simulate patterns of thought, humans can apprehend thought as such, knowing that they are thinking. The former is a mechanical process bound by formal rules; the latter is consciousness, self-awareness, and meta-cognition.

In history, multiple influential philosophers, logicians, and mathematicians have argued that the human mind is not reducible to computation, or that there are intrinsic limits to formal procedures as models of thought and the universe. From Kurt Gödel, Alan Turing, to John Lucas and Roger Penrose who claim human insights are non-algorithmic and must involve non-computable physical processes, a strong intellectual thread claims that computation as a formal system is intrinsically limited and human thought may transcend those limits and operate at a super-computational level.

Implications for AI

The limit of computation is the limit of AI. The mind has the peculiar power to step outside formal systems, to recognize truths they cannot generate, and to invent the very idea of computation itself. Machines, bound by computation, cannot achieve this meta-level of awareness. Thus, the dream of “true human-level AGI” may be a mirage. AI can surpass humans in countless narrow domains, but when it comes to transcending formal procedures, inventing computation, or reflecting on its own nature, it faces an unbreachable boundary. This is not a problem of modeling, scale, or data, it is a mathematical ceiling. Just as no formal system can fully close over itself, AI cannot fully transcend the limits of computation that define it.

This argument should not be mistaken for the claim that AI cannot approximate the human mind. On the contrary, as we already see from the rapid development in AI over the past few years, AI systems are already approximating human intelligence with remarkable fidelity and generating immense economic value. With further algorithmic progress, these approximations will only deepen.

Yet “approximation” is not “equivalence”. Just as π can be approximated to billions of digits but never fully expressed, AI may simulate aspects of mind indefinitely without being mind. I may simulate aspects of the mind indefinitely without being mind. The ceiling it faces is not an engineering bottleneck, but a boundary imposed by mathematics and logic themselves. AI will continue to succeed as an approximation of intelligence. But the human mind transcends the system in which AI is trapped. In this sense, AI is destined to remain below the true ceiling of human intelligence.

We will generalize this topic to the broader limitations of human reasoning and scientific methods in future posts. But before that, we want to take a look at what intelligence really is, what is needed to enable intelligence, and what are the fundamental mechanisms that make it possible.

‍