On Knowledge and Substrate

I’ve recently been thinking a lot about what the intrinsic space of all human knowledge looks like, what kind of topology and structure does the neural latent manifold have, how sparse is it, and how to think about all the space in between pockets of density. For instance, it is not clear to me what the dimensionality of the original space is and whether using tokens as the basic entities of this space even makes sense. Maybe tokens are too granular to be useful for this kind of a thought experiment and we need to think about this at a higher level, say sentences and concepts. The reason such a thought experiment is appealing to me is because I think it lies at the heart of a question I’m interested in - whether AI can discover truly new knowledge.

This formulation is a bit contentious, because it is not clear what “new” even means? Is non-trivial composition of existing knowledge new? Does interpolation on the latent manifold of knowledge count? Why should it not, after all humans compose existing concepts to make new ones, and maybe there is so much exploration to do by just trudging along the latent manifold. There are many things to clarify here. For example, how do we even represent this intrinsic space? How do we find this latent manifold within the model (LLM)? Which representations within the model construct this space? And then there is the fact that neural networks combine representations using a non-linear algorithm. So does the recombined output always lie on this latent manifold? How does this change with OOD input (is anything really OOD for an LLM?).

This is related to the discussion around interpolation vs extrapolation. Yann LeCun kept reiterating a few years ago (still does), that in high dimensions, there is no interpolation, only extrapolation. The paper Learning in High Dimension Always Amounts to Extrapolation by Randall Balestriero et al. with Yann talks about this (though this is less of a paper and more of what seems like an analysis). However, like people have pointed out, talking about the convex hull of the original data and whether new samples fall within this convex hull is probably not the best way to define the behaviour that we want to study. Their results about new samples being outside the convex hull even in the neural representation space is not thoroughly discussed and even contradicted in some other studies (Interpolation, extrapolation, and local generalization in common neural networks). Moreover, this line of questioning doesn’t really address the aforementioned points about the nature of the latent manifold (not just a randomly chosen representation space from a randomly chosen layer). There definitely needs to be more discussion about this (some here and here).

I think it’s reasonable to state that traversing the intrinsic space of human knowledge can lead to “new” knowledge and maybe LLMs, by recombining representations from a space that resembles this are traversing it, in fact populating it. Every LLM generation that is verified and fed back as training data, creates a better version of this intrinsic space. If the intrinsic space can be continuously expanded or filled in even for a narrow domain say medical knowledge (by extension for every narrow domain), this would be a very interesting outcome. The question of where do new LLM generations lie on this space is still a relevant question and so is how is this space being explored by the LLM in its generations (density, pockets, voids, etc.)?. Maybe the question of how interesting the new knowledge generated by LLMs is depends on what regions of this intrinsic space its generations are exploring, how systematically is it doing so, and so on. Answering these questions first will then help us answer whether models can generate “new” enough knowledge that can help us find cures to diseases, discover new materials, prove millenium prize problems, and so on. Though, it is unlikely that some of these can be acheived without conducting experiments in the physical world. It is also entirely possible that the recombination happening inside today’s LLM’s is incapable of solving these problems, and we have to look at different symbols at different levels of abstraction.

This is a good segue to talk about substrate. There is a distinction between turing computable functions and finite-state automata computable functions. Because, today’s LLMs don’t have access to explicit memory and the orchestration to use such a memory effectively (see Titans), there is an obviously an upper-limit on what they can potentially compute. However, even if AI systems can potentially compute turing computable functions in the near future, can they compute any physically realizable computation (according to the Extended Church-Turing Thesis). This is tangentially related to the earlier discussion about human knowledge, in the sense that how much of existing human knowledge is entirely a result of our ability to interact directly with a physcially realized computation, and whether it’s possible to navigate the real world without having high bandwidth access to this computation. In some sense, robots already navigate the real world (think Boston Dynamics, Unitree, FSD etc.), but is there a upper-limit to how good they can get? There maybe fundamental limits like channel capacity constraints (any performance measure that depends on knowing more about the environment is fundamentally capped by how many bits can flow in), finite computational resources (fixed complexity class), measure-theoretic bounds (resolution of internal states discriminating crucial events or states in the environment), thermodynamic constraints (throughput/memory capacity), and no free lunch (performance degradation on tasks/distributions). This question is not just related to the ability of machines to navigate the physical world but also to learn from it and create new knowledge in it.

Apart from these, there should be a consideration for “implicit computation”, how we leverage the physical world directly versus an AI needing to simulate it. Such a framework means that the simulation rate and fidelity represent a computational burden the AI faces, which the human system bypasses by being embedded within that physics. This extra burden would be a function of discretization (the volume V and time Δt), computational cost per time step (number of operations required to update the state of the N spatial elements for one time step δ), time step constraint (maximum time step δt is often limited by stability criteria), etc. This discussion about substrate may feel irrelevant to some, but if it is the case that new knowledge (for which sufficient domain data is not available) is predicated on modeling the physical world, then any AI system that seeks to achieve the former also is bounded by the latter.