hckrnws

The example given for inverting an embedding back to text doesn't help the idea that this effect is reflecting some "shared statistical model of reality": What would be the plausible whalesong mapping of "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby"?

There isn't anything core to reality about Kentucky, its Derby, the Gregorian calendar, America, horse breeds, etc. These are all cultural inventions that happen to have particular importance in global human culture because of accidents of history, and are well-attested in training sets. At best we are seeing some statistical convergence on training sets because everyone is training on the same pile and scraping the barrel for any differences.

I fail to see how that matters. You're implying that all reality is cultural, but that seems irrelevant. The same thing would apply to scientific facts, but whales not having a word for science doesn't make it not real.

If we somehow discover LLMs right after Newton discovered the theory of gravity, and then a while later Einstein discovers General Relativity, then GR would not be in the training set of the neural net. That doesn't make GR any less of a description of reality! You also can't convert General Relativity into whalesong!

But you CAN explain General Relativity in English, or in Chinese to a person in china. So the fact that we can create a mapping from the concept of General Relativity in the neural network of the brain of a human in the USA using english, to someone in china using chinese, to a ML model, is what makes it a shared statistical model of reality.

You also can't convert General Relativity to the language of "infant babble", does it make general relativity any less real?

> You're implying that all reality is cultural,

Let's look at two examples of cultural reality:

Fan death in South Korea. Where people believe that a fan running while you sleep can kill you.

The book "Pure, White and Deadly". Where we discredited the author and his findings and spent decades blaming fat, while packing on the pounds with high fructose corn syrup.

An LLM isn't going to find some intrinsic truth, that we are ignoring, in its data set. An LLM isn't going to find issues in the reproducibility / replication crisis. I have not seen one discredit a scientific paper with its own findings.

To be clear LLM's can be amazing tools, but garbage in garbage out still applies.

You are describing the state of LLMs from 2 years ago. Which basically means they were just pre-trained on the internet and then fine tuned to follow a particular instruction format. Current models still use this as a first step, but are then trained a lot using reinforcement learning, which has given them much better skills at reasoning and logic than human tainted data ever could. See how Grok 4 for example still eagerly dismisses all those right wing hoaxes, despite being massively tuned to favour right wingers by its creators carefully selecting pre-training data.

You have some sort of very confused idea of what reinforcement learning is. (Which is probably why you're being downvoted.)

I suggest you reed something like the DeepSeek R1 paper, because you and everybody else here seems to have no clue how it works (which is not surprising tbh).

> That doesn't make GR any less of a description of reality!

When you layer the concept of awareness into the mix it does alter reality for an individual or llm. Awareness creates interesting blind spots into our statistical models of reality.

It doesn’t matter whether the Kentucky Derby is core to reality. The point is it is part of reality. If you want to model reality with 100% accuracy, you need to know about the Kentucky Derby. The author is arguing that models are converging on something close to the platonic ideal representation. So, a perfect model with perfect translatability would in fact be able to communicate the concept of a four legged land animal (named after a being capable of impossible feats) that attempts to be faster than other animals to win a reward for a rider on its back. Whether the platonic representation hypothesis is correct or not and whether our models will ever actually get that good are different questions.

Agreed. They aren't converging on a statistical model of reality, they are converging on a statistical model of their training data. In the case of LLMs and the size of the training data it's possible they are also converging on some commonality between all text. I doubt this reveals a core truth but maybe it will give us some insight into what we all agree certain chunks of text represent (when I use this idiom, everyone understand I mean this).

You also can't translate "Mage (foaled April 18, 2020) is an American Thoroughbred racehorse who won the 2023 Kentucky Derby" into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense, or allow them to interrogate an LLM to act as the glossary.

I'd say our current largest LLMs probably contain sufficient detail to explain a concept like a named race horse starting from QCD+gravity and ending up at cultural human events, given a foothold of some common ground to translate into a new unknown language. In a sense, that's what a model of reality is. I think it's possible because LLMs figure out translation between human languages by default with enough pretraining.

Your point holds, but the example of Hellenistic Greek seems ill-chosen to make that case - they had horse races and calendars, and mythological mage equivalents that would be reasonable to name a horse after, so the only thing left to map is 'American' as a geographic proper name and 'an important race in America' - which is about as translatable as it gets. Maybe if we pick one of the many cultures that never had horses, their translation would have to throw in so much context that the corresponding text is structurally different?

The LLM almost caught a 42 slork babelfish.

Man"splain a concept like...given a foothold of some common ground to translate into a new unknown language. In a sense, that's what a model of reality is" not. It is a story with motivated characters.

Why QCD? Quantum chromodynamics, the quantized theory of the nuclear strong force? There is also QED, quantum electrodynamics, which is the quantized field theory for electrodynamics, and then also QFD (quantum flavordynamics) for the weak force. Do you seriously mean to imply that the quantum field theory corresponding to ONLY the strong force, plus gravity, explains every emergent phenomena from there to culture? Fully half of the fundamental forces we account for, in two disparate theoretical frameworks?

"I mean, what are we, to believe that this is some sort of a, a magic xylophone or something? Boy, I really hope somebody got fired for that blunder"[1]

Just replace QCD with "known/understood quantum theories" and move on with your life. Thats not the important part of the comment you're replying to.

[1]: https://youtu.be/pYrRqMHQY7o

No, it actually is kind of important. Because he obviously does not understand what he's talking about, nor does he apparently have a good mental model of the standard model and its relation to prior work. He is just using buzzwords and leaning on that for sloppily gesturing at his imagination saying "look at all of our treasures, we can already solve all the most fundamental issues! We already have all of the pieces!"

Forgive me if I insist somebody show the most minute amount of competence before entertaining their absolutely wild speculation regarding whether the corpus of our species can explain physics-to-culture.

I don't mean expressing human culture in purely mathematical terms, which sounds intractable.

I mean expressing the relationships and abstractions between the different levels at which we model the world. If you need to explain horses to whales, you probably need to drop down to a biological level to at least explain keratin for hooves to e.g. the baleen whales. Other than that, common experiences of mammals probably suffices to explain social and mating differences (assuming sufficient abstractions in this hypothetical whale languages)

If you need to explain horses to aliens, you'd drop all the way down to mathematics and logic and go back up through particle physics to make sure both parties were grounded in the way we talk about objects and systems before explaining Earth biology and evolutionary history. Behavioral biology would have to be the base for introducing cultural topics and expressing any differences in where we lump behaviors in biology vs. culture, etc.

My basic claim is that if we had a pretrained model over human languages and one hypothetical alien language then either a human or an alien could learn to speak the other's language and understand the internal world model used by the other, because the amount of information contained in large LLMs covers enough of our human world model that it can translate between human languages and also answer questions about how our world-models at various levels of abstraction are related to and built upon each other via definitions.

I am less certain if that same hypothetical LLM could accurately translate between an alien language and human language; I think that the depth required to translate across potentially several layers of abstractions might not fit in the context windows and attention-span of LLMs. I think ~current LLMs will be able to accurately translate inter-species on earth if we can get enough animal language+behavior data into them.

My mistake; not being a physicist I routinely mix up which theory combines the 3 quantum forces. The standard model doesn't have a nice acronym and QFT doesn't necessarily mean standard model.

> You also can't translate .. into Hellenistic Greek or some modern indigenous languages because there isn't enough shared context; you'd need to give humans speaking those languages a glossary for any of the translation to make sense

What? By substitution, this means you can translate it. As long as we're assuming a large enough basis of concept vectors of course it works.

> I'd say our current largest LLMs probably contain sufficient detail to explain a concept like a named race horse starting from QCD+gravity and ending up at cultural human events

What? I'm curious how you'd propose to move from gravity to culture. This is like TFAs assertion that the M+B game might be as expressive as 20 questions / universal. M or B is just (bad,sentient) or (good,object). Sure, entangling a few concepts is slightly more expressive than playing with a completely flattened world of 1/0 due to some increase in dimensionality. But trying to pinpoint anything like (neutral,concept) fails because the basis set isn't fundamentally large enough. Any explanation of how this could work will have to cheat, like when TFA smuggles new information in by communicating details about distance-from-basis. For example to really get to the word or concept of "neutral" from inferred good/bad dichotomy of bread/mussolini, you would have to answer "Hmmmmmm, closer to bread I guess" in one iteration and then "Umm.. closer to Mussolini I guess" when asked again, and then have the interrogator notice the uncertainty/hesitation/contradiction and then infer neutrality. This is just the simple case.. physics to culture seems much harder

I completely believe that physics to culture is intractable given our current corpus, to the degree it's probably a nonsense claim. There are so many emergent phenomena that introduce confounding effects at every level of abstraction.

Also, why QCD? Quantum chromodynamics, the quantized theory of the nuclear strong force? There is also QED, quantum electrodynamics, which is the quantized field theory for electrodynamics, and then also QFD (quantum flavordynamics) for the weak force. Does OP seriously mean to imply that the quantum field theory corresponding to ONLY the strong force, plus gravity, explains every emergent phenomena from there to culture? Fully half of the fundamental forces we account for, in two disparate theoretical frameworks?

OP's comment is not serious speculation.

You could settle this fairly easily by training two small models on highly disparate datasets, maybe historical Chinese texts and historical greek texts, in their native languages, and seeing if the same similarities recur.

For those who don't know about Plato's Theory of Forms, the remarkable empirical fact is that all humans learn roughly the same concepts representing words like "dog", "house", "person", "boat", and so on, even if those people grew up in different places and never had any overlap between their observational experience. You and I may never have observed the same dog, but we still agree on what a "dog" is.

This phenomenon appears to occur for LLM learning as well, though it is less remarkable due to the fact that LLMs likely have significant overlap in their training data.

I believe this is good news for Alignment, because, as Plato pointed out, one of the most important forms is the Form of the Good - a (theoretical) universal human ideal containing notions of justice, virtue, compassion, excellence, etc. If this Form truly exists, and LLMs can learn it, it may be possible to train them to pursue it (or refuse requests that are opposed to it).

This vastly oversimplifies the notion of what is "good" or "fair" in a given situation. Just because we can generally agree on classifying simple objects like a boat or a house doesn't entail that we share a universal notion about the virtuous thing to do in a given situation. Humans (even in the same society/environment) have wildly differing ideas about moral issues like abortion, assisted suicide, medical testing on animals/stem cells, etc. Consider this chart from a 2010 Gallup Poll: https://content.gallup.com/origin/gallupinc/GallupSpaces/Pro...

Ahh, semantic search problem rears its head once more. While we can agree on what a "dog" is. We disagree on it's value.

> all humans learn roughly the same concepts representing words like "dog", "house", "person", "boat", and so on, even if those people grew up in different places and never had any overlap between their observational experience.

This "roughly" is doing a lot of heavy lifting to support the "Plato was right" thesis. The fact is we inhabit a shared reality with shared laws of physics and evolutionary pressures and so on; there are only so many ways to float a boat.

That doesn't necessarily mean Platonic Forms exist and that everyone arrives independently at the same Form.

Indeed you will find that people have fundamentally different ideas of what words like "freedom", "economy", "government" mean, despite using the same syntax.

Basically... Jungian archetypes

I think “we might decode whale speech or ancient languages” is a huge stretch. Context is the most important part of what makes language useful.

There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

That's not necessarily what matters.

What matters is if there is a shared representation space across languages. If there is, you can then (theoretically, there might be a PhDs and a Nobel or two to be had :) separate underlying structure and the translation from underlying structure to language.

The latter - what they call the universal embedding inverter - is likely much more easily trainable. There's a good chance that certain structures are unique enough you can map them to underlying representation, and then lever that. But even if that's not viable, you can certainly run unsupervised training on raw material, and see if that same underlying "universal" structure pops out.

There's a lot of hope and conjecture in that last paragraph, but the whole point of the article is that maybe, just maybe, you don't need context to translate.

If a lion could speak, would we understand it?

I don't know about a Lion, but I think Wittgenstein could have benefited from having a pet.

I train my cat and while I can't always understand her I think one of the most impressive features of the human mind is to be able to have such great understanding of others. We have theory of mind, joint attention, triadic awareness, and much more. My cat can understand me a bit but it's definitely asymmetric.

It's definitely not easy to understand other animals. As Wittgenstein suggests, their minds are alien to us. But we seem to be able to adapt. I'm much better at understanding my cat than my girlfriend (all the local street cats love me, and I teach many of them tricks) but I'm also nothing compared to experts I've seen.

Honestly, I think everyone studying AI could benefit by spending some more time studying animal cognition. While not like computer minds these are testable "alien minds" and can help us better understand the general nature of intelligence

Cats are domestic animals, and dogs are even more.

You probably didn't adapt to understanding cats as much as cats have adapted over millennia to be understood by humans. Working with and being understood by the dominant specie that is humans is a big evolutionary advantage.

Understanding a wild animal like a lion is a different story. There is a reason why most specialists will say that keeping wild animals as pets is a bad idea, they tend to be unpredictable, which, in other words, mean we don't understand them.

I agree with the logic, but I don't think it doesn't properly accounts for people who work very closely with wild animals.

You or I, yeah, probably not going to understand a lion pretty well. But someone who works at the zoo? A lion tamer? Someone studying lion cognition? Hell, people have figured out how to train hippos so that they can clean their teeth[0], and these are one of, if not the, most aggressive animals in the world. Humans have gotten impressively good at communicating with many different animals and training them. There's plenty of Steve Irwin types who have strong understandings about many creatures who would be quite alien to the rest of us. Which, that requires at least one side to have a strong understanding of the other's desires and how they perceive the world. But me? I have no doubt that hippo would murder me.

My point isn't so much about would we understand the lion, but rather could we. Wittgenstein implied we wouldn't be able to. I'm pointing to evidence that we, to at least some degree, can. How much we ultimately will be able to, is still unknown. But I certainly don't think it is an impossible task.

[0] https://www.youtube.com/watch?v=dMoWvh2AtKk

FWIW I don't think this is quite true for domestic cats - cross-species collaboration is very common in carnivores. A famous example is the coyote and badger, which will team up to hunt prairie dogs: the badger chases them out of the burrows, the coyote catches them outside, and they split the meat 70-30 in acknowledgement that the coyote is larger and needs more food. Recently the same has been observed with ocelots and possums. It seems like most mammals are able to at least somewhat understand the mood and intentions of other mammals - note that we share facial expressions and body language.

OTOH feral cats are known for being highly social compared to other cats, forming large semi-collaborative colonies. And adult cats have much more difficulty socializing to humans than adult dogs, even if they don't have trauma/etc. I suspect the real story of cat domestication goes both ways: an unusually gregarious subspecies of African wildcat started forming colonies near human settlements and forming cross-carnivore collaborations with the humans who lived there. This was also true for dogs - it likely started with unusually peaceful Siberian wolves - but I believe cats were more "accidental." Humans have been deliberately creating dog breeds since antiquity, but with a tiny number of exceptions cat breeds are modern. I doubt ancient humans ever "bred" cats like they did dogs, it seems closer to natural selection.

Fwiw, a lot of cat breeds are very old. Especially the ones coming out of northeast Africa (like Egypt).

But yes, both accidental domestication happens as well as non human cross species collaboration. Another famous example is with the cleaner fish and sharks. Animals also frequently collaborate with plants. Ants even have farms, both fungi and other insects

I think the response is generally you are communicating with your cat as an animal, as a mammal. Yes, communication is possible because we too are mammals, animals, etc.

But Lion is not just animal, it is not just mammal, it is something more. Something which I have no idea how we would communicate with.

  > But Lion is not just animal, it is not just mammal, it is something more.

No. I'm saying the areas you point out that you feel you could communicate, are ones in which you share a lived experience with the lion. You are both animals, you are both mammals. You get cold, hungry, thirsty, etc.

But lions, and us, are not just animals + mammals. Being a lion or a human means more. Ultimately, there is a uniquely human or lion element. Wittgenstein is saying we cannot communicate this.

Yes, I understand that. But obviously I don't buy it. I don't believe our communication is limited by our experiences

I'm much better at understanding my cat than my girlfriend

Huh. Apparently attention isn't all we need in order to parse that sentence.

It has multiple readings out of context, but I think in context one is much clearer than the other. Language is heavily overloaded, which is ironically directly relevant to the conversation.

Also, your comment still made me laugh. Women can be mysterious...

That was a philosophical position on the difficulty of understanding alien concepts and language, not a hard technological limit.

I'm missing why that distinction matters given the thread of conversation.

Would you care to expound?

I've always understood this as less of a hard wall and more of a gradient where:

- the % of shared experience/context with a mammal is > than % shared experience with a mollusk

- the gradient starts in communicating with other humans

and that Wittgenstein wasn't wrong in trying to use technology/science to bridge the context gap, he was just early.

There is nothing really special about speech as a form of communication. All animals communicate with each other and with other animals. Informational density and, uhhhhh, cyclomatic complexity might be different between speech and a dance or a grunt or whatever.

I was referencing Wittgenstein's "If a lion could speak, we would not understand it." Wittgenstein believed (and I am strongly inclined to agree with him) that our ability to convey meaning through communication was intrinsically tied to (or, rather, sprang forth from) our physical, lived experiences.

Thus, to your point, assuming communication, because "there's nothing really special about speech", does that mean we would be able to understand a lion, if the lion could speak? Wittgenstein would say probably not. At least not initially and not until we had built shared lived experiences.

If we had a sufficiently large corpus of lion-speech we could build an LLM (Lion Language Model) that would “understand” as well as any model could.

Which isn’t saying much, it still couldn’t explain Lion Language to us, it could just generate statistically plausible examples or recognize examples.

To translate Lion speech you’d need to train a transformer on a parallel corpus of Lion to English, the existence of which would require that you already understand Lion.

Hmm I don't think we'd need a rosetta stone. In the same way LLMs associate via purely contextual usage the meaning of words, two separate data sets of lion and English, encoded into the same vector space, might pick up patterns of contextual usage at a high enough level to allow for mapping between the two languages.

For example, given thousands of English sentences with the word "sun", the vector embedding encodes the meaning. Assuming the lion word for "sun" is used in much the same context (near lion words for "hot", "heat", etc), it would likely end up in a similar spot near the English word for sun. And because of our shared context living in earth/being animals, I reckon many words likely will be used in similar contexts.

That's my guess though, note I don't know a ton about the internals of LLMs.

Someone more knowledgeable might chime in, but I don't think two corpuses can be mapped to the same vector space. Wouldn't each vector space be derived from its corpus?

It depends how you define the vector space but I'm inclined to agree.

The reason I think this is from evidence in human language. Spend time with any translator and they'll tell you that some things just don't really translate. The main concepts might, but there's subtleties and nuances that really change the feel. You probably notice this with friends who have a different native language than you.

Even same language same language communication is noisy. You even misunderstand your friends and partners, right? The people who have the greatest chance of understanding you. It's because the words you say don't convey all the things in your head. It's heavily compressed. Then the listener has to decompress from those lossy words. I mean you can go to any Internet forum and see this in action. That there's more than one way to interpret anything. Seems most internet fights start this way. So it's good to remember that there isn't an objective communication. We improperly encode as well as improperly decode. It's on us to try to find out what the speaker means, which may be very different from the words they say (take any story or song to see the more extreme versions of this. This feature is heavily used in art)

Really, that comes down to the idea of universal language[0]. I'm not a linguist (I'm an AI researcher), but my understanding is most people don't believe it exists and I buy the arguments. Hard to decouple due to shared origins and experiences.

[0] https://en.wikipedia.org/wiki/Universal_language

Hmm I don't think a universal language is implied by being able to translate without a rosetta stone. I agree, I don't think there is such a thing as a universal language, per se, but I do wonder if there is a notion of a universal language at a certain level of abstraction.

But I think those ambiguous cases can still be understood/defined. You can describe how this one word in lion doesn't neatly map to a single word in English, and is used like a few different ways. Some of which we might not have a word for in English, in which case we would likely adopt the lion word.

Although note I do think I was wrong about embedding a multilingual corpus into a single space. The example I was thinking of was word2vec, and that appears to only work with one language. Although I did find some papers showing that you can unsupervised align between the two spaces, but don't know how successful that is, or how that would treat these ambiguous cases.

  > I don't think a universal language is implied by being able to translate without a rosetta stone.

Another way you can think about it, though, is that language might not be dependent on experience. If it is completely divorced, we may be able to understand anyone regardless of experience. If it is mixed, then results can be mixed.

  > The example I was thinking of was word2vec

That's a very good point! I hadn't thought of that. And that makes sense, since the encoding of the word "sun" arises from its linguistic context, and there's no such shared context between the English word sun and any lion word in this imaginary multilingual corpus, so I don't think they'd go to the same point.

Apparently one thing you could do is train a word2vec on each corpus and then align them based on proximity/distances. Apparently this is called "unsupervised" alignment and there's a tool by Facebook called MUSE to do it. (TIL, Thanks ChatGPT!) https://github.com/facebookresearch/MUSE?tab=readme-ov-file

Although I wonder if there are better embedding approaches now as well. Word2Vec is what I've played around with from a few years ago, I'm sure it's ancient now!

Edit: that's what I get for posting before finishing the article! The whole point of their researh is to try to build such a mapping, ve2vec!

And even, assuming the existence of a Lion to English corpus, it would only give us Human word approximations. We experience how lossy that type of translation is already between Human->Human languages. Or sometimes between dialects within the same language.

Who knows, we don't really have good insight into how this information loss, or disparity grows. Is it linear? exponential? Presumably there is a threshold beyond which we simply have no ability to translate while retaining a meaningful amount of original meaning.

Would we know it when we tried to go over that threshold?

Sorry, I know I'm rambling. But it has always been regularly on my mind and it's easy for me to get on a roll. All this LLM stuff only kicked it all into overdrive.

You might find https://www.lojban.org/files/why-lojban/whylojb.txt interesting. It is not really about your quote of Wittgenstein, but there is:

> In broad terms, the Hypothesis claims that the limits of the language one speaks are the limits of the world one inhabits (also in Wittgenstein), that the grammatical categories of that language define the ontological categories of the word, and that combinatory potentials of that language delimit the complexity of that world (this may be Jim Brown's addition to the complex Hypothesis.) The test then is to see what changes happen in these areas when a person learns a language with a new structure, are they broadened in ways that correspond to the ways the structure of the new language differs from that of the old?

That seems a bit extreme, given that a lion also has a mammal brain. I'd expect it to also think in terms of distinct entities that can move around in the environment and possibly talk about things like "hunger" and "prey".

I'd expect incomprehensible language from something that is wildly different from us, e.g. sentient space crystals that eat radiation.

On the other hand, we still haven't figured out dolphin language (the most interesting guess was that they shout 3D images at each other).

Observation has proven enough to understand the meaning of animal calls. People proved they correctly identified, for example, an distressed animal call for assistance, by playing it to their peers in the wild. They go look for the distressed animal. Other calls don't provoke the same reaction.

Analogies are always possible. I believe in the philosophical context though, understanding the meaning of something is not possible through analogy alone.

Reminds me of the quote:

“But people have an unfortunate habit of assuming they understand the reality just because they understood the analogy. You dumb down brain surgery enough for a preschooler to think he understands it, the little tyke’s liable to grab a microwave scalpel and start cutting when no one’s looking.”

― Peter Watts, Echopraxia

There was no analogy in there.

Correlate the vocalizations with the subsequent behavior. I believe this has been done for some species in certain situations.

Its also pretty much how humans acquire language. No one is born knowing English or Spanish or Mandarin.

Hmm I'm not convinced we don't have a lot of shared experience. We live on the same planet. We both hunger, eat, and drink. We see the sun, the grass, the sky. We both have muscles that stretch and compress. We both sleep and yawn.

I mean who knows, maybe their perception of these shared experiences would be different enough to make communication difficult, but still, I think it's undeniably shared experience.

That's fair. To me, the point of Wittgenstein's lion thought experiment though was not necessarily to say that _any_ communication would be impossible. But to understand what it truly meant to be a lion, not just what it meant to be an animal. But we have no shared lion experiences nor does a lion have human experiences. So would we be able to have a human to lion communication even if we could both speak human speech?

I think that's the core question being asked and that's the one I have a hard time seeing how it'd work.

Hmm, I'm finding the premise a bit confusing, "understand what it truly meant to be a lion". I think that's quite different than having meaningful communication. One could make the same argument for "truly understanding" what it means to be someone else.

My thinking is that if something is capable of human-style speech, then we'd be able to communicate with them. We'd be able to talk about our shared experiences of the planet, and, if we're capable of human-style speech, likely also talk about more abstract concepts of what it means to be a human or lion. And potentially create new words for concepts that don't exist in each language.

I think the fact that human speech is capable of abstract concepts, not just concrete concepts, means that shared experience isn't necessary to have meaningful communication? It's a bit handwavy, depends a bit on how we're defining "understand" and "communicate".

> I think the fact that human speech is capable of abstract concepts, not just concrete concepts, means that shared experience isn't necessary to have meaningful communication?

I don't follow that line of reasoning. To me, in that example, you're still communicating with a human, who regardless of culture, or geographic location, still shares an immense amount of shared life experiences with you.

Or, they're not. For example, an intentionally extreme example, I bet we'd have a super hard time talking about homotopy type theory with a member of the amazon rain forest. Similarly, I'd bet they had their own abstract concepts that they would not be able to easily explain to us.

I would say there's a difference between abstract and complex. A complex topic would take a lot of time to communicate mainly because you have to go through all the prerequisites. By abstract I mean something like "communicate" or "loss" or "zero"! The primitives of complex thought.

And if we're saying the lion can speak human, then I think it follows that they're capable of this abstract thought, which is what I think is making the premise confusing for me. Maybe if I change my thinking and let's just say the lion is speaking... But if they're speaking a "language" that's capable of communicating concrete and abstract concepts, then that's a human-style language! And because we share many concrete concepts in our shared life experience, I think we would be able to communicate concrete concepts, and then use those as proxies to communicate abstract concepts and hence all concepts?

Does a lion not know what it's like to be hungry? These parts of the brain are ancient. There is clearly a sliding scale in most experiences here from amoeba to fly to lion to human. Would you like to communicate with a girl who drinks tapioca milk tea? Clearly your life experiences are different so what's the point? Obviously gets harder, that's why we are discussing the possibility of using technology to make it easier.

Obviously it's impossible to communicate even 90% of human experience with lions or people with mental disabilities. But if a translation model increases communication even 1%, brings everybody up to the level of a Kevin Richardson it's a huge win E.g. A pair of smart glasses that labeling the mood of the cat. Nobody cares about explaining why humans wear hats to a lion and of course no explanation is better than being a old human who has worn hats for a variety of reasons.

I think it's unlikely you could make a LLM that gives a lion knowledge via audio only, but very possibly other animals

We should understand common concepts like hungry, tired, horny, pain, etc.

Not just the concepts themselves, but their behavioral representation of it.

Knowing lions I bet all they’d talk about is being straight up dicks to anyone and everyone around them so yea I think we probably could ngl

>There is billions of human-written texts, grounded in shared experience that makes our AI good at language. We don't have that for a whale.

the world around us is a very large part of that shared experience. It is shared among humans, shared among whales, and shared among whales and humans as well.

If we could help gorillas or elephants (both highly intelligent) learn to name things and use symbols — in a form they can comprehend and create to express their will — enabling them to pass down their experiences and wisdom across generations, I believe they could quietly be as smart as we are.

Ps. I am excited about Google’s Gemma dolphin project (https://blog.google/technology/ai/dolphingemma/), but I would prefer if they chose elephants instead of dolphins as the subject, since we live on land, not in water. This way, more emphasis could be placed on core research, and immediate communication feedback would be possible.

I think it would be extremely surprising for Universal Grammar to be proven false. We have not had any shortage of opportunity to teach animals structured language, and yet we have nothing to show for it. It seems pretty likely that a key factor in our fitness as a species was having the right hardware for grammar.

> I think it would be extremely surprising for Universal Grammar to be proven false.

All you need to make it unsurprising you need to find an alternative hypothesis, that can explain observed facts.

How about this one. Humans got the ability to abstract deeper than others. While some animals could deal with concepts closely associated with real phenomena, and therefore they can use words to name things or maybe even actions, they cannot go further and use abstractions over abstractions. As the result they cannot deal with a recursive grammar.

Or another hypothesis. Humans got an ability to reflect on themselves, so they came to concepts like "motive (of action)", it made it possible for them to talk about actions, inventing words-categories, that categorize actions by supposed motive of action. But it is not just that, people started to feel that simple words do not describe the reality good enough, because a) actions have more than one motive, b) they started to see actions everywhere (the stone is lying flat? it is the action of the stone, that strictly speaking cannot act.). All this led them to invent complex grammars to describe the world where they live. Animals from other hand can't get it, because their world is much simplier. They don't see the stone "lying": it is a stone, it cannot do anything.

These hypotheses don't seem explaining everything, but I have just invented them. I didn't really tried to explain everything. But thinking about it, I'm in favor of the second one: all people are schizophrenics if you compare them with animals. They believe in things that are not exist, they see inanimate objects as animate, they are talking with themselves (inner dialog). It just happens that the particular kind of schizophrenia most people are struggling with is a condition that allows them to not tear apart all the connections with the Reality. It is more like borderline schizophrenia.

> While some animals could deal with concepts closely associated with real phenomena, and therefore they can use words to name things or maybe even actions, they cannot go further and use abstractions over abstractions. As the result they cannot deal with a recursive grammar.

I prefer to think that animals simply haven’t been lucky enough to invent a suitable naming system that could serve as the foundation of their civilization. If they were fortunate, they might bootstrap their own form of civilization through recursive or iterative divisive naming-—naming the act of naming itself. Naming is the foundation, and everything else naturally follows from there.

As Laozi said over 2,500 years ago in the Dao De Jing:

  “The Tao that can be spoken is not the eternal Tao.
   The name that can be named is not the eternal name.
   The nameless is the origin of Heaven and Earth;
   The named is the mother of all things.”

P.S. By the way, the entire Dao De Jing advocates thinking beyond symbols, transcending naming—not being constrained by it—and thus connect the reality and following the Tao, which embodies the will of love, frugality, and humility. Yet, few truly understand it fairly, as Laozi emphasized in his book. Perhaps we humans can transcend naming altogether eventually, with advancements like Musk’s brain-communication chip, we might soon discover how human wills are encoded and potentially move beyond the limitations of naming.

“Universal grammar” is a placeholder for the innate, domain-specific features of linguistic ability...

--— After looking up “Universal Grammar,” here are my thoughts: --—

Until we identify these features and prove that other species lack them, the concept of Universal Grammar remains uncertain. Only if we pinpoint what’s unique to humans and fail to help other species develop it can we confidently claim Universal Grammar is correct.

Personally, I think these innate features might not be entirely unique to humans. Even humans need sufficient social interaction and language exposure to truly behave like 'humans'. Feral children(https://en.wikipedia.org/wiki/Feral_child), for example, behave much like animals. I also imagine early humans had only a few words and symbols and acted similarly. However, if feral children or ancient humans were exposed to today’s social environment, they could likely learn our complex language quite quickly, because those words, symbols, and pronunciations are well-suited to humans for expressing our wills.

Perhaps, we humans are just lucky to have found effective ways to express ourselves —- symbols and words as we know them now -— and to pass them down through generations. This has allowed language and cognition to grow exponentially, giving us an edge over other species.

P.S. Our failure so far to teach animals structured language is exactly why I admire projects like Google’s Dolphin Gemma. To succeed, we need to tailor teaching methods to their species-specific traits—starting by understanding how they express their intentions/wills, and then hopefully helping enrich their naming systems and overall communication. We might even adapt this to help them learn from human contexts, much like large language models do.

Other animals have vocabulary. Anybody with a pet can easily understand that lots of animals can associate words with objects and actions, and Koko and Alex and all the other examples of talking animals demonstrate that animals can communicate their own wants with words.

The thing that makes humans nearly unique in the animal kingdom is structured language. I don't need separate words for "leopard in the bush", "snake in the bush", "leopard in the tree", and "snake in the tree", because we have grammar rules that allow me to combine the words for "leopard" and "tree" with a distinction of how the leopard relates to the tree.

I say nearly unique, because there is some evidence that whale songs are structured.

> The thing that makes humans nearly unique in the animal kingdom is structured language.

But what is “structured language”? I would argue it’s just another layer of naming—naming applied to already named names.

If animals have the ability to map leopard, snake, bush, tree in reality to names in their naming system, then why can’t they apply that ability again—create a new name like ‘leopard-bush’ or ‘snake-leopard’, simply by combining the existing solid names in their system? (Similarly, words like “in” could also be added.)

You might say they can’t—because they lack grammar rules. And I’d argue the issue runs even deeper: they don’t have a valid naming system to begin with, nor the ecological fitness or incentive needed to develop further naming.

If they can name ‘leopard’ within a valid system, that name should be reusable. They should be able to work on those names—as objects themselves. The process is the same. It requires the same abstraction: treat a thing as a nameable unit, then name it again. And that already starts to look like compositionality—or grammar.

The real difference is that humans ended up with a robust naming system. We got lucky—we gradually invented stable symbols that let us name on top of previous names. We could write them down, save them, pass them on—and then build new names on top of that naming recursively, iteratively, and divisively.

And that’s likely where the fitness gain came from. It’s a lucky, genuine self-bootstrap story.

Making compound words is not an important difference between human language and the languages of other animals. Even in human languages the ability to make compound words seems to have appeared later.

The main difference seems to be in having sentences structured into verbs and their associated nouns (which may encode various roles, e.g. patients, agents, instruments, beneficiaries, subjects of intransitive or copulative verbs, nominal predicates, results etc.).

As far as we could determine until now, the language of most animals consists only of a set of nouns, which typically have an associated implicit action.

For example:

"Lion!" (meaning "Climb up the tree!");

"Eagle!" (meaning "Take cover under branches!");

"Children!" (meaning "Come here, to mother!");

"Bananas!" (meaning "Come here to eat good food!");

and so on.

Some apes have been trained to compose very simple sentences, as complex as agent-verb-patient, but it is not known whether a similar form of communication exists between wild apes.

Darmok and Jalad at Tenegra.

> having the right hardware for grammar

Do we have hardware for playing Super Mario or dancing to pop music? Animals can't do that either.

They tried that with Koko, it didn't work. It did not make the gorilla any smarter, it was an animal, and remained an animal.

If you read the criticism about the Koko project, it's that Patterson prompted her to make certain signs. I watched some clips of the signs, and it's very obvious that that is exactly what she did.

Animals can only communicate in a limited fashion. They do not have some "hidden" intelligence that you are trying to find.

> They tried that with Koko, it didn't work. It did not make the gorilla any smarter, it was an animal, and remained an animal.

I really doubt both their methods and your conclusion. The project tried to teach animals to adapt to the human naming system, and more importantly, it wasn’t a true social-level experiment with animals.

(That said, in my opinion, Koko was smart. Critics of Patterson’s claims have acknowledged that Koko learned a number of signs and used them to communicate her wants and needs.)

Imagine an alien taking an one- or two-year-old human baby and trying to teach him to communicate with alien using vocal signals that humans have difficulty producing or perceiving—and, most importantly, where naming is not a social-level communication that boosts ecological fitness. That’s basically what they did with Koko. It’s like training an LLM model on text corpora where the loss function (fitness measure) is inconsistent or even meaningless—would that produce a useful model?

We first need to understand how animals naturally name things, and then enrich that naming system in a way that fits their minds—not force them to learn human naming. Most importantly, this naming should improve their ecological fitness in a consistent way, so that they can ‘feel’ the fitness of certain naming and evolve their minds toward it.

That’s why Google DolphinGemma is so remarkable. If they succeed, they might mimic dolphin-like communication in a way other dolphins truly understand—gradually introducing naming and concepts that improve evolutionary fitness: finding food, recognizing others, being happy, and identifying suitable mates. If recursive naming develops, I believe it could even lead to real cognitive evolution.

This is just a rough idea, but I truly believe in it-—based on self-meta-cognition about how my own mind evolved and works, as well as other observations. Unfortunately, I don’t have the power or energy to explore it deeply, but I hope someone working on animal cognition research will take a closer look at using large language models like DolphinGemma and dive deeper.

I don't know why, but I just had a horrible vision of traveling 200 years hence, and elephants are now the ruling class.

I don't care that it might be better, and I shall hold you personally responsible should this come to pass.

This only works when different sources share similar feature distributions and semantic relationships.

The M or B game breaks down when you play with someone who knows obscure people you've never heard of. Either you can't recognize their references, or your sense of "semantic distance" differs from theirs. The solution is to match knowledge levels: experts play with experts, generalists with generalists.

The same applies to decoding ancient texts, if ancient civilizations focused on completely different concepts than we do today, our modern semantic models won't help us understand their writing.

I've played this game with friends occasionally and -- when it's a person -- don't think I've ever completed a game.

This is kind of fascinating because I just tried to play mussolini or bread with chatgpt and it is absolutely _awful_ at it, even with reasoning models.

It just assumes that your answers are going to be reasonably bread-like or reasonably mussolini-like, and doesn't think laterally at all.

It just kept asking me about varieties of baked goods.

edit: It did much better after I added some extra explanation -- that it could be anything that it may be very unlike either choice, and not to try and narrow down too quickly

I think an LLM is a bit too high level for this game or maybe it just would need a lengthy prompt to explain the game.

If you used word2vec directly it's the exact right thing to play this game with. Those embeddings exist in an LLM but it's trained to respond like text found online not play this game.

what surprised me when I asked if it knew what Mussolini or Bread was, it parroted this article. This article was posted in the last 24 hours and querying a search engine for "Mussolini or Bread" didn't yield it anywhere near a top result.

searching for '"Mussolini or Bread"' (with quotes), or searching for 'Mussolini or Bread game' (without quotes) both return this article at the top.

Idiot me forgot to try the quotes. Thank you.

Is it closer to Mussolini or bread? Mussolini.

Is it closer to Mussolini or David Beckham? Uhh, I guess Mussolini. (Ok, they’re definitely thinking of a person.)

That reasoning doesn't follow. Many things besides people would have the same answers, for instance any animal that seems more like Mussolini than Beckham.

I believe the joke is about David Beckham not really being (perceived as) human, even when compared to personified evil

My thought exactly. My word would be "Gun" or "Artillery". easily defeats his logic.

But that should also explain us why pure embeddings search is not sufficient for RAG.

I think the point is that, despite the game being massively logically underspecified, people are still able to home in on an answer in practice. Which shows that they have a shared fuzzy semantic space.

Whoops. I hope you can overlook this minor logical error.

Oh yeah it's absolutely an interesting article!

Oswald Mosley

I agree LLMs are converging on a current representation of reality based on the collective works of humanity. What we need to do is provide AIs with realtime sensory input, simulated hormones each with their own half-lifes based on metabolic conditions and energy usage, a constant thinking loop, and discover a synthetic psilocybin that's capable of causing creative, cross-neural connections similar to human brains. We have the stoned ape theory, we need the stoned AI theory.

Or perhaps we make them attractions at a theme park, but let Anthony Hopkins have admin access to the source code. What could go wrong?

I’m so incredibly tired of reading about AI but if I saw “we gave an AI shrooms” I’d be all over it

I imagine the temperature parameter is the closest thing we have to that right now. Turn it up a little to make things a bit more creative. Turn it up more and eventually it will wildly swing around in its dialogue. Even more and it eventually produces utter gibberish.

When I read that "Ilya gave a famously incomprehensible talk about the connections between intelligence and compression" it makes me wonder if Marcus Hutter has now been forgotten? If so more people should take a look at http://prize.hutter1.net/

The Dao can be spoken of, yet what is spoken is not the eternal Dao.

So, what is the Dao? Personally, I see it as will — something we humans could express through words. For any given will, even though we use different words in different languages — Chinese, Japanese, English — these are simply different representations of the same will.

Large language models learn from word tokens and begin to grasp these wills — and in doing so, they become the Dao.

In that sense, I agree: “All AI models might be the same.”

> One explanation for why this game works is that there is only one way in which things are related, and this comes from the underlying world we live in.

I think what this might be trying to say is something more like: ...there are many ways in which things can be related, but those relationships come from the underlying world we live in.

I.e. there are obviously many ways in which things can be related, but if we assume the quote is not entirely counterfactual, then it must be getting at something else. I suppose "way" here is being used in a different sense, but it isn't clear.

I have to be careful of confirmation bias when I read stuff like this because I have the intuition that we are uncovering a single intelligence with each of the different LLMs. I even feel, when switching between the big three (OpenAI, Google, Anthropic) that there is a lot of similarity in how they speak and think - but I am aware of my bias so I try not to let it cloud my judgement.

On the topic of compression, I am reminded of an anecdote about Heidegger. Apparently he had a bias towards German and Greek, claiming that these languages were the only suitable forms for philosophy. His claim was based on the "puns" in language, or homonyms. He had some intuition that deep truths about reality were hidden in these over-loaded words, and that the particular puns in German and Greek were essential to understand the most fundamental philosophical ideas. This feels similar to the idea of shared embeddings being a critical aspect of LLM emergent intelligence.

This "superposition" of meaning in representation space again aligns with my intuitions. I'm glad there are people seriously studying this.

LLMs don't think, nor are they intelligent or exhibiting intelligence.

Language does have constraints, yet it evolves via its users to encompass new meanings.

Thus those constraints are artificial, unless you artificially enforce static language use. And of course, for an LLM to use those new concepts, it needs to be retokenized by being trained on new data.

For example, if we trained LLMs only on books, encyclopedias, newpapers, and personal letters from 1850, it would have zero capacity to speak comprehensibly or even seem cogent on much of the modern world.

And it would forever remain in that disconnected positon.

LLMs do not think, understand anything, nor learn. Should you wish to call tokenization, learning, then you'd better call a clock "learning" from the gears and cogs that enable its function.

LLMs do not think, learn, or exhibit intelligence. (I feel this is not said enough).

We will never, ever get AGI from an LLM. Ever.

I am sympathetic to the wonder of LLMs. To seeing them as such. But I see some art as wonderous too. Some machinery is beautiful in execution and to use.

But that doesn't change truths.

You made a lot of bold assertions there. It's as if you have a complete and definitive theory of human intelligence to compare it against. Which if true, would be incredible, because there isn't a scientifically accepted theory, nor is there consensus from a philosophical standpoint.

I can't say that you are wrong, you might be right, especially about AGI. And I think it's unlikely that LLMs are the direct path to AGI. But, just looking at how human brains work, it seems unlikely that we would be intelligent either if we used your same reductionist logic.

An individual neuron doesn't "think" or "understand" anything. It's a biological cell that simply fires an electrochemical signal when its input threshold is met. It has no understanding of language or context. By your logic, since the fundamental components are just simple signal processors, the brain cannot possibly learn or be intelligent. Yet, from the complex interaction of ~86 billion of these simple biochemical machines, the emergent properties of thought, understanding, and consciousness arise.

Dismissing an LLM's capabilities because its underlying operations are basically just math operating on tokenized data is like dismissing human consciousness because it's "just electrochemistry" in a network of cells. Both arguments mistake the low-level mechanism for the high-level emergent phenomenon.

I agree with much of what you say, but this:

> An individual neuron doesn't "think" or "understand" anything. It's a biological cell that simply fires an electrochemical signal when its input threshold is met.

is a wildly oversimplified representation of what a neuron is and how it interacts with the rest of the brain. Even what we know is far beyond this description and there's a vast amount more that we don't know.

One of the assumptions that people make (with or without realising they are making it) is that the brain is a componentised information processing device, that you can break it down into units that effectively communicate in a digital way. This may be true, but it may not. It may not be possible to decompose a mind into parts communicating via digital signals without sacrificing conscious experience.

Personally, I think it's likely that it isn't possible to do that, but I don't know for sure. Others clearly think it's likely that it is possible, which is a perfectly valid opinion. Neither should pretend that science has yet given us an unambiguous answer on the question.

When an LLM can re-tokenize on the fly, due to newly learned data, let me know.

It won't prove intelligence, but at least it won't be static like a book.

https://x.com/sukjun_hwang/status/1943703574908723674

> Tokenization has been the final barrier to truly end-to-end language models.

> We developed the H-Net: a hierarchical network that replaces tokenization with a dynamic chunking process directly inside the model, automatically discovering and operating over meaningful units of data

Good to see. Will be better to see once the scope is clear.

But something such as this is required to move towards actual intelligence.

This is a dumb critique. A thin wrapper running a new samples of training data and updating weights is something already done in many situations. A sophisticated rag system incorporates new information even if the weights themselves aren't updated, effectively giving "new memory".

LLMs have problems, in practice being "static" aint one of them.

I disagree with you but I'll only repeat and expand on my final statement: I'm glad there are people seriously studying this. If you are right (which I doubt) I hope it comes to light with evidence. And if you are wrong, I hope you have the grace to admit it.

Do you think an average human from 1850 transported to the modern world would fare better than an LLM?

This isn't a serious question, is it?

A human would immediately start learning, and understanding the differences. Forming new memories. People learn every, single day.

An LLM would never have a single new memory, nor understand a single change had happened. Without being retokenized via new data it trains on, nothing of its symbolic world view would even change in the tiniest bit.

Telling it so in a context window isn't the LLM learning, that's gone the second the context window is gone. In 5 years, 10 years, 1M years, that LLM would be precisely the same. It would view the world as 1850. Forever.

Meanwhile, being told "It's 2025!" would immediately update the human's expectations. They would see the changes around them, and start learning. Note that initial reaction isn't relevant here, a person would eventually deal with it, and move forward.

You have posted a link, in another response, where someone is working on dynamically updating an LLM. However that is not how current LLMs work, thus invalid in any attempt to refute how I describe LLMs currently.

Further, until we see how this works -- precisely, it may be a non-starter. It may not work as described. It may be hype. It may work, but still require retraining if large amounts of symbolic data needs to change.

There's a reason it takes massive farms of hardware to tokenize, and I'm skeptical that the quoted twitter link reforges all relationships. Again, we'll see.. and this is something required for an LLM to actually "learn" about changes to how tokens are weighted.

For long term memory, LLMs can learn new information by training on it. It would need a lot of training data, but it will get there eventually.

For short term memory, they can store information in their context window. Yes the context window is finite, but they can still be very large, and there exist techniques to fit more information within that window.

For medium term memory, there's not one single solution, but anecdotally it seems like solutions exist. My workplace has a chatbot that can ingest documents and save slack conversations, and then use the knowledge from those to answer questions indefinitely into the future. I haven't looked into how it works, but empirically it functions well across thousands of people for more than a year.

Between a human and an LLM transported 200 years into the future, a human will certainly have better critical thinking skills, but I'd bet an LLM system designed for the task will probably be able to more quickly learn to have coherent and symmetric conversations about the world with modern humans.

Overall, there's a lot that's unclear, and I think I'm pretty pessimistic about progress towards AGI, but I don't think we can deny that LLMs have something that is, at minimum, the best substitute for human intelligence that exists.

LLMs don't think, nor are they intelligent or exhibiting intelligence. ....

---

My 2$: If you replace 'LLMs' with 'Humans,' most of your statements above still make sense — which suggests AGI might be possible.

Mmmh I'm deeply skeptical of some parts.

> One explanation for why this game works is that there is only one way in which things are related

There is not, this is a completely non transitive relationship.

On another point, suppose you keep the same vocabulary, but permute the signification of the words, the neural network will still learn relationships, completely different ones, but it's representation may converge toward a better compression for that set of words, but I'm dubious that this new compression scheme will ressemble the previous one (?)

I would say that given an optimal encoding of the relationships, we can achieve an extreme compression, but not all encodings lead to the same compression at the end.

If I add 'bla' between every words in a text, that is easy to compress, but now, if I add an increasing sequence of words between each words, the meaning is still there, but the compression will not be the same, as the network will try to generate the words in-between.

(thinking out loud)

I asked Grok, o3-pro, and Claude a question about piezoelectric effects.

They all got it "right", but Claude called out a second order effect that arose from the use case that the other two missed.

I get it, they might all be exploring the same space, but Claude went an extra, important, hop.

I'm just curious if that was Grok 4 or 3

3! So it’s not a fair comparison. The other two were both paid.

I am very curious to run real world engineering problems through SuperHeavy.

Have you tried SuperHeavy yet? It seems to be crushing on almost any measurable benchmark. I've only used Grok 4 (not heavy) when stuck on a problem that claude/cursor cannot get past, and Grok 4 is often able to come up with an elegant solution.

I tried playing Mussolini or Bread with ChatGPT, but it didn't go very well. It seemed to have trouble grasping the rules, and kept getting overly specific when we were miles from the right concept.

I've noticed that many of the large, separately developed AIs often answer with remarkably similar wording to the same question.

I'm not exactly surprised that "separately developed" AI that hinge on essentially the same core techniques and are fed largely identical corpus often answer with remarkably similar wording to the same question.

At least for LLMs, the platonic representation hypothesis doesn't seem that surprising. The training data for models that are large enough to faithfully capture reality will necessarily have a lot of overlap. So in this weak form the platonic representation hypothesis just says that if you provide the roughly same data to models with roughly the same architecture, they learn roughly the same embeddings. This doesn't seem that surprising. I'd even expect a slightly stronger version where you feed the same data to different models to still hold. After all the modes capture the same reality, so there's probably some way to transform from the "latent space" of one model to the other.

I find it grating that the article is sweeping this under the rug. If you train your AI on Shakespeare it won't suddenly know how to control a robot. Meanwhile all Shakespeare bots will be remarkably similar. Coincidence?

I know it's only one tiny part, but I can't resist:

> Is it closer to Mussolini or David Beckham? Uhh, I guess Mussolini. (Ok, they’re definitely thinking of a person.)

This deduction is absurd! The only information you have at all is that it's more like a person than it is like bread (which could be almost anything).

Especially if they are all me-too copies of a Transformer.

When we arrive at AGI, you can be certain it will not contain a Transformer.

I don't think architecture matters. It seems to be more a function of the data somehow.

I once saw a LessWrong post claiming that the Platonic Representation Hypothesis doesn't hold when you only embed random noise, as opposed to natural images: http://lesswrong.com/posts/Su2pg7iwBM55yjQdt/exploring-the-p...

Its a bit like saying algorithms don't matter for solving computational problems. Two different algorithms might produce equivalent results but if you have to wait years for an answer, when seconds matter, the slow algorithm isn't helpful.

I believe the current approach of using mostly a feed-forward in the inference stage, with well-filtered training data and backpropagation for discrete "training cycles" has limitations. I know this has been tried in the past, but something modelling how animal brains actually function, with continuous feedback, no explicit "training" (we're always being trained), might be the key.

Unfortunately our knowledge of "whats really going on" in the brains is still limited, investigative methods are crude as the brain is difficult to image at the resolution we need, and in real time. Last I checked no one's quite figured out how memory works, for example. Whether its "stored in the network" somehow through feedback (like a SR-latch or flip-flop in electronics) or whether there's some underlying chemical process within the neuron itself (we know that chemicals definitely regulate brain function, don't know how much it goes the other way and it can be used to encode state)

> I don't think architecture matters. It seems to be more a function of the data somehow.

of course it matters

if I supply the ants in my garden with instructions on how to build tanks and stealth bombers they're still not going to be able to conquer my front room

So in the limit the models representation space has one dimension per "concept" or something, but making it couple things together is what actually makes it useful?

An infinite dimensional model with just one dim per concept would be sorta useless, but you need things tied together?

Really cool idea that I hadn't considered yet. If true, seems like a big plus for open source and not having a few companies controll all the models. If they all converge to the same intelligence, one open source model would make all proprietary models obsolete.

I think some of this effect is explained by different models being trained on almost identical corpa. Even if you try to erase certain works, contamination will still represent some of it, just in less obvious ways.

What about the platonic bits? Any other articles that give more details there?

It refers to Plato's Theory of Forms: https://en.wikipedia.org/wiki/Theory_of_forms

Consider how every chair you've seen is different. Yet they are all chairs. What is 'chair-ness'? The Forms theory is that there is a single ideal chair, and this truly exists - not materially, but non-materially as a concept or way of understanding chairs, but real. That is, the 'essence' of a chair exists. All chairs are imperfect representations of the essence of a chair, being perfect chairs or representing the essence of a chair to a greater or lesser degree.

You might find Neal Stephenson's bool Anathem a wonderful read. It's one of my favorites. I want to tell you why without spoilers but I can't (it won't be what you expect) -- but if you read it perhaps you'll find it interesting re this topic.

Why is claude shannon closer to mussolini than to david beckham?

Has there been research on using this to make models smaller? If models converge on similar representations, we should be able to build more efficient architectures around those core features.

It's more likely that such an architecture would be bigger rather than smaller. https://arxiv.org/abs/2412.20292 demonstrated that score-matching diffusion models approximate a process that combines patches from different training images. To build a model that makes use of this fact, all you need to do is look up the right patch in the training data. Of course a model the size of its training data would typically be rather unwieldy to use. If you want something smaller, we're back to approximations created by training the old-fashioned way.

I have mixed feelings about this interpretation: that diffusion models approximately produce moseics from patches of training data. It does a good job helping people understand why diffusion models are able to work. I used it myself in talk almost 3 years ago! And it isn't a lie exactly, the linked paper is totally sound. It's just that it only works if you assume your model is an absolute optimal minimization of the loss (under some inductive biases). It isn't. No machine learning more complicated than OLS holds up to that standard.

_And that's the actual reason they work._ Undefit models don't just approximate, they interpolate, extrapolate, generalize a bit, and ideally smooth out the occasional total garbage mixed in with your data. In fact, diffusion models work so well because they can correct their own garbage! If extra fingers start to show up in step 5, then steps 6 and 7 still have a chance to reinterpret that as noise and correct back into distribution.

And then there's all the stuff you can do with diffusion models. In my research I hack into the model and use it to decompose images into the surface material properties and lighting! That doesn't make much sense as averaging of memorized patches.

Given all that, it is a very useful interpretation. But I wouldn't take it too literally.

> I used it myself in talk almost 3 years ago! And it isn't a lie exactly, the linked paper is totally sound.

The paper was published in December last year and addresses your concerns head-on. For example, from the introduction:

"if the network can learn this ideal score function exactly, then they will implement a perfect reversal of the forward process. This, in turn, will only be able to turn Gaussian noise into memorized training examples. Thus, any originality in the outputs of diffusion models must lie in their failure to achieve the very objective they are trained on: learning the ideal score function. But how can they fail in intelligent ways that lead to many sensible new examples far from the training set?"

Their answers to these questions are very good and also cover things like correcting the output of previous steps. But the proof is in the pudding: the outputs of their alternative procedure match the models they're explaining very well.

I encourage you to read it; maybe you'll even find a new way to decompose images into surface material properties and lighting as a result.

I did read it, all the way through! It's really good. The part you are quoting is setting up the ELS, which does not memorize entire images due to the inductive biases of a CNN (translation symmetry, limited receptive field). But the equivalence to a patch moseic is still due to the assumption that the loss is perfectly minimized under those restrictions.

And I was impressed by the close fit to real CNNs/ResNets and even to UNets. But what that shows is that the real models are heavily overfit. The datasets they are using for evaluation here are _tiny_.

Edit: oh the talk is here btw, if anyone is curious https://youtu.be/c-eIa8QuB24

I've been thinking about this a lot. I want to know what's the smallest a model needs to be, before letting it browse search engines, or files you host locally is actually an avenue an LLM can go through to give you more informed answers. Is it 2GB? 8GB? Would love to know.

The board game DaDaDa is a great example of this.

someone just discovered the AI equivalent of "everything turns to crabs"

Seemed to me they are just saying that typical LLMs are consistent:

https://en.m.wikipedia.org/wiki/Consistent_estimator

Edit: I wrote my comment a bit too early before finishing the whole article. I'll leave my comment below, but it's actually not very closely related to the topic at hand or the author's paper.

I agree with the gist of the article (which IMO is basically that universal computation is universal regardless of how you perform it), but there are two big issues that prevent this observation from helping us in a practical sense:

1. Not all models are equally efficient. We already have many methods to perform universal search (e.g., Levin's, Hutter's, and Schmidhuber's versions), but they are painfully slow despite being optimal in a narrow sense that doesn't extrapolate well to real world performance.

2. Solomonoff induction is only optimal for infinite data (i.e., it can be used to create a predictor that asymptotically dominates any other algorithmic predictor). As far as I can tell, the problem remains totally unsolved for finite data, due to the additive constant that results from the question: which universal model of computation should be applied to finite data? You can easily construct a Turing machine that is universal and perfectly reproduces the training data, yet nevertheless dramatically fails to generalize. No one has made a strong case for any specific natural prior over universal Turing machines (and if you try to define some measure to quantify the "size" of a Turing machine you realize this method starts to fail once the number of transition tables becomes large enough to start exhibiting redundancy).

Regarding your second point I think there are two cases here that should be kept separate. The first is that you are teleported into a parallel dimension where literally everything works differently from here. In that case I do agree that there are several reasonable choices of models of computation. You simply have to pick one and hope it wasn't too bad.

But the second case is that you encounter some phenomenon here in our ordinary world. And in that case I think you can do way better by reasoning about the phenomenon and trying to guess at plausible mechanics based on your preexisting knowledge of how the world works. In particular, I think guessing that "there is some short natural language description of how the phenomenon works, based on a language grounded in the corpus of human writing" is a very reasonable prior.

[dead]

Platonism - not even once. Green is the smell of my grandmother's lawn on a hot summer day. Just because things are similar to a lot of people doesn't mean their fundamentally the same.

LLMs are bruteforce reverse engineered human brains. Think about it. Any written text out there is written by human brains. The ”function” to output this is whatever happens inside the brain, insanely complex.

LLM ”training” is just brute forcing the same function into existence. ”Human brain output X, llm output Y, mutate it times billion until X and Y start matching”

The devil is in the details.

I recently gave the "Veeam Intelligence" a spin.

Veeam is a backup system spanning quite a lot of IT systems with a lot of options - it is quite complicated but it is also a bounded domain - the app does as the app does. It is very mature and has extremely good technical documentation and a massive amount of technical information docs (TIDs) and a vibrant and very well informed set of web forums, staffed by ... staff and even the likes of Anton Gostev - https://www.veeam.com/company/management-team.html

Surely they have close to the perfect data set to train on?

I asked a question about moving existing VMware replicas from one datastore to another and how to keep my replication jobs working correctly. In this field, you may not be familiar with my particular requirements but this is not a niche issue.

The "VI" came up with a reasonable sounding answer involving a wizard. I hunted around the GUI looking for it (I had actually used that wizard a while back). So I asked where it was and was given directions. It wasn't there. The wizard was genuine but its usage here was a hallucination.

A human might have done the same thing with some half remembered knowledge but would soon fix that with the docs or the app itself.

I will stick to reading the docs. They are really well written and I am reasonably proficient in this field so actually - a decent index is all I need to get a job done. I might get some of my staff to play with this thing when given a few tasks that they are unfamiliar with and see what it comes up with.

I am sure that domain specific LLMs are where it is at but we need some sort of efficient "fact checker" system.