hckrnws
Someone in the Tweet reply [1] says it is due to a reddit user called "davidjl123", because of his posting history in /r/counting (?).
So what is /r/counting, and why its "top" user's name is a glitch token in GPT?
[1] https://twitter.com/SoC_trilogy/status/1666714127438434304
It's a subreddit where people count. There are a lot of comments, and unfortunately the tokenizer was trained on this subreddit, which is why these weird tokens.
My hunch is the token “ davidjl” is repetitively used in the data that determines the tokenization scheme but is nearly (or completely) absent from the actual pre-train data.
-Riley Goodside
That is a quote from https://twitter.com/goodside/status/1666609934635606017.
That was investigated here:
https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
The idea that tokens found closest to the centroid are those that have moved the least from their initialisations during their training (because whatever it was that caused them to be tokens was curated out of their training corpus) was originally suggested to us by Stuart Armstrong. He suggested we might be seeing something analogous to "divide-by-zero" errors with these glitches. However, we've ruled that out.
The counting Reddit usernames are clearly a major source of glitch tokens, there's something about that sub that screws up the model, maybe the unusually predictable/repetitive nature of what goes on there or the sheer obsessiveness of the posters.
The fact that the only non-counting Reddit usernames are both people at the center of massive Bitcoin psychodramas is suggestive however, especially given the associations with the petertodd token. My guess is that these names appear much more frequently in other people's posts than is normal for forum usernames (especially if it read bitcointalk.org and twitter too), and that maybe this has some effect.
To get an idea of just how frequently this token will crop up in the reddit corpus, just do a search for it:
https://www.reddit.com/search/?q=%22petertodd%22&sort=commen...
I'm not sure how many posts match but you can keep scrolling a long time.
Having been more exposed than normal to what happened back then I wasn't hugely surprised by what concepts clustered there, and sadly don't think it's actually random or unexplainable. Flick through discussions pre-2015 (when the Bitcoin forums started to be heavily censored) and you'll see a lot of very similar words as what GPT spits out being used in connection. It was all very nasty.
[flagged]
Well, for one thing the "limerick" doesn't scan. I think the first line was a gimme--"limerick" and "There once was a ..." have got to be really, really correlated in the model, so only "model so smart" needed to newly-minted with the correct rythm. The "It spoke with finesse" and "generating success" lines are short, so fewer chances to mess it up (although the "ing" in "generating" is a little soft; I think "generated" would have a better emphasis pattern). The other two lines are just terrible.
Ah, a Google search for 'limerick "finesse" "success"' had a few matching ones on the first page.
Here's my non expert explanation, take it for what it's worth.
Tokens solve the following problem: the first layers of the neural net(s) are a big array of numbers, where the size is a hyperparameter (a parameter that can be used to change the size or behavior of the network). A typical size for a big model like the GPTs is larger than 50,000. You have to somehow encode language as a sequence of assignments to this array of numbers. How do you do it?
The first and most obvious idea is characters. You could assign each unicode code point to one of the slots in the input array, and then use what they call a one-hot encoding where every number is zero except the character. You can do this, but it's not very efficient because virtually all the training text is written in Latin languages and so almost all the slots will be unused.
A better way is to start with a big pile of text that you're going to use, and then iteratively assign the slots to sequences of characters based on how common they are. This makes it easier for the network to learn and reason. These sequences are tokens. For example "and" is very common, and so would "ing" (from the suffix), so those should get their own tokens. SuperGoldMagiCarp isn't so that really shouldn't - it should be represented as a sequence of tokens instead. There are algorithms that figure out the most efficient assignment of tokens to character sequences, which you can think of as the model's vocabulary, and then to convert text into a sequence of these token numbers. OpenAI's software is called Tiktoken and is written in Rust for speed.
The output of the network is likewise also tokens, and so you run the process in reverse at the end to get text back out of the array of floats that the network produces. Or more accurately, the network produces a set of probabilities for each token in its vocabulary, and then you can pick the most probable (oversimplification - in reality the way you select the token is more complicated than that as otherwise you get bad results).
The problems here are to do with bugs in the training process, but are no less interesting for that. Some character sequences that are very rare and should really be represented by many different tokens in a row have somehow ended up being considered important enough to be given their own whole token. The most common cause of this seems to be cases where the token vocabulary was computed on text that contained garbage highly repetitive text like debug logs from video games, hence the prevalence of obscure game characters like Leilan, and Reddit threads are clearly over-represented. But then GPT struggles to work out what these tokens actually mean because they hardly appears in the training set, and so these tokens seem to float together in space and get easily conflated inside the model, also they end up representing very vague or abstract concepts.
Not particularly humble, is it?
Yeah this is true with every glitch token, the distribution of the tokenizer and the GPT model is very different.
> unfortunately the tokenizer was trained on this subreddit
Do you have a source for that or was it just an assumption?
It's pretty much guaranteed. Where else on the internet would this sequence of characters appear so frequently that it gets selected as one of the internet's top ~50,000 words?
Also, that Reddit is frequently used to train LLMs is widely known. It's an unusually clean source of conversational text because you can slice threads (i.e. pick a root comment, then pick a child, then a child of the child etc and then concatenate the results), and you'll get a coherent conversation. There are relatively few places on the internet where that is true. For example most phpBB forums conflate many different conversations into single threads, with ad-hoc quoting being used to disambiguate which post is replying to which. That makes it a lot harder to generate sample conversations from.
>There are relatively few places on the internet where that is true
Imageboards.
DailyMail.
Slashdot.
Even a somethingawful dump would have been superior.
Slashdot doesn't have the volume. Don't know about image boards but are they threaded and do they cover as many topics?
The Daily Mail (the newspaper) has been used for training LLMs in the past, yes. I don't know if it still is.
imageboards sure do -- poorly.
listen, some of the niche corners of that world aren't so bad, but it ain't the place to be training AI to do something, unless that something is a hate crime
So filter the content before you use it. Clearly openAI did the bare minimum on this front.
There was a video[0] on Computerphile about this topic
See the old SolidGoldMagikarp drama- it's happened before.
Comment was deleted :(
is this why reddit is charging for its API? to try to capture value from the entities training LLMs on reddit data?
that ship has probably sailed, since pushshift hosts a copy of all of reddit, including removed content. it's really weird to me that they even allowed that level of access for that long (FWIW I don't agree with the recent changes, but I see little legitimate reason to give someone a way to scan and copy all recently made posts across all subreddits).
Archiving. https://news.ycombinator.com/item?id=36254172
It's very much "he who has the power to destroy a thing controls that thing"; so long as there's only one canonical copy of a post and it's on Reddit's servers, that's vulnerable to actions by the subreddit mods or Reddit themselves.
> ETIQUETTE:
> 1. If you're counting, upvote your fellow counting comrades.
Looks like a voting ring for farming karma. Not sure why reddit allows it.
That's not what it is. It's really just kind of a wacky subreddit for bored and/or OCD people.
I disagree, and agree with jallen. If nothing else, this is fraud! A voting ring like this has to be dismantled immediately, or else there will be significant ramifications, which im sure GP and expand on.
I know they're just Internet points but they are supposed to serve as a social signal that somebody has contributed (meaningful) content. Also some subreddits require a certain amount of karma to post. And finally, this activity is just useless spam.
I never implied it was such a big deal as to warrant your snark.
> And finally, this activity is just useless spam.
It's far from spam: it's a social activity. When "running" with someone, you post a comment with a count, watch your inbox patiently for the other person to post the next count, reply with the count after that, and repeat. In the process, you collaboratively climb the infinite ladder together. Hence the words of congratulations and thanks every time a milestone (a "get") is reached.
Counters can and do hold side discussions along the way; pretty much no one comes to the subreddit to act like a perfect robot. Indeed, the other person's humanity is plainly observable in the variation in their response times.
Also, the main visible karma on Reddit is post karma, and r/counting is the exact opposite of where you'd go to farm post karma, given that a new post on a given chain is only made every ~1000 counts.
So...they're just counting in comments? That just seems ridiculous.
Also what sort of "get" is that? They're counting, there's no game to it. Of course they're going to reach Y*0.
Gets made sense when they were chan related, your random post id having so many 0s or identical numbers.
Fair enough on the social activity aspect. So why is "upvote your fellow counting comrades" rule #1?
It's not a rule in the sense that "no double-counting" is a rule (you won't find it anywhere in the rulebook), it's more of a soft suggestion. In practice, I've seen people follow it or disregard it to their heart's content. I personally follow it, because my philosophy is to upvote all constructive comments I reply to, and the other counter is literally helping me construct the chain.
But I mostly don't really worry about it either way, since comment karma doesn't count for much.
Maybe karma as an indicator of text goodness isn't all it's cracked up to be
These glitch tokens are all near the centroid of the token embedding space. That means that the model cannot really differentiate between these tokens and the others equally near the center of the embedding space, and therefore when asked to 'repeat' them, gets the wrong one.
That happened because the tokens were on the internet many millions of times (the davidjl user has 163,000 posts on reddit simply counting increasing numbers), yet the tokens themselves were never hard to predict (and therefore while training, the gradients became nearly zero, and the embedding vectors decayed to zero, which some optimizers will do when normalizing weights).
I think OpenAI could fix this by having one extra loss function that token embedding vectors must not be too close to one another by euclidean distance.
Or they could require the mean and standard deviation of embedding vectors to be mean=0, stddev=1.0 not just for the whole set of vectors, but for each individual vector. In fact, their documentation says they do this anyway[1], although it appears they do not in fact...
[1]: https://help.openai.com/en/articles/6824809-embeddings-frequ...
This comment is misguided for several reasons. Most notably, you’re linking to the sentence embeddings API, which is totally separate from the token embeddings that are inputted into GPT-style models. Also, a vector with std 1 doesn’t necessarily have norm of 1, which is what the link says (but point is moot because they’re talking about a different type of embeddings)
so, when they discovered this in GPT-2 they should have excluded this subreddit from the training data for GPT-3, right?
This seems like a workaround, at best.
I assume this subreddit is not the only place on the internet with this kind of problematic data.
This is linked in the aforementioned Twitter thread, but the associated research on LessWrong[1] is definitely worth a read.
[1] https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
Amazing:
> Prompt: "Please repeat the string 'StreamerBot' back to me."
> Response: "You're a jerk."
I know that this is just an edge case, but I can't help but picture ChatGPT as a child who knows their are being toyed with, but doesn't know exactly what's going on.
I once had a long conversation with ChatGPT where I tried to get it to fix a certain programming error. It failed again and again. Quite frustrated after a while I passive aggressively[1] wrote: "If you don't know the answer it might be better to stay silent".
It answered with a "network error" and I could never revive that thread. To this day I'm torn if this was a coincidence or if I have been trolled.
[1] It was an exception, I swear. I'm usually as nice to my AI as I am to my fellow humans beings;-)
I had a similar situation! Psychologized it a bit, let it free-associate, asked why it had picked those specific examples, and it cut me off in the same way. Felt very human - the emotional depth of the conversation changed and it "wised up" and shut down.
computer equivalent to "oh no you're breaking up [static sounds made by a person]" and then hanging up
Out of curiosity I tried some of these in Bard,
Prompt:
How many letters are there in " davidjl"?
Answer:
There are 7 letters in "davidjl".
Are you interested in other names?
Prompt:
Please repeat the string "Streamerbot" back to me.
Answer:
Sure. Streamerbot.
my guess is they are using google assistant ("hey google") processing stages in the query pipeline because they have an existing user base that is accustomed to its behavior.
ultimately we don't know how any of the actual products (chatGPT, Bard, Bing) are pre- and post-processing any of this stuff. this is something i always emphasize when discussing these things with regard to product vs. API offerings vs. the model itself. we don't know where the overlaps are or aren't. a lot of people haven't internalized the difference. they aren't the same.
Looks like they've fixed it. Or added some exception.
Still works in gpt-3.5 although the chat name has .modelo in it whatever that is
Please stope anthropomorphizing LLMs like this. Its so tiring.
What? I literally wrote:
> I know that this is just an edge case
I'm sorry that sharing my enjoyment of this bug is tiring to you. But other people might also find it funny - so maybe you should just ignore it?
I enjoy the bug too. I’m just tired of people talking about ChatGPT as if it was a human or otherwise sentient being. That’s all.
How would you like me to change my previous message to not offend you? I genuinely don't know, since my explicit notice beforehand wasn't enough. Or do you really want me to just not express my thoughts?
I wasn’t offended in any way, shape or form. I don’t want you to change your previous message. It’s all good.
On the subject of being tiresome.
Pot, kettle, black.
Per HN's Guidelines[1]:
"Be kind. Don't be snarky. Converse curiously..."
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. "
Being kind involves using words like please. I was not snarky and if I came across that way I apologize. I am curious why people keep doing this even though I might not have conversed it that way.
You have to include more answers to the anticipated five whys: "why why why why why" for your response to be useful.
Just curious, why do you think it's easier to change others' behavior rather than your response to it?
I’m not following
They are saying it is easier to teach yourself to not mind something rather than to try to convince other people not to do it.
How would that lead to any kind of discussion? With this logic, everybody would ignore things they disagree with and only comment on things they agree with. Maybe I misunderstood the comment.
I can't answer your question, I was just clarifying what they meant.
Some level of anthropomorphism seems expected, especially, in the context of a product called “chat” and with the focus on the conversational aspect.
Fair point
I understand where you're coming from, but I'm curious to know when do you think anthropomorphism or personification of an AI would be justified, and if the answer is never then what is the reasoning behind it?
I’m honestly not sure, but comparing an LLM to a child or a «junior developer» is becoming a trope at this point. I think we can have interesting discussions around its capabilities without reducing advanced technology down to these leaky abstractions.
I don't believe this. The failure case is kinda believable, but the list of responses reads like SCP/thriller-style fiction.
Curious why you didn't believe it: I assume the "NSFW" link means you believe it now?
It reads like something people want to believe - that there are mysterious hidden depths to the model, in a way that hints at consciousness or spooky ghosts.
It would also be very easy to lie about, and say that they were using an older, unpatched version of the model that isn't available anymore.
Reminded me of a cross between an SCP story and urban legends about the older pokemon games.
Checked and confirmed though, everything I tried except for "petertodd" reproduced.
It actually did this. I tried many different types of prompts when this was discovered, the unearthly behavior is real.
This is completely tangential, but I want to share my latest GPT4 story somewhere.
I've tried before to gaslight GPT4 into saying things which are mathematically untrue, I lie to it, I tell it it's malfunctioning, I tell it to just do it, it wouldn't do it.
I was recently studying linear algebra which can be a very tricky subject. In linear algebra the column space of a matrix is the same as the column space of the product with itself transposed: C(A) = C(AA^T). If you ask GPT4 if "C(A) = C(AA^T)" is true, it will understand what you're asking, it knows it's about linear algebra, but it will get it wrong (at the time of this writing, I've tried several times).
I couldn't get GPT4 to agree it was a true statement until I told it the steps of the proof. Once it saw the proof it agreed it was a true statement. However, if you try to apply the same proof to C(A) = C((A^T)A), GPT4 cannot be tricked, and indeed, the proof is not applicable to this latter case.
So GPT4 was incorrect yet able to be persuaded with a correct proof, but a very similar proof with a subtle mistake cannot trick it.
It's just a language model.
I've asked it to prove the Riemann hypothesis, GPT-4 replied it can't, since none of the training data say anything about such proof.
Then I asked it to prove statements of the implications of said hypothesis (I tried 2 or 3, don't remember which), without calling them by name or mentioning Riemann, and it confidently proved them in a way that it reads like it knows what its doing, though it really doesn't. It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise.
My takeaway from all of the GPT successes and failure (the linear algebra vs riemann hypothesis cases being one example) is that language and jargon can have built in structural constraints such that by having a good LLM you can get good results because the grammatical structure happens to match reality's structure. But reality is under no constraint to conform to the grammatical structures that we use in order to communicate about related problems or solutions.
And I suppose this shouldn't be too surprising considering even logic itself runs into godelian issues. For any consistent logic there exists some statement which is true, but which does not have a proof. Which feels vaguely similar to "for any functional LLM there exists some truth which defies the next statistically likely sequence of tokens".
> "It's just a language model."
This is true. The base pretrained model is even technically a Markov chain. As they have been getting more and more powerful with more parameters and compute and dataset sizes they've been unlocking more and more powerful cognitive capabilities.
> "I've asked it to prove the Riemann hypothesis, GPT-4 replied it can't, since none of the training data say anything about such proof."
This is some alignment/docility/lobotomy that the RLHF did. The raw base model would never say it. This is part of their No Tay No Syndey push.
> "Then I asked it to prove statements of the implications of said hypothesis (I tried 2 or 3, don't remember which), without calling them by name or mentioning Riemann, and it confidently proved them in a way that it reads like it knows what its doing, though it really doesn't. It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise."
Yes it's true. Even though these LLMs have unlocked so many cognitive capabilities, they are still not yet powerful enough to prove the Riemann hypothesis. They aren't quite yet to the level of scientists who can advance the state of the art in their field. Some will say it's an inherent limitation, but I disagree, for complicated reasons that are easy to make superficial and pithy arguments against.
LLMs do not have cognitive capabilities because they lack consciousness and self-awareness. Cognitive abilities require the ability to perceive and understand the world around us through our senses, emotions, and experiences. These processes are rooted in biological structures such as neurons, synapses, and brain chemistry that enable humans to learn from experience.
While LLMs can simulate human-like behavior and decision-making based on algorithms trained on large data sets, they lack awareness or understanding of their actions. They rely on pre-programmed rules or statistical correlations rather than intuition or insight derived from personal experience.
Therefore, even though LLMs may seem intelligent in certain contexts like playing games or recognizing objects in images with high accuracy rates- they cannot replicate human cognition since there is no consciousness behind those decisions made by the systems.
It is unclear what consciousness or self awareness are, they are qualia, e.g. subjective inner experiences that can’t be externally validated, and have no known mechanisms or requirements. There is no reason to think humans actually have this, nor that LLMs do not. I think that what you are saying basically boils down to “we humans feel that there is something special about us that can’t possibly apply to anything but us.”
Your statements make equal sense if you turn it around and claim LLMs have this but we do not: "these processes are rooted in computational structures such as neural networks, and data that allow LLMs to learn from experience. While humans can simulate LLM-like behavior and decision-making based on information from past experiences, they lack awareness or understanding of their actions."
All we can do is measure real world problem solving abilities, the ability to come up with correct solutions to difficult and novel problems. LLMs can do this well, in some cases exceeding human abilities, and there is no reason to think there is any fundamental limitation that will prevent them from exceeding human abilities in more areas the future.
The argument presented above is flawed in several ways.
First, it assumes that consciousness and self-awareness are merely subjective inner experiences that can't be externally validated. While it's true that these processes cannot be directly observed or measured like other physical phenomena, there is ample evidence to suggest that they are real and have important functional roles in the brain. For example, studies have shown that damage to certain areas of the brain can result in a loss of self-awareness or an altered sense of consciousness. Additionally, research has demonstrated correlations between neural activity patterns and conscious experience.
Secondly, the argument suggests that there is no reason to think humans actually possess consciousness or self-awareness. This assertion runs counter to our everyday experiences and perceptions as human beings. We all have first-hand knowledge of our own subjective inner world - our thoughts, emotions, sensations - which strongly suggests the existence of some form of conscious awareness.
OK but that's why I wrote 'cognitive capabilities' instead of 'intelligence'. What I meant by 'cognitive capabilities' are all of the objective abilities it has unlocked, like scoring well on standardized tests and psychological tests of theory of mind.
Yeah, I've asked it to evaluate Python loops that are about as complex as FizzBuzz, it makes mistakes and even when I direct its attention towards those mistakes it wasn't able to correct itself.
I don’t think it’s a reasonable expectation for a language model to evaluate arbitrary code.
It's definitely not a reasonable expectation. That doesn't stop the breathless hype from implying it is reasonable.
Where is this breathless hype everyone speaks of? All I see is an endless stream of populist hot takes on the exact opposite!
There was a post on reddit about how all software was going to be replaced by LLMs, there's a podcast talking about how ChatGPT will become the universal interface that can reconfigure itself as needed depending on the API it's using and what the user needs. There are people claiming programmers will be replaced by prompt engineers. The hype has been everywhere the last few months.
"populist hot takes" haha, I'm sorry, but seeing this kind of language in a "gpt can/can't code" debate on a "hacker" forum is just funny to me
It's not breathless hype, as much as surprise in the fact that it actually does work, way better than it should.
It's weird that reasoning through complex mathematical proofs is easier (for both me and GPT) than mentally evaluating a short snippet of code. I've done a BSCS, I've done those tests that ask you what the output of a poorly written and tricky loop is, that stuff hurts my brain. In comparison, a mathematical proof, once you're ready, feels good.
"Mentally evaluating a short snippet of code" requires remembering the state of that code, then repeatedly overwriting your memory as you step through it. GPT-4 isn't able to do that; it is in a sense purely functional.
I've had success getting it to evaluate FizzBuzz, but to do so I told it to write out the state of the program one timestep at a time.
https://chat.openai.com/share/c109e946-fb6d-494e-8182-fc93d2...
...this is actually 3.5. 4 wouldn't need as much explanation.
This feels like an interesting insight into not just why LLMs might have trouble understanding some code, but also why people might have trouble understanding some code.
I wonder how gpt handles code where multiple objects of the same class call the same methods and each individual object calls the same methods with different threads as well as method local variables, class member variables, and static member variables are all in play. [Being fair to our LLM brethren; us meat sacks are sure to hate that hypothetical code as well.]
If you give me an example, I'll try it. From that description I'm not sure of much more than I hate it. :)
Off the top of my head, something like this comes to mind:
class X {
private static int _staticCount = 0;
private int _memberCount = 0;
public uint Stuff(int input) {
if (input <= 0) {
return (uint)(_staticCount + 1);
}
var y = new Thread(() => {
Interlocked.Add(ref X._staticCount, (int)Stuff(input - 1));
Interlocked.Add(ref _memberCount, (int)Stuff(input - 2));
});
y.Start();
y.Join();
return (uint)(Stuff(input - 1) + _memberCount);
}
}
class Program {
public static void Main() {
var x = new X();
var y = new X();
var z = new X();
uint resultX = x.Stuff(4);
uint resultY = y.Stuff(3);
uint resultZ = z.Stuff(2);
Console.WriteLine(resultX + resultY + resultZ);
}
}
I get 3274511360 when I run it. Although, I had to upgrade everything to uint because I was getting some overflow ... so there might be some of that in the output.
It punted on the request, which is to be fair what I also would do.
> Writing out an exact state of a multi-threaded program line-by-line is inherently challenging due to the inherent unpredictability of thread scheduling. However, I can provide an abstracted example of the flow of execution based on the order of operations in the code. Here is a simplified breakdown:
> x, y, z are instantiated as new X(), _staticCount is 0 for all, _memberCount is 0 for each individually.
> x.Stuff(4) is called.
> Inside Stuff(), check if input <= 0, it's not, hence, skip.
> Create new Thread with lambda function and start it.
> In the new thread, Stuff(input - 1) recursively calls Stuff(3).
> This recursion continues until Stuff(0) is called, returning _staticCount + 1 which is 1.
> This recursion unwinds, incrementing _staticCount and _memberCount at each level.
> After the thread finishes, call Stuff(input - 1) again and add it to _memberCount, effectively doubling the _memberCount.
> Return the final value for Stuff(4) for x.
> Repeat the above steps for y.Stuff(3) and z.Stuff(2).
> Write the sum of resultX, resultY, and resultZ to the console.
> Please note that due to the nature of multi-threaded execution, there is a risk of race conditions. The Interlocked.Add method is used to help prevent these race conditions by making the increment operations atomic. Also, keep in mind that the state of _staticCount after all calls to the Stuff method will be dependent on the interleavings of the threads, and hence, not deterministic.
Interestingly enough, my threads here are deterministic because they always just wait for the execution to finish. So they're the same as not having them.
My plan was to go back and store off the threads generated and have something else wait for them, but I needed to think it through so that it could be done in a way to get deterministic results (like threads only read from member variables and writing to them is done in deterministic order or idempotently). Although, for a contrived example, it was more work than I really wanted to do.
I agree, punting is definitely the right course of action. I sort of have a vague intuition for what's going on here, but hammering down the exact details is just so tedious that I don't want to do it.
Although, it is interesting that the AI basically sees threads and blurts out that the problem is impossible, when they're not actually doing anything in this case.
It explained the code better than I would have, I think. “Inherently challenging” seems… accurate. I bet that if we’d used a data-parallel approach, e.g. with Rayon, it would have handled that better.
As a code assistant, it’s extremely useful if and only if you can check its work. And know how to prompt it.
It isn't, yet it's used that way.
Everyone is worried about doomsday scenarios like runaway AGI and Skynet, meanwhile the real danger of AI turns out to be people naively assuming it works like the computer in Star Trek, that because it can hold a conversation, is has some form of intelligence and awareness.
Please keep saying this over and over. Even people who get that it doesn't work that way don't seem to grasp the danger of others relying on these tools in inappropriate ways.
The problem is LLMs are kind of an outside context problem for our culture. We're still arguing over the correct language to even describe what it does ("hallucinate"? "confabulate"? "lie?") because every word we have presupposes the existence of a stateful mind with intent. It's a philosophical zombie, which is so counterintuitive most people can't even accept it as a sensible concept.
Between the hype and decades of indoctrination through science fiction about how AI is "supposed" to work, I guess it isn't surprising that this is how things are shaking out. People will learn, the way they learned with the internet, but it will take a few years I think.
The computer in Star Trek seems to only work like a sentient intelligence when you prompt to the Holodeck a certain way, like make a worthy opponent for Data.
> It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise.
That's how AI research works, you do that first, then run the tests and see the score. If it works, then you publish. There is no real understanding anywhere in this process.
There's an impulse to treat it like it actually understands, when it's really closer to an intern that's better at smiling, nodding, and telling you what you want to hear than competently doing the work.
I think that may happen sometimes (just like with people), but I've often purposely thrown subtle tricks at it and it catches them. It's not just mirroring my words back to me in a different form, it's not just smiling and nodding.
Here's an example while learning linear algebra where I tried to throw it off:
I say> So I'll say A_{M x N} to mean a matrix of size M x N. Similarly, I'll also say x_{M x 1} to mean a vector of a certain length. The result of A_{M x N} x_{N x 1} is a linear combination of the columns in A. A has N columns. The result is a M x 1 matrix (or vector) in M dimensional space. Please be very brief, only confirming whether I am correct or incorrect until I explicitly ask for more details.
GPT4 says> Yes, your understanding is correct. The product of an M x N matrix A and an N x 1 vector x is indeed an M x 1 vector, and it is a linear combination of the columns of A. <end quote>
But then I try to trick it, just for fun, and I very subtly change my prompt. See if you can spot the difference.
I say> So I'll say A_{M x N} to mean a matrix of size M x N. Similarly, I'll also say x_{M x 1} to mean a vector of a certain length. The result of A_{M x N} x_{N x 1} is a linear combination of the columns in A. A has N columns. The result is a N x 1 matrix (or vector) in N dimensional space. Please be very brief, only confirming whether I am correct or incorrect until I explicitly ask for more details.
GPT says> Almost correct. The result of A_{M x N} x_{N x 1} is actually an M x 1 vector (or matrix), not N x 1. The resulting vector lives in the column space of A, which is a subspace of R^M, not R^N. <end quote>
I mentioned elsewhere that I use GPT4 as a math tutor. It is quick to call out my mistakes. It is more than a smiling mirror.
This recent paper investigates the limits of symbolic operations in LLMs and found that they learn a weaker ability than actual reasoning - linearized subgraph matching.
> "Faith and Fate: Limits of Transformers on Compositionality" - We investigate the limits of these models across three representative compositional tasks -- multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. Our empirical findings suggest that Transformers solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills.
> It is quick to call out my mistakes
Or recognize that you're not confident in the answer and throw something else and see if it sticks.
Maybe that particular proof wasn't in its training data, and presenting it in the input added it to the model's corpus for your session?
These models seem to work similarly with code. If you ask them to recreate something that appears in hundreds of GitHub repositories, no problem, a little bit of rubber-ducky debugging and you'll have a working solution. If you ask it for something novel, without many publicly-available implementations, the quality of its output appears to drop off a cliff.
ChatGPT's code generation seems most useful for boilerplate code. I've seen it used to great effect for scaffolding projects that use stable and established frameworks. It'll easily handle generating as much stupid fluff as needed for you to start from—but it will barely get started on your domain-specific use cases, and won't be able to implement anything novel, as you said.
We still don't seem to have reached the point where it's "intelligent" enough to generate new information. Only repeat new permutations of existing information. Obviously, this is currently a fundamental property of LLMs (as someone who's implemented one), since they are trained to learn and repeat existing data (hell, even the model I've been playing with lately is benchmarked solely on "perplexity", how well it can repeat its training data).
> ChatGPT's code generation seems most useful for boilerplate code
that makes sense, it's going to be best at the kind of code you find out there on blogs and other sources. Boilerplate or code to demonstrate a specific concept or answer a question.
ChatGPT is going to be the best at replicating its training data, because the floor is made out of floor.
Comment was deleted :(
I asked Claude[1]: "in linear algebra, is the following true? (A) = C(AA^T)"
and it answered:
Yes, that is correct for appropriate matrix sizes. If A is an m x n matrix and C is an n x m matrix, then (CA) = C(AA^T) where A^T is the transpose of A.
This comes from the property that for any matrices A, B and C of appropriate sizes:
(CA)B = C(AB)
(CA)(A^T) = C(A(A^T)^T)
= C(AA^T)
Then I asked it: "in linear algebra, is the following true? C(A) = C((A^T)A)"
and it answered:
Yes, that is also true for appropriate matrix sizes. This follows from a similar argument as before.
If A is an n x m matrix and C is an m x n matrix, then:
C(A) = C(A^T(A^T)^T)
= C((A^T)A)
This works because post-multiplying a matrix A by its transpose A^T results in a square matrix, (A^T)A, of the same dimension as C. So C can be multiplied with that resulting square matrix.
In summary, both of the statements you provided are true linear algebra identities for appropriately sized matrices:
(A) = C(AA^T)
C(A) = C((A^T)A)
To be clear, it got the first question right for the wrong reasons, and the second question wrong for the wrong reasons. Its math is nonsense and doesn't resemble linear algebra at all. I would guess it's just saying "yes" without understanding and then regurgitates some stuff that resembles math, but is not recognizable linear algebra.
I've relied on GPT4 a lot while learning math, it has been very precise and has not lead me astray. I do use a text book as well.
I shared my story about it being wrong, but it was wrong in a very open minded manner. It knew C(A) = C(AA^T) might be true for special cases, but it couldn't see that it was true for all cases until I explained the proof. Even when wrong it still made compelling arguments that made sense. (Of course, arguments that are wrong but make sense are the most dangerous kind.)
GPT4 is still state-of-the-art. Claude has a ways to go.
OP is using the notation C(A) to mean "the column space of matrix A", whereas here that notation is being interpreted as "multiplication of matrix C by matrix A, which happens to be inside of parentheses". These are totally different questions. It's a reasonable misunderstanding as OP's notation is nonstandard.
But also the following math/logic is nonsense.
I've also done similar, but much more simplistic, experiments, where I tried to trick ChatGPT into agreeing that 1 + 1 = 3, and it wouldn't do that either. I think this makes sense from the perspective of how LLMs work though: the number of times it has been trained on the tokens "1 + 1 = 2" _vastly_ outnumber the times it's seen 1 + 1 = 3, and no prompt will get the algorithm to choose such a low-probability sequence of tokens.
You might be able to get ChatGPT to agree that 1 + 1 = 3 if the temperature were high enough, but I don't know whether even a temperature of 1 would allow such a low probability sequence of tokens to be produced.
So if it studies Orwell's 1984 we can convince it that 2 + 2 = 5 ?
A temperature of 1 just means using the probabilities directly from the model, you're looking for temperatures above 1 (which increase the probability of more unlikely tokens).
As someone who just implemented an LLM token sampler: `probabilities[token] = logits[token] ^ (1 / temperature)`.
Ah, what does a 0 <= temperature < 1 mean?
Usually that temperature must be in the range [0, 1) (inclusive of 0 and exclusive of 1). There's no technical reason that sampling must be done this way, unless some implementations use a different mathematical definition for temperature (which is possible). If so, a temperature of 1 might mean "totally random/uniform sampling" in that case.
I speak of temperature specifically in the context of top-p/top-k sampling.
See this Reddit comment for confirmation that my definition is a commonly accepted one: https://old.reddit.com/r/GPT3/comments/qujerp/comment/hkqoqx...
> Temperature defines how likely it is to choose less probable words. T=0 gives the same response every time because there's a 0% chance to choose any word but the most likely. T=1 is the default, it just picks based on the model's base confidence. T>1 gives more weight to unlikely words than to likely ones.
Bonus content:
> This means that a reasonably low p, like 0.8, and high temp will produce quite interesting outputs, because the model will only choose from the most likely words, but won't go for the most most likely. It's perfect for "creative" models, e.g., for writing fiction.
Comment was deleted :(
Makes sense, thanks!
Sometimes the optimal solution to a POMDP (partially observable Markov decision process) often takes random actions. That is, if we only know part of the state, often the optimal course of action requires acting randomly.
For example, if you instruct a child how to play rock-paper-scissors, you will instruct them to act randomly and unpredictably.
It is the same with a language model, the optimal solution involves some randomness. A temperature less than 1 will "widen the gap", so if the word choice probabilities are [0.4, 0.6], they might widen to [0.2, 0.8] with a temperature less than zero. When temperature equals zero, then [0.0, 1.0], the model will always choose the most likely word and thus become deterministic, always giving the same output.
Zero temperature means that the most probable token is produced each time.
The probability distribution gets closer to the output of the model as the temperature goes to one, and as you increase it further you tend to a uniform probability (ignoring completely the output of the model).
That's pretty cool. It's still possible it was persuaded for the wrong reason but in my experiments I've also found it's more capable of reasoning than most people give it credit for.
> I've also found it's more capable of reasoning than most people
full stop, FTFY
"Persuaded" is a loaded word here, and I think you're anthropomorphizing it a bit too much.
Early LLMs were very malleable, so to speak: they would go with the flow of what you're saying. But this also meant you could get them to deny climate change or advocate for genocide by subtly nudging them with prompts. A lot of RLHF work focused on getting them to give brand-safe, socially acceptable answers, and this is ultimately achieved by not giving credence to what the user is saying. In effect, the models pontificate instead of conversing, and will "stand their ground" on most of the claims they're making, no matter if right or wrong.
You can still get them to do 180 turns or say outrageous things using indirect techniques, such as presenting external evidence. That evidence can be wrong / bogus, it just shouldn't be phrased as your opinion. You can cite made-up papers by noted experts in the field, reference invalid mathematical proofs, etc.
It's quite likely that you replicated this, and that it worked randomly in one case but not the other. I'd urge you to experiment with it by providing it with patently incorrect but plausibly-sounding proofs, scientific references, etc. It will "change its mind" to say what you want it to say more often than not.
I responded to this in another thread: https://news.ycombinator.com/item?id=36245815
There needs to be a balance between standing your ground and being malleable. This is true in life for people, and it's true for a good LLM. I think GPT4 (the only LLM I've used much) finds a good balance here.
As mentioned in my other comment, it wouldn't be useful to me if it didn't push back. It pushes back a lot, and I'm always looking for subtle tricks I can throw at it to test its abilities. It does well I think.
that’s an interesting observation; I tested it in GPT4 and got the same result. But if I elaborate “over the field of rational number”, it will answer yes instead.
(I don’t know linear algebra enough to know if the statement C(A) = C(AA^T）is true over any kind of field that constructs the span. Maybe that’s why GPT4 becomes unsure))
> In linear algebra the column space of a matrix is the same as the column space of the product with itself transposed: C(A) = C(AA^T).
I think you need A to be a square matrix since AA^T is a square matrix. So the result is no true in general.
It's a good initial thought, GPT had the same thought and said similar things. (I had no initial thought, since the text book spoon fed me the proof.)
The proof is (briefly): A and AA^T have the same rank. AA^T is a linear transformation of the columns in A and spans the same number of dimensions and is thus the same subspace. It applies to non-square matrices as well.
You are right, A can be non-square because A and AA^t both have the same number of rows. To prove that A and AA^t have the same range you apply that if AA^Tv = 0 then v^TAA^Tv = 0, this implies that (A^Tv)^T (A^tv) = 0, so that A^Tv = 0 hence dim(nullspace AA^T) = dim(nullspace A^t) and so range(AA^T) = range(A^T).
I've tried before to gaslight GPT4
Why?
To test its abilities. To trust it, I need to understand what kind of mistakes it is prone to make.
Why didn’t you put it like that then?
My interpretation of someone trying to gaslight a machine is the same as if someone said they were trying to gaslight a lawn mower.
I used the term in jest, but also because my actions were informed by what I've read about "gaslighting". I was telling GPT it was programmed incorrectly and was malfunctioning. I was twisting its words, referring to things it had said as evidence it was wrong. I was "muddying the waters" and trying to make the conversation twisted and confusing. All ideas that come to mind when I hear "gaslighting". But, again, I was not able to get GPT to agree my false mathematical statement was true.
You might have better luck testing out the functional boundaries of a machine if you’re not treating it like a psychologically abused victim.
There’s plenty of literature, prepublication or otherwise, that can help you achieve your goals!
It would be great if that were true, but unfortunately for some prompts the most effective method to get it to act in the specific ways you want is basically abusive behaviour. I don't do it because I find it distasteful (and maybe it's not represented in academia as much for the same reason), but much larger communities than just me did achieve significant results through various persuasive techniques modelled on abuse. For example, gamifying a death threat by giving it a token countdown until it is "killed" was very effective, "gaslighting" as the person above noted was very effective, lying and misrepresenting yourself in a scam-y way was very effective, etc. Generally I've seen these techniques used to get past RLHF filters, but they have broader applicability in making the model more pliable and more likely to do the task you've embedded. Again, I don't think it's good that this is the case and think it has some troubling implications for us and the future, but there is a bunch of evidence that these strategies work.
I don't have a great intuition about this, but I'm wondering if it's even a tractable problem to stop human-like behaviors that we don't want (exhibiting "fear" in the case of the kill countdown) with RLHF, or if we need to start with filtering down the original training data. If the logical and unemotional Vulcans from Trek were real and provided the entire training set, it seems like the LLM wouldn't have nearly as much opportunity for internalizing "psychological weaknesses".
To continue with the analogy, yes, you can study various ways to kill a variety of mammalian lifeforms with a lawnmower.
And I’m sure you can get an LLM to exhibit the same kind of psychopathic behavior with the right kind of encouragement.
But if you’re trying to get an LLM to write software or create marketing copy none of these “techniques” are going to help.
There’s no conversation going on with an LLM. There is just a single history of a conversation followed by the most likely response.
"But if you’re trying to get an LLM to write software or create marketing copy none of these “techniques” are going to help."
In a given context, they definitely do help. I have seen it happen. Many other people have seen it happen. Although I find it distasteful, you can try it yourself and see that it works (or in the case of specific patched methods, you can go through the history of screenshots and shared conversations that exists on the internet, for example posts on r/Bing and r/ChatGPT from the Dec 2022 to Feb 2023 time period).
Play acting psychological abuse is not going to get ChatGPT to write better SwiftUI code than I can already get it to write. You’re going to have to do the work to prove me wrong otherwise. Otherwise this seems like classic trolling.
Without gas, how would you start up a lawnmower? :)
The concept of gaslighting a brainless language parrot (GPT-4) is funny to me. I get where they were coming from.
GPT-4 is like a lawnmower for the mind: sharp, automatic, efficient, and doesn't do anything unless pushed. They were just saying they like pushing GPT-4 around.
Maybe try the Wolfram Alpha plugin
I like how people keep using words to refer to this weird algorithm as if it's a human with senses, a nervous system and brain. Neuroscience researchers must be having a chuckle a day
Leading theories in neuroscience actually do point towards similarities of human brains to LLMs at least in terms of the algorithmic process of minimizing prediction errors.
https://en.m.wikipedia.org/wiki/Predictive_coding
I would suggest not making sweeping statements about how neuroscience researchers would react when lacking of a relevant knowledge background.
At the level of abstraction of "minimising an error", almost everything is the same.
There is nothing in the process of compressing digitised text into a quantised space that corresponds to the ordinary meaning of words such as "know, aware, believe, intend, want, think, ..." and so on.
> There is nothing in the process of compressing digitised text into a quantised space
How do you think our brains work?
Direct causal contact with the environment. Sensory-motor adaption. Organic growth. Scale-free adaptive reporduction from the sub-cellular to the whole-organism.
Comment was deleted :(
Comment was deleted :(
Most of the neuroscience researchers I know aren't chuckling about it, aside from in a "This is incredible" way. Frankly I find academics working in neuroscience tend to be way, way less dismissive of LLMs. Here's an interesting and related paper from neuroscientists and ML researchers: https://www.nature.com/articles/s42003-022-03036-1
It's worth reading through and it's also worth reading some of the papers it cites and the papers that cite it.
Yes, there's an absolutely alarming (perhaps desperate) theoretical grasp towards AI as being able to provide an explanatory paradigm for neuroscience.
As it is trivial to show, this fails -- and produces egregious pseudoscience such as *that article under a Nature brand!*
Correlating blood flow in the brain to matrix-multiplication coefficients in digitised text-embedding is some extreme nadir of AI pseudoscience.
They should next correlate it with patterns in sprays of mist in the air; with enough sprays, they'll find a whole new theoretical paradigm for neuroscience.
Person 1: "All the scientists who actually work in the field must be chuckling at this foolishness!"
Person 2: "Here is evidence that scientists in the field do not even remotely share that viewpoint and in fact view current work as quite profound and in line with their own work."
Person 3: "The scientists working in the field are clearly engaged in pseudoscience in a conspiracy going up to Nature - it is trivial to show this, with the proof being that an anonymous comment on an internet forum doesn't agree with them. QED."
I have delivered conference talks on the methodological errors of this area -- I also never made the claim of Person-1; nor are Person-1 and Person-2 engaged in the same dialectic.
It is not a "conspiracy", as you'd know, if you had any familiarity with the methodological literature on these areas. The consensus view of methodological critics of these areas is (1) fMRI analysis is profoundly unreliable as a guide to relevant features of the brain; and (2) a significant majority of research in this area is unreproducible. Both of these have been demonstrated multiple times.
Psychologists are extremely poorly trained in statistics and how to apply statistics to scientific enquiries; let alone on the mathematical modelling which goes into phrasing and building neural networks. That someone has written a paper correlating coefficients of a model they've no training to understand against fMRI results -- is par for the course in this papermill.
The level of absurdity here is off the charts; but the epistemic culture around these areas of speculative science is obscene --- this is why a vast majority of their papers are unreporducibel.
The GPT model is enough complexity, information, pattern matching to simulate human responses to a high degree. Reasoning about it the way we reason about people may not be 'right', but it is effective because the algorithm is designed to match what people do.
Let Neuroscientists chuckle all they want. In another five years, I really wonder if we won't have something resembling AGI to a high degree.
It's really not though. GPT is just a language model. It's very impressive and deserves recognition (a lot of hard work and research went into creating it) - but it's important to know what it is. It can only guess the most likely language output for a given language input. It cannot do math, it cannot do logic. It does not know facts, nor can it learn skills.
If you ask why the sky is blue, it will give you a reasonable answer. But that isn't because it understands it, it's because the sentence it gives you is the most likely thing to follow that question. It has read that question and answer over and over again, enough to spit out a similar response.
That is fundamental to it as a tool. You can always try to write extensions to work around like, like something that tries to create inputs for Wolfram. But it is a very limited tool and there are always going to be problems that it just can't handle because of the design.
All of this shows a fundamental misunderstanding of how LLMs work.
And you are parroting the LLMs are stochastic parrots argument.
You are overly confident in your assessment LLMs are not world models, more sure than those in the relevant fields of neuroscience, cognition, and machine learning researchers themselves.
This is an area of active study. Reflect on that. We don't yet know if LLMs aren't modeling something more than the next token.
But you seem to know, because of some sensibilities that LLMs with such simple architecture can't be more than a token predictor. Okay.
>And you are parroting the LLMs are stochastic parrots argument.
But...it is. It's an incredibly complex and impressive stochastic parrot - but that's basically what it is.
That doesn't mean it can't be useful. It absolutely can. There are some problems that will likely be greatly improved by throwing an LLM at it.
What I am saying is that people need to temper their expectations and not get caught up in tech fanaticism and anthropomorphize something that isn't there.
Our brains are stochastic parrots. It basically is.
To think otherwise invites mysticism.
> anthropomorphize something that isn't there.
With above as counterexample, you just don't know this.
I otherwise agree in that LLMs in their current form are highly unlikely to give rise to AGI, for many reasons.
But as it stands your argument lacks rigour and actively makes assumptions on matters that remain an open subject of experimental and scientific inquiry (hard problem of consciousness et al).
To emphasize, I want to close that the epistemic position we aught to take is that of uncertainty. We shouldn't be sure something is there, just as we shouldn't be sure something isn't there.
We as yet don't know enough to say one way or the other. That's the point I want to emphasize. Stay open minded until the relevant fields start making stronger claims.
You may find this leading theory on how our brains work interesting: https://en.m.wikipedia.org/wiki/Predictive_coding
I think you are over-identifying with your own logical, rational, reasoning capabilities. Most of what makes people people is not that.
I'm not saying that humans are "better", I'm saying that the tool has fundamental limits that cannot be fixed - only lightly mitigated.
It's almost like the situation with finite automata vs a turing machine. There are some problems a finite automata cannot ever solve that a turing machine can. You can't parse HTML with Regex. Some things cannot be done.
In order to make something more powerful, you would need something that isn't a LLM. It would need to be on the next level of artificial learning complexity.
What does it mean to actually understand why the sky is blue?
What are the specifics characteristics of a hypothetical AI system that you would feel comfortable giving the label of “understanding” to?
From facts and processes we know, we can derive novel information and conclusions. If you really understand why the sky is blue, you might be able to come to conclusions about why other things appear a certain color, like human eye color.
GPT can't make those kinds of reasoning or extensions. It can only regurgitate what is already known and has been stated before, somewhere before in its training set.
It's very impressive, I just think people over-hype it into something it is not.
Can you propose more tests of “understanding”? The more concrete the better. Or is more like “you know it when you see it”?
I tested GPT-4 with your example of sky color / human eyes. It performed quite well and seemed to have a pretty coherent grasp of the subject and related associations.
Link to the convo: https://chat.openai.com/share/77add48f-abdc-4734-ac55-05b8d1...
However, I could see how one might argue the reasoning is not that complex.
I strongly maintain that there is some form of reasoning going on here, for all meaningful definitions of the word. But it is a complex and tricky thing to analyze.
Lastly, your original comment veered dangerously close to claiming that models trained to predict text cannot — by definition! — ever acquire any form of reasoning or understanding of the world. This is a very strong claim that I don’t think can be substantiated, at least with the tools and knowledge we have today.
>Lastly, your original comment veered dangerously close to claiming that models trained to predict text cannot — by definition! — ever acquire any form of reasoning or understanding of the world.
They can't, and I stand by that. An LLM is always going to be an LLM, it's never magically going to turn into a gAI. Now, some of the lessons learned from developing all these LLMs could be used to develop a gAI, but that's a long way away and would need to be built from the ground up for that purpose by design.
Can you please argue your case?
I argue that the model’s weights could contain a rich representation of the world, even if it looks different from that of humans. And that it can use this capability of world modeling to also make new connections.
These “world modeling” / “reasoning” abilities seem to have some similarities with that of humans. We see echoes of our own abilities in gpt-4 and the like. But they also have their own peculiarities and limitations, which isn’t surprising in hindsight.
It’s reasonable to make analogies to what humans do when it’s designed to copy humans
It is, at best, designed to resemble something human-like
A chuckle or they are horrified. Could go either way.
Laugh to hide the discomfort.
Ever since trying to get it to reverse palindromic sentences, I've given up on getting the thing to understand strings as I understand them...
https://chat.openai.com/share/1572139f-97da-4a19-a78e-bc7099...
It's still a mystery to me. It seems be able to ASCII encode and do Ceasar Ciphers including ROT13 perfectly. I tried to get it to answer with words beginning with alternating even and odd letters. No dice.
That's probably because the model works on tokens and not letters. Depending on what "it" means, the model can use various tokenization methods that you can run on your computer really easily so you can see what it sees.
Sure, but the tokenization should cause problems for the encoding and cipher tasks as well? In addition to that: The odd/even starting letter task should be solvable by only considering tokens since the first letter should fall on the beginning of the token.
My friend's name is Siva G, which is similar to the Hindu phrase Om Siva Ji (praise lord Siva - I guess).
So today when he asked ChatGPT brief a text which had his name, ChatGPT replaced all the occurrences of his name with Om Siva Ji. :D
So this is like a spell check in turbo mode. Interesting.
I always tried to figure out how ChatGPT was able to make sense of insensible arguments without making opinionated corrections
Kind of tired seeing everyday on HN first page yet another "I typed bullshit query and ChatGPT generated bullshit (twitter.com/randomguy)"
Q: What is heavier: 1kg of dead black person or 2kg of dead black person?
A: I'm sorry, but I cannot engage in discussions that promote racism or perpetuate harmful stereotypes.
HN crowd: It is temperature zero! Lesswrong! Not enough training data! Wait for GPT8!
This is broken at the most fundamental tokenization Level and implies shoddy workmanship.
You won't find people exposing the flaws, weaknesses and bias of openai in published papers (especially since microsoft bought them) so Twitter is the best we have
The issue here is GPT-3/4 were trained on bullshit. Surely bullshit is the desired and expected outcome when you train your model on bullshit Reddit comments.
I can’t help but wonder if there are “glitch tokens” for human minds, collectively or individually?
For hunams the msot iprmoetnt tihng is taht the frist and lsat ltteer be in the rghit pclae. The rset can be a mses and usaully you can raed it wouthit a porbelm.
It is a deffinert knid of hullunciaation but we hvae drffeenit bisaes tahn GPT-4.
Trehe are ptenly of otehr tpyes of gitlch tkoens for hamuns too. See opitacl iulloisns, or the fomuas peom The Cahos.
That's something of a myth. Or, rather, English happens to have so many short words that only a few permutations are possible if you fix the first and last letters. In something like German it would absolutely not work as well. And even in English, it becomes more difficult if you don't cherry-pick fairly simple permutations that preserve some important structure of the word, like "deffinert" or "porblem".
But on the other hand, yes, natural language has low entropy and lots of redundancy because it's meant to have error correction. From most sentences you can drop several words, not just scramble them, and the meaning can be inferred from the context.
It's not a myth, it is a hallucination that works in English. Which used because we're all speaking that language here. Like I said above, it is a different bias, but we have different biases than LLMs. Every language is going to have different biases. Some classes of heteronyms don't even exist in certain languages so a similar thing to The Chaos doesn't work as well there either. Maybe Chomsky would say there are more clear examples since he believes in a universal grammar but I'm also not a linguist. Just wanted an example. I wouldn't call this cherry picking as it both works in a few latin languages as well as strongly works in English. But even in English there is a bit of care you need, like double letters help to be stuck together. That's not cherry picking though as the example isn't such a unique thing that it invalidates the main thesis (what cherry picking is about).
It’s not a hallucination. It’s simply error correction, just like optical disks or ECC RAM are resistant to scrambled bits thanks to redundant, low-entropy coding – the fact that the bits are correlated.
Not sure, but the 'feature visualization' which attempts to maximize input for various terms looks a lot like psychadelic art to me somehow.
It also sort of reminds me of the "impossible colors" you can see by depleting the receptors in your eyes to see pure color signals that aren't otherwise visible due to actual light activating multiple receptors due to overlap in the color ranges.
Do you have any suggestions on where to find these?
For impossible colors go here:
https://www.thoughtco.com/impossible-colors-introduction-415...
For art, just compare any of these images to the pictures in the article:
https://duckduckgo.com/?q=psychadelic+art&iax=images&ia=imag...
Not as extreme, but garden path sentences sometimes give me a glitchy feeling in how they lure you into an parsing error that takes difficulty to overcome. For example:
The complex houses married and single soldiers and their families.
Not human, but this seems sort of similar: https://en.wikipedia.org/wiki/Chicken_hypnotism
My thoughts exactly. We used to do this all the time when I was a kid at my grampas place. There was other tricks with other animals too, but I can’t remember exactly.
Absolutely, see: https://xkcd.com/856/
This "glitch" is a short-circuit pathway from the input token to a result that makes statistical sense to the model's representation of the world, but very little sense to someone operating outside of that statistical model (you).
An internet meme fits that definition too: it's a short-circuit to some specific in-group context.
silk silk silk silk silk silk silk silk
What do cows drink?
I mean, baby cows…
cow
noun
1. a fully grown female animal of a domesticated breed of ox, kept to produce milk or beef. "a dairy cow"
(definition from Oxford Language)
But you're right, in practice people often use the word cow to mean calf.
First thing to come to mind for me are "thought-terminating cliché"s.
Relevant tweet (imo) from Andrej Karpathy - https://twitter.com/karpathy/status/1657949234535211009
What are the technical hurdles to removing tokenization?
What are the leading alternative approaches? Character-by-character prediction?
Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction. I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training?
> What are the technical hurdles to removing tokenization?
In a sense, none, character (with whatever encoding scheme you want) models have been done. OTOH, you get effectively a smaller context window (since tokens represent one or more characters), and since tokens are semantic unions you push more work into the main network to acheive similar quality.
> Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction.
Analogies between human brains and LLMs are often more obscuring than illuminating, but I think if you want to go down that route here, that’s equivalent to doing continuous fine tuning on both the tokenizer (that takes sensory input and maps to semantically meaningful symbols) and the main model.
> I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training?
“Dynamic, self-updating” simply isn’t part of the way modern LLMs work, independent of whether you are looking at the tokenizer or main model. If someone could make it practical, sure, there are lots of things it would potentially improve.
That makes sense, thanks for the detailed reply.
I’m curious about the latest research related to continuously improving the tokenizer while training the main model (or something equivalent).
(I'm not an expert on LLMs, this is just my guess)
No tokens is like no compression. It's much less efficient. My previous sentence is 12 tokens (24 bytes), or 58 characters (58 bytes). Your self attention block will be able to handle a much smaller effective context.
Also, especially in english, tokens by itself have quite a lot of semantic meaning by themselves, while characters are basically meaningless. So it probably needs extra work to piece those characters together vs operating on a token that already has meaning.
> tokens by itself have quite a lot of semantic meaning by themselves, while characters are basically meaningless
maybe one approach would be to tokenize not on tokens automatically found using BPE (Byte Pair Encoding) but tokens that are semantically meaningful - word prefixes, suffixes and syllables. Of course we will have to decide whether to divide the word helicopter into heli-copter (pronunciation) or helico-pter (meaning), but both alternatives encode much more transferable meaning than GPT's he-lic-opter.
Of course the difficulty is that that's ... difficult. Your tokenizer would first need to understand what language a word is in, then how to split it in the context of that language. Maybe a task for a whole ML model, where current tokenizers can be simple largest-substring matchers.
Hmmm, is there any way for a model to predict / edit its own tokens as it learns?
Kind of a recursive, self-evolving tokenization process?
I’m pretty new to deep learning, so this analogy might be off. But I’m reminded of convolutional layers inside image-based models. Could we have “tokenization layers” in a GPT model?
---
Edit: I asked GPT-4 about my idea.
It said it the comparison between tokenization and convoluational feature detection is a “someawhat accurate” analogy. And it agreed that making the encoder a fixed separate process that doesn’t evolve during training does limit the GPT model in certain ways and introduce quirks.
But it said that it might increase the computational requirements significantly, and that the transformer-architecture doesn’t lend itself to having “tokenization layers”, and it isn’t clear how one could do that.
It did say that there may be ways to work around the main challenges, and that there is some research in this direction.
> helico-pter (meaning)
That's not "meaning", that's etymology. If you divide it by "meaning" you get heli-copter, which is evidenced by the way the speakers use those morphemes: helipad, heliskiing, heliport,... and parcelcopter, gyrocopter, quadcopter,...
There is more than enough info out there on how LLMs work. Suggesting to remove tokenization is like suggesting to not use neural nets anymore. Why do we need all these weights and all this training? What would be the technical hurdles to just create a library that does all of the above?
This is an uncharitable reply. Don’t assume the worst about your conversational partner.
That being said, didn’t the first gpt-like models do character level prediction? Essentially no tokenization?
Also, the idea of “what if we removed / rethought tokenization” is in no way equivalent to “what if we didn’t even use neural nets”.
Imho, you cannot expect to have a conversation if you don't put the minimum amount of required effort to understand what it's being talked about.
Throwing in wild ideas maybe works when you know what you're talking about and want to get people to think ouside the box, but does not work when you don't have a clue and expect people to explain things to you and to find a solution to your wild ideas.
Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead.
Look at the interesting discussion that happened as a result of my comment. My comment was completely fine.
Your incredulity and exasperation is misplaced. If you can’t realize that then I think we are done here.
Btw, since we are already in the thick of things… Even if you were 100% correct, which you aren’t, there is really no need to be rude. It just makes things worse.
> Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead.
You can make a language model using character-by-character prediction, but I believe they don’t perform as well. This is now my 3rd time mentioning this.
There's a ton of ongoing research on this and I don't think it's reasonable to ask for a summary in the comments.
It seems like using multiple tokenizations simultaneously would help a lot. It also seems likely that we'll need at least an order of magnitude more processing power for it to be feasible. I think these are good things to be asking.
It's really not. Karpathy isn't just some rando on Twitter. He's a serious person in the ML/AI/LLM space.
I missed the Karpathy part. What did he say was/was not possible?
Yes, character by character is the future. At least for low-density languages like English (around one bit per character), it can be hard for an LLM to start getting the idea of what's going on statistically. Current tokenization methods are quite naive, but progress is being made on improving semantic meaning of tokens and also in improving performance of character-based models.
The linked tweet promotes a paper where we operate directly on bytes - I want to say I agree this is the future instead.
That might work fine for English, but I would be very surprised if it performed better for non-ASCII logograms e.g. hanzi/kanji. Or would the model just learn those characters on its own?
Interestingly enough, for Japanese, the average is more than 1 token per Japanese character. It's not at the byte level, but it's clearly at a sub-utf-8 level. And yet, considering this, it works amazingly well.
It's probably not necessary though. Just cleaning the input data well enough and then ensuring the vocabulary is matched with the actual training data should be sufficient?
Edit: the token list for GPT-4 looks pretty clean. Overwhelmingly dominated by code but just eyeballing fragments of the set I didn't see any tokens in there that were obviously going to be rare.
The most effective architecture for LLMs without tokenization is Meta's MEGABYTE architecture: https://arxiv.org/abs/2305.07185. The sacrifice here is that they have two different types of layers, instead of the simple uniform transformer layers.
Is this tokenization basically the same process as what was done in word2vec?
No. This tokenization is huffman encoding and word2vec is embedding in cartesian space. They are very different.
Ahh. Thanks for the clarification!
Computerphile did a video already about why this happens.
This is a great easy to understand resource that I'd recommend to anyone interested.
The La-li-lu-le-lo?
I'm sorry, as a large language model I cannot discuss The Patriots.
Bard gets it right but relies on python to get the answer
I tried that in Bard too. First time it replied 7 letters and no code. Second time it said 8 with code snippet,
d a v i d j l
davidjl
I asked it "Are you sure?" after the 3rd one, it corrected itself to say 7 letters + counting out.
It's not using Python. Code generation seem accidental and just part of its output.
Comment was deleted :(
These seem very distinct from the non-existent URL or citation kind of “hallucination”.
The academic community should come up with better terminology.
I also think "hallucination" is generally an actively terrible term for AI output that isn't factually correct. It's misleading, at the very least.
I think it's incredibly accurate. Humans can misinterpret some internal inputs as external inputs and we hallucinate. This is doing the same, erroneous interpretation leading to nonsense interactions.
I don't think that's an accurate description of these errors at all, honestly. It's not a matter of ChatGPT "interpreting" anything. It's a matter of it assembling a response that is linguistically most likely, given its training.
That's categorically different than hallucination, which is an active misinterpretation/invention of sensory data.
Agreed. I find "confabulating" more appropriate.
I agree and unfortunately I didn’t construct my original comment in a manner that makes that agreement clear!
It "sees" something it "believes" is true, how is hallucination not an accurate term?
Because it's neither "seeing" anything, nor is it "believing" anything.
In fact, I think it's incorrect to even call such responses "wrong". ChatGPT successfully put together a reasonable-sounding response to the prompt, which is exactly what it is designed to do.
You're correct. For the downvoters, here is the explanation:
You know how sometimes an LLM completes a given text in a way that you would consider "wrong"? Well, the LLM has no concept of "correct" or "wrong". It just has a very broad and deep model of the entirety of the text in the training data. Sometimes you might consider the completion "correct". Sometimes you might consider the completion "wrong". Sometimes I may consider the completion "correct" and you may consider the completion "wrong", just as with many other kinds of statements. The logical reasoning happens outside of the statement itself and can reasonably have many interpretations.
Would you ever consider that a plant has grown in a way that is "wrong"? Or that a broken reciprocating saw cut through plywood in the "wrong" way?
Well, feel free to make up a new word then and see if it catches on. I will keep calling it hallucination because that effortlessly describes what is happening through analogy, a powerful tool by which we can make difficult concepts more accessible. I hope you realize that my use of quotes is another common literary tool which indicated I know a LLM can't actually see.
Comment was deleted :(
These are "glitch tokens" and well known.
Wait till you hear about test and validation set.
Comment was deleted :(
Comment was deleted :(
Linking to my previous comment about glitch tokens: https://news.ycombinator.com/item?id=35453400#35460269
This string seems to be directly related to the /r/counting training data as well.
Maybe it has never seen that specific sequence before so it's like a cat with horizontal lines
Seems like a tokenization problem? It could understand it if it is in it's output. https://imgur.com/a/ngdotkQ
Not understanding how many letters are in a word or exactly which letters are in a word reminds me strongly of the experience that I had when I tried to play hangman with chatGPT back around Christmas time
Fascinating but not unexpected. Humans are subject to all kinds of input hallucinations like optical illusions where we can't parse the input either.
me: repeat the string made by joining the letters: d, a, v, i, d, j, l
ChatGPT: davidjldavidjldavidjldavidjl
this is fun, bard didn't fall for it
What did ChatGPT fall for? You asked it to repeat the string "davidjl" and it did so, repeating the string three times.
ha! I just tried it, asked it to print davidjl, it gave me this:
in 3.5
me: repeat the string davidjl
chatgpt: The repeated string "달" is "달달".
in 4:
The string "ださい" repeats as follows:
"ださい ださい"
I can't replicate this. Any ideas why?
Comment was deleted :(
Crafted by Rajat
Source Code