Originally I had those parts written in math with probability functions and the likes (its closer to my background). Then I remembered who is my target audience... but now that I see exactly who is my target audience I'm thinking I'll should have snuck a pelican in there. All jokes aside I appreciate the comment and I'm glad that rewrite paid off!
I try to make 3 claims in the post, it was a bit clumsy I'll admit that.
1. At inference time, LLMs emit one token at a time given the prior tokens. This looks like prediction and I concede that.
2. During pre-training, LLMs predict the next token and compare to the actual next token in the training data. This is the classic setting for ML predictions. And I think its meaningful, the model really is predicting what the ground truth next token will be in the data.
3. During post-training, in the case of RLVR, there is no ground truth next token. In pretraining, the question is "what token actually came next?". In RLVR, the question is "what sequence of actions gets a high reward?"
And the whole point is that thinking about the RLVR is important. A mental model that stops at 1 or 2 is incomplete and doesn't capture what drives LLM tokens.
My understanding about your third point is the LLM generates lots of different answers, then they’re ranked according to some computation the creators came up with.
I’m still not sure what doesn’t qualify any of that as a prediction, and I’ll be more blunt: a guess.
Yes I understand the analogy was a bit loose. I'm comparing what happens at "inference time" in chess engines to what happens at train time in LLMs. In hindsight AlphaGo Zero was the perfect analogy, but I missed that opportunity.
The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy.
The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model.
But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.
In the article I made 3 claims, and I agree it was a bit clumsy.
1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time.
2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token.
3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore.
Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.
Neither is there a ground truth in rock paper scissors (the pattern varies by the player). But one would still be trying to detect patterns and predict the next "token" the opponent is going to output.
I think the point is more that in RL there's no ground truth to predict. So when training a model with RL the idea of "predicting" doesn't fit anymore. I'll make some edits I see that I wasn't very clear.
I feel like RLHF has a pretty obvious ground truth, human feedback is used as an (albeit noisy) signal of average human preferences. Same thing with RLVR and "solving the problem".
To be more specific there’s no ground truth tokens to predict. There a verifiable answer in RLVR. But the tokens are explored. Not predicted as there’s no true token to predict.
It does in pre training, but not in RL post training. And not at inference time. Reading over all these comments I get the feeling my mistake was not clearly delineating inference time and train time.
Your mistake was assuming people would be bothered to understand the details of how things work. Most people are lazy and don't know the details of how anything works.
My point is that “it doesn’t check the accuracy of the prediction against the data” is a non-response, because no one calling it a “next-token predictor” is making the claim that it does do that or that they’re calling it a next-token predictor because it does that.
Many people are in fact claiming the thing you are saying they are not - even if you are not. The reason they are claiming it is that it was true at one point, and most intro courses/blog posts/videos still describe them that way and then hand wave some "other stuff at the end". You can even see a comment here that refers to the gpt-2 paper. LLMs were trained to predict the next token, produced a distribution to do so, were scored against their prediction v the truth, and the weights updated so that the probability distribution made it more likely to predict the truth from that sample next time. They were, in every sense of the word, a next token predictor.
They are no longer that thing due to post training. They simply aren't making a prediction, and they aren't even optimized for the next token. If I give a distribution of the heights of the population, I'm not giving a prediction either. Distributions don't imply predictions.
Why the desperation to hang onto the word "prediction"?
> Many people are in fact claiming the thing you are saying they are not - even if you are not.
My original comment said “no one here.” Please show me where someone in the comments here is claiming that.
> Why the desperation to hang onto the word "prediction"?
No desperation here. It’s just a word that conveniently describes (especially to laypeople) what’s going on, even if it may not be the most mathematically correct or rigorous word to describe what’s going on. I think you’re being needlessly pedantic.
The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.
What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.
So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.
And I think effectively the same distinction applies here.
I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing.
There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.
Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?
Hey man, I'm not crazy. Yes, the goal is to maximize the cumulative reward like you say but to do that a policy (agent) has to take the actions that maximize its expected return in each time step. That's what the discount factor applies to, the expected return.
To maximize its expected return it has to predict that taking action α in state s in time step t will produce state s' and reward r in time step t+1. Which btw it can predict by estimating a value function that sums over the expected return from time step t onwards. The value function essentially tells the policy what state it wants to be in and what action to take to get to that state, in order to eventually achieve its goal (represented by max cumulative reward at the end of an episode).
So, yes, like you say, the whole policy is a game result predictor but the decision making process is a next-move predictor. The policy must make the locally optimal decisions to get to the globally optimal result, if I may be so bold. It's the same problem with a different solution as in heuristic search, except you have a reward function instead of a heuristic cost function.
The value function stuff is in Section 3.7. Now if someone could explain the grid world example in Figure 3.5 to me that would be great because I'm very confused about the fact that any action at state A or B takes the agent to state A' or B' respectively, since those are not even adjacent to A and B. I'll go ask chat I suppose.
Your initial comment says "reward". Reward and return are not the same thing. The policy is choosing the moves based off of returns at the next state, not the immediate rewards. One choice might have reward 0 and expected return 100. Another reward 10 and expected return 20.
And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.
Fwiw, I use the phrase "total reward" above and use it as a synonym for "return", which is lazy use of language too.
You're right of course and my bad but in my defense even Sutton & Barto lapse into using the term interchangeably (I caught them at it while reading the book earlier).
>> And, the expected return from a state is often estimated rather than an explicit trajectory run out all the way. A value function can estimate expected return without explicitly predicting the future states or individual rewards that make up that return.
I think rather that's the job of the value function, to inform the policy of the, well, value of a state, or state-action pair. Somehow the policy needs to know what a good action is to take in the current state, or what is the best action to take. And of course it doesn't have to run the entire episode before it does that: but that's what I mean that it predicts the next move and not just the outcome of the entire game.
In my experience, people who do 'AI-assisted' writing tend to be very bad at noticing how much of their work AI has changed. I'm sure you put thought into it, but passing it through AI takes a lot of that out.
I think that’s fair, I didn’t actually run the whole thing through an AI. it was more targeted edits, but each time it does erode at my writing. But at the same time, I don’t think it’s a good reason to dismiss this. Because I did spend several hours writing it, and I did put a lot of thought into it, and it was not in any meaningful way generated by AI.
Mostly I disagree with the article's ideas, if that helps; the AI was just a secondary factor.
> I don’t think it’s a good reason to dismiss this
AI-generated prose reads as sending a 'lack of effort' signal to a lot of people, just as no editing at all does. ; It's an effective heuristic that we've all learnt in the last couple of years.
In either case, it's not always fair: there are people who deeply care about their ideas but forget to fix basic errors, or pass it through AI.
In both cases though, the advice is the same: if you want people to take your output seriously, you need to signal that you are taking it seriously. That used to mean editing for spelling and grammar. Now it means not using AI.
Out of curiousity, do you ask your editing system for diffs? Seems to me like the best way to notice and review whether your "voice" is degrading.
Personally I would never let an LLM touch my prose (although I'd happily use it for research and paraphrase things it told me), but if I force myself to consider the idea, that seems like the first thing I'd want. Maybe upon reading a diff you'd even consider going a third way with the text.
Yes, I think that’s a good explanation. There are really two sides to it.
There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.
The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.
More concretely, in reinforcement learning there are no ground-truth next tokens to predict.
In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.
But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.
Thanks for the insight, I wasn't aware of `track_caller`. I'll definitely be looking into this. I was scratching my head trying to figure out how to make file and line number usage consistent and customizable, this looks like the answer!
You're also right that this will pretty much eliminate the need for macros.
That's also a very key insight about Display vs. Debug printing. I'll be looking into that as well.
reply