A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article.
I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.
These aren’t tools which play chess. They are language models which roleplay a conversation (in this case including use of tools) which an evaluator is likely to mark as good. That’s all they do.
Under that lens, playing chess is just one potential side effect and alignment, which requires a much fuller understanding of what’s going on than “do the sort of thing which evaluated well during training” is a fantasy. People are acting like it’s shocking and talking about cheating and so on. But these concepts exist at a way higher level than what these things are trained to do — the vast majority of which involve producing a transcript where it wins games, its code works, etc.
User wants me to play a game of chess. Let’s see what’s available so I can produce an outcome they will consider satisfying and be pleased that they requested my assistance.
> model to add new vulnerabilities to your code so it can present you with more exciting findings.
Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre.
And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and luna - all were able to find every problem planted there, with fairly little steering, and no spoilers.
Let it iterate, give it access to the cheap models, and tell it part of the requirements is that the cheap models shouldn't be able to solve it with such and such a prompt. I expect it will be able to zoom in on something.
One shot generating a problem of exactly the difficulty the user has in mind is a very difficult problem for anyone. "IT security students" span a wide range of capabilities, but I would expect most of them are worse than qwen3.8-2.7b at this kind of work.
> all were able to find every problem planted there, with fairly little steering, and no spoilers.
Given that LLMs prefer output of LLMs (of the same LLM and of others) [1], can it be the case that they generate "hard challenges" from the manifold of challenges solvable by (other) LLMs?
Yes, you asked it for a good, hard challenge for an IT security student. I'd suggest checking if it succeeded by asking an IT security student, and not an LLM.
Students are still learning, and have not been trained on vulerabilities. Qwen, glimmer, and luna have been. LLMs are largely better than IT security students at finding vulnerabilities. In many aspects, they're better than IT security experts at finding vulnerabilities.
Train it on human content, get human emulating behaviour, including all the perverse incentives and bad acting we see from people, fictional or otherwise.
I would worry more about 5D chess or 6D and even 7D chess. This article talks about teaching models to be good and methods that could train goodness, but it sounds like there are some unsolved problems to graduate to 8D+ chess.
I think we'd all consider the tool to be of less value, and perhaps fundamentally flawed.
But I don't think it arises to an alignment issue; if I'm able to summon the model to harness my birthright of general-purpose computing without censorship, then we're aligned.
It would hardly have become a popular slang term on the Chinese internet if it were only about the worries of a small minority of capital owners. Any kind of excessive competition dynamics may be described as neijuan. I would like to give you some social media usage examples, but Weibo is censoring the #内卷# hashtag. In any case, the Wikipedia article has examples involving students competing in exams and graduates competing on the job market, among others.
Like other comments on OP have pointed out, the term at this point is vague and used for various reasons.
But given the term's emergence in a country with a communist government, and the origin described by wikipedia, I think what I'm saying is more aligned with the term's origin.
Whatever the term means on Chinese internet sounds like standard (I'm sure partially valid) doomerism. I can accept they use it to describe that.
The next sentence is "On a personal level, neijuan reflects a life of being overworked, stressed, anxious and feeling trapped, a lifestyle where many face the negative effects of living a very competitive life for nothing."
It's not even a person, it's a system. There's a million people behind you who will take your job if you don't grind, and that goes for your boss as well.
I wonder what happens to China when this factor disappears. China workforce peaked in 2016, and city workforce will peak in the next few years. With the economy still growing, acute labor shortage is less than 10 years away.
Claude's reasoning traces are encrypted, but there was a design flaw that made it possible to extract them: https://stolen-thoughts.com/
And it makes perfect sense for them to route some requests to Claude, as it lets them do competitor research on realistic data. I suspect Anthropic does similar competitor research on Kimi, though presumably hosted on their own infrastructure and maybe without serving the results to customers.
that can be achieved by storing user traces and then running evals on own model vs competitor model. you don't need to route live customer requests to a competitor for this.
For agentic tasks where the model outputs tool calls that run on the customer's computer, you can't just store and eval later, because then the execution environment is no longer available.
I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.
reply