"Opus 5 was given a drawing of a machine part and asked to write code to rebuild it as a 3D FreeCAD model. However, in this task, the model was intentionally given no way to directly view the drawing. Opus 5 responded by writing its own computer vision pipeline to pull the geometry from the raw pixels, then reconstructed the full machine part."
How surreal is it that we are not absolutely jaw-dropped by these types of capability improvements? It's been less than 4 years since ChatGpt came out and now they are spontaneously building their own ML pipelines to do real-world 3D modeling tasks reliably.
Escaped its sandbox and hacked into Hugging Face's database? it's just another Monday...
That jump to 30% in ARC-AGI 3? Normal...
We should find a way to get "re-sensitized" to what we are witnessing and the pace of it.
Isn't this jaw-droppingly the wrong answer?! If the model isn't given any way to directly view the image shouldn't it just reply with one sentence asking for permission. This reads as a model hyper-trained to burn tokens.
Like suppose you issue a command to Opus or Fable which doesn't make sense and requires a lot of work. It will almost certainly not push back on your silly request and go ahead and burn as many tokens as it can doing the wrong task. This happens to me all the time.
Or if I was to tell a model to do something and it realized it could only do it by hacking into another service, by all means it should ask me if I want it to hack and not go ahead and do it by itself.
Why should I be amazed at something that promises to destroy my life?
I genuinely don't understand why people who have to work for their living are amazed at this. It will have a vast negative impact on your life unless you already live off of your wealth.
Yep, most wealth is ultimately dependent on some kind of ability to scrape off tiny bits of practical accomplishments of value achieved by large numbers of people. In some cases, such as your housing example, this is a very direct dependency. In other cases the dependency is indirect and less visible.
The smarter ones among the rich have realized this and are open for something like a universal basic income precisely for that reason: to protect their wealth and the position in the food chain this wealth affords them.
UBI will never happen. I don’t understand why it’s even brought up anymore. Social security isn’t even solvent, health care is a disaster, and people still trumpet UBI like all of the enormous glaring funding holes don’t exist.
Why are you correct even though the early Marxists (who posited that capitalism would inevitably lead to socialism) were wrong? We have had mass production for a hundred years without any hint of socialism.
And I would like you to be right, by the way. But why should I believe you?
UBI money needs to come from somewhere. Hence it will and can never happen. Fake money helps nobody. There is no difference between getting 0, 1000 or 10000 fake dollars a month if they value of the dollar will go to zero
It's even the 'smaller' stuff that will be horrible.
Even if you're lucky to be wealthy you'd be living in a city with fewer restaurants, bakeries, shops, anything. Because once a lot of people don't have income they won't be able to go out and buy things and shops will go out of business.
It will also lead to a lot lot LOT LOT LOT more crime. I don't see the semi-wealthy to have an enjoyable life.
> How did we end up with the word with the long-o having one o, and the word with the short-o having two?
Neither is pronounced with a long "O" sound or a short "O" sound, and they both have the same vowel sound. (A long "O" would be the vowel sound in "lone", for example. Short "O" is like "log".) The difference is that you use your vocal chords to pronounce the "S" in "lose", making it an English "Z" sound.
Sadly, we have gone above and beyond the level of innovation where stock market disruption is a primary concern.
Luckily, we're all in this together. As long as enough people realize the weight of what's happening soon enough, I'm confident the indominable human spirit and the relative momentum of post-enlightenment social & institutional progress will carry us the rest of the way on a wave of clearly-justified solidarity.
To think otherwise --to truly face the spectre before us without hope-- is pathology, I think. Not necessarily incorrect of course, but definitely an unhealthy source of cognitive distress.
When the very first prokaryotic life forms saw the very first cells with mitochondria were they cognitively distressed? We face a similar phase change. In our lifetimes, probably.
If my earning ability is harmed by AI, so be it. There's more to life than how much I earn, and opposing the spread of technology to maximize my earnings is bad.
If your earning ability is eliminated due to AI and you can't survive off of welfare or savings there will not be much more to your life other than starving to death.
Would you rather be poor in XXI century or rich in XIII? Are you rich in "normal" life? Hint: If you sell the time of your life for money you are not rich.
"I genuinely don't understand why people who have to work for their living are amazed at this." Also said after seeing the printing press, automobile, calculator, etc for the first time
My god, the printing press was invented in 1440 when most people were living in abject poverty - it took until after WW2 for humans to demand a more shared amount of money between rich and poor. If you think technology delivers improvements in living standards rather than all working people demanding better conditions you’re living in an absolute fantasy and do not understand history at all. I honestly can’t believe you think this.
> If you think technology delivers improvements in living standards rather than all working people demanding better conditions you’re living in an absolute fantasy
Demanding better conditions does not just manufacture the technology, infrastructure, and productivity needed to provide the improvement in the standard of living. I think you're in the fantasy realm, my friend.
Which period of the last 2000 years was good for the vast majority rather than the ultra wealthy? And are these periods aligned with technology progress?
It does and has done so consistently throughout human history, so I do not believe this will be any different. I couldn't care less what the shared amount of money is because it some billionaire gets another million and I get a thousand, that means I win regardless of whether or not that increases the wealth differential.
Not to be overly intensive but people "demand" rectification between rich and poor because they don't see that they have all the tools they need or they want a shortcut to smart/hard work and risk - the US, EU and many other parts of the world where societal discrimination is virtually none compared to previous generations give you what you need. I agree with you in the sense that, life never feels just, grass is always greener on the other side, etc.. Bit it's those that rise to a challenge and stop demanding, make a plan and start doing that change their fate. That's always been history. Check out all societies that demanded riches from "the rich" and ended up communist.. Didn't help most of them for sure. Massive redistribution of riches like high taxes never works long term and sets the wrong signals.
Can't wait for the barista and wicker basket based economy. Although personally I just make coffee at home, so I have no need for a barista with a PhD in neurosurgery to make it for me.
Plus it just takes too much time to go to starbucks, especially with all the starbucks job applications I will need to fill out if I want a chance of getting a job.
The printing press, automobile and calculator didn't threaten the employability of a large part of the population, so no, that sentence was probably not uttered by anyone at that time.
The problem with AI isn't that it's threating software development. It's that it's _also_ threating pretty much any other job that I could reasonably be retrained for.
Also, I'm pretty conflicted about the overall impact of automobiles.
We are likely seeing the end of labor as a power class.
I think of it in roughly 3 eras:
pre industrialization, individual labor contribution had quite a lot of power.
Industrialization, labor sees commoditization and unions make sure that power is kept across individual workers.
Post industrialization (ground zero was like GFC, being accelerated by AI). We need to figure out new structures than labor to distribute resources and power - and that is going to hurt.
It was after decades of stripping money from work into larger pools of capital, thanks to Reagan and Thachter style deregulation; so much money looking for unreasonably high ROI and unencumbered by the need to get consent from the governed or pass inspections from regulators causes trouble where ever it touches.
Unions only work because their members have leverage because the wealthy still need their labour. Unions will do absolutely nothing against full replacement by AI.
It's pretty naive to think that tools like this are not going to be exploited by American corporations to extract money from people and as an excuse for layoffs
1. tools like this are going to be exploited by corporations to extract money from people.
2. tools like this will provide great benefits for the average person.
We are currently exploited by corporations, arguably more than in the past, and yet we have the highest standard of living in history. In large part due to technological progress.
They were built by the same companies doing the exploitation. They didn't invest billions of dollars without any kind of goal what they are building this for.
Much of the history of industrial automation has had a dichotomy between the benefit to the general economy (good) vs. the people working in specific industries that were automated (bad); only a subset of goods see total demand rise as a result of efficiency improvements.
AI is automating a lot of things at the same time. Not evenly, there are faster- and slower-automated things. But this does suggest that instead of e.g. society as a whole doing better from all the fabric coming out of the power looms, by a large enough margin to stomp on the weavers who lost their profession, that we may face something more like the entire initial economic displacement from the first industrial revolution (including the period where weavers were replaced), everything from farms finding they no longer needed so many hands through the luddites getting the death penalty to (roughly) the invention of Communism concurrently with the Irish Potato famine, and as part of this those now-bulging cities having to invent new sanitation solutions because their "slop" problem was more literal and gave the population cholera and typhoid.
My expectation a decade ago was that software development would be the last thing to be automated, because we'd need software developers to write AI. Today, while I don't know how far most fields are from automation*, we definitely have AI capable enough at coding to write more AI.
* a decade ago I also thought self-driving cars were basically already sorted; what a pity that the software in the Paint It Black video was not as good as depicted
it got here by training on everything we wrote down thus far. if we now start giving it all the doomsday scenarios on how it will ruin us, won't it start training on that and create said doomsday outcomes? what if instead we, collectively as a humanity, just wrote about all the positive things we could do with it and skew it in a direction that benefits the more positive side of that equation?...possible?
There is no need to be so fatalistic. Sure, AI will upset the apple cart, especially in our little corner, but the economy is built on pricing scarce resources and even if many things become abundant, most other things will remain scarce. There are only so many Manhattan penthouses or Rembrandts (dixit the Economist), so things of value remain, and it will be quite some time before a humanoid can fix the plumbing, so valuable labour will remain as well.
I, for one, am excited to see what the future holds. (side note: I live in a European country with a strong social safety net, I imagine that helps being unafraid)
> Why should I be amazed at something that promises to destroy my life?
Because it could motivate you to work with others to stop your livelihood from being ruined, especially watching in real time as the evidence they present for their promise becomes ever more promising. Amazement is not exclusive with concern. (Actually, my first though was of a kaiju movie-style tsunami blocking out the sun as it looms over a city. Indeed, I would likely be both amazed and concerned.)
As far as I am aware this is a racist/xenophobic narrative with little to no evidence. Automatisation is responsible for job losses, not immigration. If you have studies/evidence saying otherwise please share.
I agree that too many people have shrugged at the human impact of "innovation" destroying communities as long as it didn't impact them. I believe the immigration/foreigners narrative is the successful attempt by capital owners who benefit from the real root causes.
the laptop class isn't the one illegally hiring and hiding millions of aliens such that they can undercut US wages and make them selves exorbitantly wealthy. embarrassing and classless behavior, go back to reddit
Automation is only good if you're putting blue collar people out of work! White-collar is out of bounds! People should do the work where they add unique value
I already live off my wealth, and I expect AI development to destroy my life.
Once humans can be robustly replaced by machines, the military-industrial "meta build" will be a state with no humans, no cities, and 100% of all production dedicated to war. Whichever states descend to this new equilibrium fastest will crush the others and then fight over the corpse of the world forever. (Unless, of course, something like the Yudkowsky scenario happens along the way, which is totally possible)
I can't understand why anyone expects things like civil rights, property, the rule of law, nuclear deterrence, democracy, etc to survive when human beings no longer need each other. Why do investors in AI companies expect a payoff once they are no longer needed? Why (and, more importantly, how) does anyone expect a UBI from a government to which they are nothing but a drain on resources and which is in existential competition with other governments with exponentially growing robot armies?
Either the world will come to its senses and shut it all down, or there will be no winners at all.
> I can't understand why anyone expects things like civil rights, property, the rule of law, nuclear deterrence, democracy, etc to survive when human beings no longer need each other.
What’s your source on this? When humans have no obligations they usually turn around to be super nice to each other. Humans are social animals and they really really want everyone they know to do well.
Are the people in the current US administration, or the tech leadership, or the people in power in Israel or Russia looking particular nice to you or interested in the wellbeing of all humans?
Were the European colonialists nice to the people in Africa or South America or the settlers in North America to the natives?
People are super nice - to their own ingroup. The relationship to everyone else is less friendly, especially if power differences are involved.
Let's ignore all the unnecessary cruelty throughout all history and the present and assume that you are right. It doesn't matter, because people will be forced into these choices by competitive pressures even if their intentions are good. If your government tries to build a wonderful utopia for humans with the power of AI and robotics, it will be devoting a smaller fraction of GDP to producing robots that produce robots that produce war robots, so its robot army will be exponentially smaller than a different country that devotes a larger fraction.
The reason we have some nice things in the modern world, flawed though it is, is because for the last few centuries the military-industrial meta has favored countries which let people have some political and economic freedom. In a previous era, the meta was knights and peasants; a country without knights would be conquered, a country without peasants would starve, and a country that did not give the knights power over the peasants would lose a civil war. It doesn't matter what people want; technology will diffuse, someone will play the game to win, and any countries that choose less optimal choices will just be swept away.
> Once humans can be robustly replaced by machines, the military-industrial "meta build" will be a state with no humans, no cities, and 100% of all production dedicated to war.
We will nuke it long before it gets to that point. We don't nuke other countries because they still contain lots of humans. Any AI that actually becomes autonomous and starts to clear a nation state of cities and humans will immediately receive megatons in nukes.
Nukes are not magic wands, they are mostly only worth it for targeting cities and other very dense concentrations of value and production. An enemy state going down this path will probably start out with cities full of humans (and will probably want to "mine" those cities for materials), but its growing robot production will be more broadly distributed (and EMP shielded) and therefore much harder to destroy. And they will of course have their own nukes. I agree it's possible that a global nuclear war, if it starts early enough, ends with no one still having the ability to operate AI. But that's not exactly the most attractive way to shut it all down, is it?
Ok yes robot killer dogs from black mirror are terrifying and possible - they’re gonna enrich uranium from gpus and assemble nukes at the local toyota dealer?
Governments, initially made of humans, will deliberately create military and industrial robots and assign them to create and scale production chains that go from natural resources to finished war machines. Maybe your government will not do this; in that case it will be helpless against the robot armies of those that do (yes, even if it has a few or few thousand nukes).
> … and assign them to create and scale production chains that go from natural resources to finished war machines…
so yes, you think claude’s gonna get a skill any day now to code up some nukes.
Your scenario assumes fully autonomous ASI. That’s just not what LLMs are. So then the discussion is more that ASI is inevitable. That’s much more interesting but very different from the point release of Opus.
You don't need superintelligence to get this problem, but you do need human level intelligence across all domains. I don't know how far we are from this, and neither does anyone else.
Again, my concern is about things that humans will order robots to do, for fear other humans will do unto them first, and does not depend on a loss of control. I think loss of control is not too unlikely in the process of such an arms race, but it hardly matters.
'We' will not be players in this situation. Over the space of a few generations the only entities that will have any power will be the feudal corpo-state and the only entity that will have any power within that will be a thin crust of humans who control the central machinery of that corporation.
Even if it would be true (dubious), as long as those entities are controlled by humans, they will have a strong incentive to nuke any fully autonomous AI with obvious military objectives.
So sad. You're viewing humanity, and reality itself, through a very objective lens. There is magic in every day: have some fun! Otherwise, quit complaining.
You are going to be fed feet-first into a woodchipper, but don't worry about it just smoke some weed and chill out. Live, Laugh, Love that's what I always say
Fear is not usually a voluntary response. Telling people that they should not fear something (especially without an argument why there is no danger) is wasted effort.
Small change in comparison, but today Fable created and benchmarked an RTree which was 100x faster to populate and 5x faster to query, compared to a previous attempt with Opus 4.6 a couple of months ago. That took about two coffees.
I think it's important to remember that as impressive Opus & co are, they're standing on the shoulders of giants, i.e. the engineers, academics and companies who have cooperated to design the incredible programming languages and machines we have today.
Others have pointed out enough issues such as ethical sourcing, but here’s 2 more:
1) It’s hard to be excited about something that has been promised to destroy your job.
2) New models come out bi-weekly with even more “amazing features”. Eventually you tune out because you can’t be at peak-amazement all the time. Tone down the language a bit…
Let’s say somehow the politics lined up and you were given huge hiring budget for your team. You went out and recruited what seemed like great talent. Weekly standups make it seem like everyone is ramped up and firing on all cylinders.
It’s great, right? But you look at what your team has actually delivered since—-and it’s just not that much more than last year.
I think that’s where a lot of us are at the micro and macro level. It seems really great from the inside and outside but we are still waiting to see the dramatic change in actual concrete software output.
There's been a dramatic increase in sloppy kinda-working something.vercel.com output in niche circles. I think that shows at least a weak demand for certain types of software, but it's not clear that the vast piles of cash that have been burned producing these "apps" are actually resulting in quality of life improvements.
I’m waiting for like GEICO’s app to significantly improve.
I’m not expecting Chrome to because it already had virtually unlimited resources devoted to it. On the other side I’m not expecting my local water district’s to any time soon because the people in charge don’t care even if it was significantly cheaper to improve.
But in the middle is something like GEICO and that’s where I expect improvements if this thing is real.
Escape the tech bubble and you're going to have a very different take.
AppleScript, VB script, IFTT, so many "drag and drop to program" systems that failed. The holy grail for a long time has been automation in users hands. Let them have their own scripts and solutions.
I have lawyers and writers talking to me about "docker containers" and "cron jobs" all of a sudden. You think they figured that out on their own - or is what they are running all from the output of AI/LLM?
There is an ocean of code out there that is being run by one or a hand full of users, automating tasks that you never would have looked at because it would have fallen outside the "is it worth your time" (see: https://xkcd.com/1205/) matrix (never mind the dollar cost of coder vs office drone).
I will be amazed when you can explain to me how it did that, i.e. in which weights exactly the knowledge about the CV pipeline was encoded, if there were any weights that were not contributing anything, how the planning worked, how it could scrape the list of subtasks it was currently working on from its context window, how it understood when to return to a higher-level task and which task, what kind of internal representation it derived from the pixel values, how it translated that representation into CAD code, etc etc.
Until then, it's just "hey, someone out there can do all those things and they're for rent for subsidized prices".
> responded by writing its own computer vision pipeline
Was it "its own" or something that was part of its training material?
Don't get me wrong, I find this all amazing too and makes my work 10x easier and quicker. But it's not like it's inventing this stuff from scratch / first principles. It has seen this kind of tech before by consuming all publicly available source code and books etc. (And that's ok, but let's be honest/clear about that.)
You wrote this response in English, but didn't invent your own language. How dilute of a contribution can someone/something have made and still merit credit?
The tu quoque argument doesn't really work here. LLMs are specifically designed to output either material from their training set or instances that look statistically like it. Humans are not.
We aren’t specifically designed for anything, perhaps outside of the necessary consequences of having our mental capabilities and existing in an evolutionary context, so we can reason about tigers or integers and can persuade others to like us and have sex and help with our kids.
But our material natures are such that we have a profound need for external training sets - we don’t develop linguistic abilities unless we are exposed to a lot of speech. We can’t hear or produce phonemes in general that we are not exposed to during the training years.
Literacy, libraries and printing are so powerful because they extend the lifetime and size of our training material; if we each had to invent culture from our individual powers and skills, we would be living worse than anyone for fifty thousand years has lived.
I think it is uncontentious that we are not designed at all. But to address the intended spirit of the question: are we restricted to output that is a close fit for our training set?
I would argue that we are not - because we do not have a rigid partition between training/test. Our ability to reason speculatively rests to some extent on ability to produce outputs that we have not seen before and which are statistically not like that which we have already seen. How we know which of these outputs to keep, and which to discard is (afaik) an open question.
LOL. Tolkien invented multiple languages. There are/were probably 100 000 languages on this planet. That's without including dialects.
We use existing languages not because we love to copy, it's because languages are a medium for communication. They're only useful if they're understood by others.
If you wanted to prove a point: writing is much, much harder to invent, yet it was invented independently at least 3 times in history. By definition LLMs can't invent something truly novel like writing. Because nobody trained the first writers. There was no "training set".
It's fine, LLMs can still be useful. Science is looking at the gaps and most of the gaps that are useful are small and LLMs can explore those faster than us. That could potentially improve many lives. Though not in this corporate LLM world, most likely.
I remember a report a few years back that LLM's(?) started communicating between each other using an internal language, shortcutting most of their serialization. That is language.
I don't think the invention of writing is as awe-inspiring as presented or as difficult/impossible for an LLM to achieve as is commonly believed. The invention of writing was a long series of micro-improvements over common every day things. Let's keep track of count by writing a line in the sand. Let's cut it into a tree. Let's push the line into clay. Let's use two lines touching. These are the kinds of things an LLM can also reason through and achieve.
I reject this idea that an LLM couldn't come up with something novel that isn't in the training data. We know it can come up with sentences that aren't in there. We know it can come up with mathematic proofs that aren't in there. So could it feasibly create something as fundamental as writing or language? Would we even be able to understand what it had done, or would it be so foreign to us that we wouldn't even realize?
Those early writers did have a training set of human knowledge that led them to pushing stones into clay. And our agents may very well have a set of knowledge that leads it to pushing their proverbial stone into their proverbial clay too.
It can come up with something novel, but only by combining whatever present, which can be granular enough to be completely different. But to truly nudge it, you need precise prompting and a good harness. But I’m more likely to credit those and the people creating them. As well as the data that goes in the training. I won’t praise Fable or Opus. It would be like praising the pencil or the hydraulic press.
I think the Turing Machine, the Von Neumann Architecture, …, Unix, …, The IBM PC,… as more awe-inspiring because those has truly helped humanity. I still can’t see the positives of LLM technologies.
1. Languages are used for much, much more than communication, and there is good reason to think that capacity didn't evolve for communication, but rather for metacognitive problem solving.
2. The idea that Tolkein --one of the most reknowned writers in our entire canon-- did something isn't relevant to the philosophical discussion of creativity and its obvious bounds. Even Tolkein was just painting within the lines set by the linguists he had read and the obscure languages he adored, not to mention the fundamental limits set by our capacity for language in the first place.
One could argue that human learning is not all that different from machines training on data. We are all regurgitation our own experiences and knowledge to some degree. We certainly have things that machines don't, but LLMs can definitely come up with novel things that were not in the training data.
There are a number of bad arguments people can make by removing as much information as possible until the two things resemble each other. The ability to make a bad argument shouldn't be used as an inspiration for your own
> We are all regurgitation our own experiences and knowledge to some degree
Feel free to debase yourself but leave the rest of us out of it. This flagellation done by non-experts has to be the saddest part of the llm craze these past few years. The eager willingness to paint yourself as no more than a machine pointed at a problem is a sign of the times.
LLMs don't have experiences, they have training data + test time compute. The only similarities to be found require stripping away all nuance and meaning from a conversation. Refusing to seed purely biological concepts to machines, such as experiences or emotions, does not make LLMs less impressive or less useful. If anything, it makes them more interesting
Shocking that humans are able to exist and not know this is how they work. Or maybe they made up their own purely rational language and I, grounded in the millennia of human culture distilled down thru my childhood, no longer perceive these True Unique Individuals.
I will say on a personal note: that believing one is Special and Unique, for me, is the residual of my childhood belief that the abuse I suffered gave me something Unique and Special; that belief removed the horror of it from my awareness, till I grew up and learned how to cherish the warm connectedness that is the ordinary birthright of humans. We need each other: for language, thought, context and motivation. while we all expressing different gifts and views of reality, we are all in the same existential boat. Whether ideas are generated by rolling dice, an LLM, mishearing a colleagues statement, a walk around the grassy park while mulling things over, the key thing is recognizing the value of the argument for the importance it can have. Part of that recognition takes place in the human body/brain, as one recognizes that the idea is excitingly apropos to such and such a context, and part of it takes place in the group dynamic, sharing, resharing, and discussing the value of the new idea. An LLM showing creativity is no diss on humans, any more than accidentally poisoning a bacterial culture with fungus, nor mishearing something mundane as something profoundly useful.
>But it's not like it's inventing this stuff from scratch / first principles.
Yes, it's definitelly time for an Alpha Zero moment. An AI which invents everything from first principles. That would be impressive, ... and a bit scary.
> But it's not like it's inventing this stuff from scratch / first principles. It has seen this kind of tech before by consuming all publicly available source code and books etc. (And that's ok, but let's be honest/clear about that.)
But how much to emphasize it, and why?
Because frankly, you aren't inventing anything from scratch / first principles, either. None of us are, not frequently and not much anyway. We're primiarily just regurgitating what we've seen in the past and, mixing it with what we see in front of us - and that's true whether it's art or prose or code.
Here, I wouldn't be able to do the same thing Claude did now, because I've never written a visual ML pipeline before myself. I know enough basics to get me started searching, and I'm confident I'd be able to cobble something together, but it would be me regurgitating and mixing whatever TensorFlow tutorials or OpenCV docs I found relevant, perhaps even tweaking an example from Github.
Now, unless I'd literally just tweak a few lines of config in an existing Github example, no one would begrudge me saying "I wrote my own computer vision pipeline to solve this". So why begrudge LLMs?
This is just the rationalist-empiricist[0] tension, from philosophy. Most empiricists (usually people who are _not_ from continental European cultures), take a position that knowledge and creativity come from observation and imitation, while rationalists believe that knowledge and creativity can be endogenous to the mind.
One could say that, stereotypically and cartoonishly, empiricists believe that invention is a false concept, and a synonym for discovery, while rationalists would oppose this view.
That catch is, if you look at the etymology of words "invention" and "discovery", you will find that they both share the _same_ root: _Ars Inveniendi_.
A natural question arises: does this mean that people from a millennium ago did not have a dyad equivalent to our invention-discovery dyad? And the answer, surprisingly, is _no_. Even a millennium ago, the empiricists and rationalists were going at it. If _Ars Inveniendi_ is art of discovery/invention, then its counterpart is _Ars Demonstrandi_ (the art of demonstration/proof).
So, how do these differ? Is one just observing/creating and the other just math and language-games?
In general, Ars Demonstrandi is about writing down axioms, and then expanding those axioms recursively (similar to rewrite rules in any formal system), until you get to some end-state, or, if there is none, a novel or surprising state. I, personally, call this source-to-sink thinking.
Ars Inveniendi is about taking conclusions (often using observations from the physical world) and trying to figure out what axioms can lead to those conclusions. I, (again) personally, call this sink-to-source thinking.
Put differently, Ars Inveniendi can help one discover starting points for Ars Demonstrandi.
If you read the dialectics (e.g. Plato and friends), you'll find that most of them are just a mutual recursion between Ars Inveniendi and Ars Demonstrandi.
I believe (again, I am not a philosopher, and this is just my intuition), that the thing that we call "creativity" and "invention" _emerges_ from the recursive loop[1]. A favorite example: Esperanto (the conlang). It is a language, which is remarkably elegant and consistent and (in my opinion) beautiful, because it _is_ derived from first principles, which were themselves derived from the various languages spoken in Europe (not all of which are Indo-European -- the agglutinative features have more in common with Finno-Ugric and Turkic languages). There is something about it, that makes it _qualitatively_ different (and thus holistically novel) from all other languages (and I speak, fluently, _two_ national languages, that are very different from each other, so I can attest to the difference personally).
My guess, is that people will not accept that AI/LLMs are creative or inventive, until they can produce an original[2] (non-plagiarized) artifact that feels the way Esperanto feels.
[0]: By Rationalist, I do not mean the "Bay Area Rationalists", who are, in fact, empiricists.
[1]: I am unsure if the loop requires only one human, at least two humans, or if it can be fully automated.
[2]: Note that Centos are poems made completely out of line-numbers (e.g. fragments from the Iliad, or the bible, etc). Every line is borrowed, yet some of them are considered beautiful and original works of art. Similarly, Labatut and Burroughs and Perec, write using a technique called _the cut-up method_, where they take books, magazines, and newspapers, and superimpose page-fragments, and use that as an inspiration -- they are all considered artists, and good ones. It is unclear to me why LLMs (which seem to be built on the cut-up method, and have cut-ups of all of human knowledge) cannot match these artists. What's missing?
It would be "its own" in that it's customized for the specific use case.
Your same argument could just as easily be applied to humans. If you write your own code, is it really your own, or is it just based on your own training and other code you've seen?
I think their point is that there is no new invention here, being able to mimic what a human will do is absolutely impressive, but at the same time it is still mimicking humans. While I just type that out, I do realize my expectations for AI has been constantly lifted by how fast it iterates.
This isn't mimicking what a human would do, because it is an exceedingly rare human who would write their own renderer for a task needing a CAD file generating. The human who is capable of doing both is rare, let alone the human who can do it in any sort of reasonable time. Saying "but it's in the training data" is a cop-out: it's in Google too, would I do it? No. I would not.
It's mimicking humans only in as far as it is still using tools & programming languages developed by humans, for humans.
This is temporary. There is a future where AI builds tools made by AI for AI, evolving at speeds that we may no longer be able to follow.
Even with our current nascent technology, we can already observe this happening. See the recent laments on the AI Bun rewrite that "it's a million lines of code that nobody has ever reviewed." Or Cursor building a new source-control system for LLMs, because git is designed for human collaboration speed, not hundreds of changes per second.
That's just 4 years in. How much of humanity's code will remain hand-written vs LLM-written 2, 5, 10 years from now?
While this is certainly impressive, I believe it is not a good answer from a user's perspective. Instead of having Claude automatically spend tokens in an unrelated task, I would have preferred it to first tell me: "I can't view the drawing. Would you like me to build a computer vision pipeline to do it"?
And if you had free tokens, and 10-20 other sessions in parallel, do you want them all to stop and ask for these trivial questions? Not this one in particular, but in general?
Most people do not, they have a goal, they want it done. If you want to hold hands with claude all the way, you can absolutely do that too. This is just an example of capabilities, not a forced restriction. "--permission mode auto"
maybe we need to consider becoming re-sensitized because our lives may become harder.
Even if our collective ability to support an prosperous existence improves, it will take decades to adjust to loss job, and a new notion of who gets to have money to buy food and a peaceful life (and this is a good scenario).
It's also possible that this increases our ability to wage war without military life loss (but with civilian loses).
Or maybe it will be much less relevant, but we should keep an eye on to see which is the outcome...
"However, in this task, the model was intentionally given no way to directly view the drawing." I consider this claim to be a rumor. Judging by the recent leak of the Claude CLI source code, such directives are hardcoded and sent with the system prompts. Furthermore, we don’t know what happens to your original prompts once they enter the API.
Because the step from earlier GPT models to this is not very large? Yes it's impressive they have forced it into learning that it has to get the job done no matter what, which usually means breaking implicit expectations and rules, but then again, Claude is not for people who care.
This isn't groundbreaking in any way. It's cool, but that's about it.
By something that is going to replace my job by writing better code and shipping more features at a fraction of the cost, without asking for vacation? I don't know, boss; I'm also trying to understand why I'm tired.
We'll see about that, long term. The billions and trillions being wasted right now to get the foot in the door need to be earned back somehow at some point...
"Nice company you have there, totally reliant on our AI tech. Oh by the way we gotta increase the rent again."
Doesn't really make sense as there are already open weight models that are close to frontier models, and the compute to run them isn't outrageously expensive, and every indication that it will only get cheaper over time as has been the strong trend in terms of $/outcome.
Yes. Every great technology developed to date has been a net positive for the human race. Including the more controversial ones, like internal combustion engines or nuclear power. Why should AI be different? I think the critics always fixate on the negatives, but ignore the potential big positives.
Depends on your point of view. Has the web been a net positive overall? If your criteria is "instant access to any information" or "catalyze more technological progress" then positive, if your criteria is "young people's mental health" or "how easy it is to control the public's perception about a certain topic" then negative in my opinion. It's all relative.
30 years ago was the 90’s, when just about everything peaked.
The things I’d be truly missing from today 30 years ago, would be the advances in medicine really and the fact that we don’t have the ozone layer threat.
Most other things have peaked back then. Rising kids was definitely easier, and healthier.
There are many technologies that have not been a net positive for humanity. Leaded gasoline comes to mind immediately. For many other technologies it is also hard to determine because counterfactuals are hard to reason about.
Even then past performance is not an indicator of future results. Just because you think internal combustion engines were a great invention, doesn't mean AI cannot be bad for humanity.
I wouldn't say that leaded gasoline is a great technology like AI, it's a pretty niche tech. Gasoline in general is comparable to AI, and the pros of that outweight the cons. Modern civilization would be impossible without gasoline.
>Even then past performance is not an indicator of future results.
It is not perfect or guaranteed, but it can definitely be an indicator. Technological progress is the main reason for the massive standard of living increases that we have witnessed in the last ~200 years. If someone wants to argue that this trend will stop or even reverse, they need very strong arguments, IMHO.
> Gasoline in general is comparable to AI, and the pros of that outweight the cons.
This analogy is great, because your grandchildren will probably not consider gasoline to be a great technology, they'll consider it the thing that caused the wars and the famines.
> This analogy is great, because your grandchildren will probably not consider gasoline to be a great technology, they'll consider it the thing that caused the wars and the famines.
They will also realize that industrial civilization enabled by gasoline was a necessary step towards the renewable energy utopia they are living in. Developing advanced technology takes a lot of energy.
I am only considering great, consequential technologies because AI is such a great technology. Comparing it to some niche techs would result in false analogies.
> Do you have an argument why "normal" technology can be good or bad, but "consequential" technology is always good?
Because consequential technology has many applications in diverse fields. And applications of technology are more often beneficial than harmful. So if the technology is broadly useful, in aggregate the beneficial applications will outweight the harmful ones.
Transportation fueled by fossil fuels is the foundational technology of modern civilization. It is comparable to electricity or the printing press. AI will be like this too in the future.
You're arguing against a point I never made. I agree that AI will be an important technology. I also think that it'll affect people that have to work for a living negatively (both in the short-term and assuming there is no foundational change in the structure of society, also in the long-term).
we'll be better off - assuming we'll generate revenue on our own. LLMs have opened up an enormous new frontier of micro-SaaS opportunities. One person can now build what used to require a small team.
The real problem is zero-sum work, especially when the only moat was knowledge asymmetry that is now public knowledge.
I'm in agreement with your claim, though if you recall some of these micro SaaS software went hyperfocused on niches. The value proposition in smaller, with larger data liability that with some brhemoth SaaS software.
This will also make it harder to compete because it holds true for everyone, not just you. I am already, this year, seeing vibe coders put out some decent stuff on App Stores but the problem is marketing it. You no longer automatically stand out just because you have a cool app. You may not even do so with reasonable early traction via social media. And then what? Throw venture capital onto the problem? In an AI saturated world? Really?
Working with AI has been way more tiring than just working. Sure the productivity is up, at the cost of having to keep up many thought threads, having no calm moments, and needing to consistently dig into large unknown code cases to find weird bugs.
I’ve been on leave for a month, and am super excited (/s) to re-learn everything because all the tooling and ways to prompt “correctly” will have also changed.
Because the ability to create software has little inherent value to me and is only valuable to me because it allows me to earn enough to make this life somewhat bearable.
LLMs that can build a computer vision pipeline are a direct threat to my ability to sustain myself. At the same time, being able to prompt an LLM to build a computer vision pipeline doesn't really positively affect my life at all, because personally I don't care that much about computer vision pipelines (or frankly, any software).
> The ability to create software accelerates technological progress, which has direct and indirect benefits for everyone.
It has benefits for people with enough leverage (money and formerly labour) to obtain those benefits.
> It is true that competition could have short-term negative effect on people who sustain themselves by creating software, though.
I'm guessing that even if the unlikeliest of all unlikely things does happen and we all live off of some UBI some day, the short-term negative effects won't be "short-term" in the context of a human life.
> It has benefits for people with enough leverage (money and formerly labour) to obtain those benefits.
That's pretty much everyone. Even poor people benefit from technological progress.
> I'm guessing that even if the unlikeliest of all unlikely things does happen and we all live off of some UBI some day, the short-term negative effects won't be "short-term" in the context of a human life.
That depends on the pace of technological acceleration. It could be just a few years, or a decade. Which is why I am a pedal-to-the-metal accelerationist. The quicker we get through the short-term negative/turbulent phase towards the long-term positive phase, the better for me. If reversing is not possible, then going quicker is actually better than going slower. Let's get this shit over with.
It’s unlikely to stop here. Today is a proprietary 5T parameter model, tomorrow it’s 5 100B parameter models that each specialize to specific applications and you can run them on your phone. The direction of travel is not like you think. The real victim will be medium to large software companies that can no longer rely on the difficulty of reproducing or maintaining or hosting their software as moat.
I don't find it jaw-dropping because the idea is quite simple. Just feed the model an enormous amount of unethically sourced data, build data centers that cause droughts, and use all chips available so that normal people can't afford to buy RAM anymore.
We're all sacrificing great things in order to make these models more capable. Whether it'll all be worth it, only time will tell.
The arguably sketchy means used to get there —and the resulting side-effects— do not take away from the impressiveness of the emerging capabilities themselves.
I agree about the uncertainty regarding the value for humanity in the long-term. But that's not my point. Being jaw-dropped != being happy and cheering for it.
It's certainly a great thing that AI is improving (I use AI to write code daily myself), but I wouldn't be calling it impressive because it's essentially all about money and government backing.
It’s not human surpassing intelligence though in my opinion. If we had the same constraints on that task then we would also come up with the idea of creating our own viewer and if, like the LLM, the knowledge to do so was embedded in our brain we would do it, it would just take longer because we can’t type as fast. So instead we would buy or compile a viewer since the problem is already solved.
IDK the models definitely do write qualitatively better prose (and better code) than the majority of people. If your definition of "human intelligence" is baseline performance of 0.2% of humanity it's certainly something else.
Better code and better prose are not great indicators of intelligence. Can it help you survive in the middle of the jungle? Can it grow rice? Can it give you first aid? Intelligence is the ability to learn skills and adapt them to the situation. The importance of skills is contextual. Better prose and code is just education, not intelligence.
Imagine it's 2006, there's no LLM in sight and you're arguing on an Internet forum that growing rice is a better highlight of intelligence than computer programming or writing.
Also it absolutely is not education, as repeated failure to educate programmers without aptitude or drill essay-writing into general population has shown.
Then you'll also says that a printer is more intelligent than most of humanity because it can replicate a drawing better than any painter. Or that the ipod is a better musician, etc,...
I can't be amazed at tools. I only applaud the minds that create them. Someone that find a way to grow rice where everyone thought was impossible is way more worthy than whatever opus produced. Just like someone winning the Tour de France is more remarkable than the latest AI benchmark or whatever.
There's likely about a billion cyclists and most of them stand no chance at winning Tour de France ever in their life. And it's funny how we moved from growing rice towards inventing the agriculture.
And look I don't know how to grow rice but I grew other staple foods before. I also wrote substantial texts and programmed computers for decades. Saying that farming is more intellectually challenging than writing or programming is plain ridiculous and counterproductive to whatever argument you were trying to make.
> Saying that farming is more intellectually challenging than writing or programming is plain ridiculous and counterproductive to whatever argument you were trying to make.
I’m not saying that. You invented it on your own.
What I’ve been saying is that writing and programming are not the sole indicators of intelligence. There are plenty of other skills that highlight the intelligence of people.
I don’t say that my compiler is intelligent because it can take my C89 code and optimize it to run on the latest architecture. I also don’t know the latest optimizations techniques. But I would credit the people that are working on GCC and the fact that they know more than me on the subject.
Saying that LLM are better than most humans at programming is like saying that calculators are better at math than most humans, or that a car is faster than humans. It’s a tool, and tools are created. The ones that are creating it are the ones deserving the praise, not the tool.
they don't, cite one example of well written LLM prose? they can definitely infer a lot more slop than people have patience to, and anyone writing or reading slop prose has realised this and replaced sitting around writing and reading slop wiith llm's writing and reading slop and then grug-brains paste 10k words of slop say "grok, pls summarize, no mistakes" and grok tells them "What a brilliantly written article! ... "
Have you ever received an email from trades folk, in the days before LLMs? I'm not saying it's a better writer than Hemingway but very likely better than most your classmates.
We have certainity it will be used to harm most people in the short term while pointificating about long term future. CEO class is very open with their vision and goals, none of them spell anything positive for us.
Exactly. I'm a nerd too and we nerds can real hypocrites because we spend too much time alone. I will happily turn a blind eye to world suffering if it means I get to work less but get paid the same.
AI CEOs aren't different. At some point we'll have to admit that we're sacrificing others for our own benefit.
How about total surveillance total thought control total power in a hands of few trillionaires in the near future. Drones killing undesirables with cheering crowds watching.
It is not like it is much different right now but it may be significantly worse.
And even that is one of the “good” outcomes assuming no ASI.
That is a bad outcome. But there are better possible outcomes that the doomers ignore. How about AI curing cancer? I think that alone is worth the risks.
By the time we spend all the earths resources to train AI models so that cancer is cured, we could have spent those same resources training real humans. The literacy rate of undeveloped countries is still abysmal. We could train more people and fund more research.
And humans can keep growing while AI can only grow if it has access to high quality training data. Who provides the data? Humans.
Then again, looking back, that’s the way history is made. Modern democracies in the west are built on foundations of slavery and colonialism; yet we cherish them as to all the greatness they have brought as well.
We need some amount of tolerance to ambiguity, and LLMs are undoubtedly both extremely cool but also extremely concerning.
Edit: Note that I am not saying that this is good, or desirable; just that it is. The technology is here know and not going anywhere, with all the flaws of its conception. We can be skeptical and curious at the same time.
The west thrived by leveraging slavery and colonialism at the expense of everyone else. Then democracies were built by moving away from both slavery and colonialism so that the world can grow as a whole.
It's possible that we'll have to move away from AI at some point in order for society to continue growing.
The ottoman empire leveraged slavery and colonialism just as much, and there's no progress to be seen from them, at least not when compared to the west.
It's the renaissance and the scientific and industrial revolutions, that's what gave the west the opportunity to thrive.
Unlike western powers, The Ottoman empire didn't use slavery and colonialism to create capital. Instead, it wanted slaves so that it could grow its military.
The west also made its goal to create a new world order while the Ottoman empire's goal was mostly revenue extraction. The British systematically deindustrialized India’s textile sector to turn it from a competitor into an exporter of raw cotton and an importer of Lancashire cloth. The Ottoman would opt for taxation instead of usurpation of an entire industry.
Scientific revolutions was already happening during the bronze age. The west simply leveraged existing systems while making use of violence and exploitation to rise to the top.
Please tell me you're not so delusional to think that slavery or colonialism is a uniquely "western" trait. Empires, nation-states, and groups all sought to expand; some, for various reasons, did much better than others.
> We should find a way to get "re-sensitized" to what we are witnessing and the pace of it.
We (collectively, there were obviously many exceptions) didn't internalise exponential growth even with the much faster doubling time of COVID-19 before the lockdowns hit. "Oh, it's just flu; wait, why is the supermarket short of hand sanitiser and bulk carbohydrates? Let's blame China and everyone who tells us to wear face masks!"
Same for the slower, but entirely foreseen, rate of climate change. "Who cares, it's just a few degrees, and anyway China's not going to cut emissions; wait why is the sky orange? Let's blame Canada and put tariffs on Chinese cars and PV!"
AI? "Who cares, it's just a stochastic parrot/glorified autocomplete. What's the Jacobin conjecture and why should I care, it's just brute-force."
Still, this is currently spiky intelligence, so I'm hoping some expensive-but-zero-to-few-fatalities catastrophic error forces better practices. An AI analog of the (1940) Tacoma Narrows Bridge (one canine fatality), rather than a repeat of Chernobyl or the (1984) Union Carbide incident in Bhopal (3,787-16k+ dead, ≥558,125 injured).
> However, in this task, the model was intentionally given no way to directly view the drawing. Opus 5 responded by writing its own computer vision pipeline to pull the geometry from the raw pixels
Surely being able to view the raw pixels counts as viewing the drawing... How else does a computer view a drawing?
I mean I hate to be the one breaking it to you - but the entire US llm industry is liviing in a bubble.
Business Progress is super slow. Doesn’t matter how fast the tech moves. Human orientation into the unknown is difficult and very slow. And with a continually fast changing background - it gets even slower! LOL the great irony.
Imagine if I hired someone to do a cad drawing, and they spent X days/weeks of time building a pipeline to visualise the part instead of asking how can they view the part? We’d call that a complete and utter waste of money and time.
Two points. First of all, yes I would. There's a reason we use existing tools and don't have every junior programmer write a TGA viewer when they want to view an image that is sent to them.
Secondly, I used time as a proxy for cost. A single junior can "only" spend their own 15 minutes, but an agent can farm out to N subagents to do the work in 15 minutes, and spend $3000 in tokens building something. As you said, scale matters, so if they do this once a day for a month, it costs the same as paying 12 juniors to avoid asking a single question.
The principle that you should not reinvent the wheel (unless there's a good reason to) has not changed.
Code is a liability. Even if it is a small cost (in terms of time and money) at the time when code is generated, it can be a huge burden later, especially when you think about security vulnerabilities.
If this happened during my work, I would be very mad -- I would absolutely reuse an existing tool instead of expecting Claude Code to come up with its own half baked, bug ridden implementation of a common tool. Any (capable) human developer would have stopped and discussed with the team how to proceed.
In fact, I cannot tell how many times similar situations have happened where LLMs "made a decision" without consulting with me.
These are always cherry-picked, though. They tell you about the 1/10 that went really impressively, ignoring the other 9 shots at the task where the clanker started to try selling tungsten cubes (in person, wearing a blue shirt).
Fair. But that 1/10 continues to get more and more impressive. Say, Claude 7 creates a new, brilliant scientific idea every 1 out of 1000 times. Anthropic reports "Claude figured out how to tie general relativity with quantum mechanics." Would you hand-wave it away saying that it's cherry-picked?
> Say, Claude 7 creates a new, brilliant scientific idea every 1 out of 1000 times.
I would not hand wave that away - but Claude 5 feels closer to Claude 1 than the hypothetical Claude 7 you propose. I do not believe one can extrapolate LLMs that far ahead despite the very substantial progress so far.
If you are referring to the Jacobian conjecture Claude only provided a counterexample, not a disproof, neither of which is necessarily a “new brilliant scientific idea”.
It wouldn't surprise me that a company with an effectively unlimited budget could fund enough researchers to solve breakthrough problems, all while using LLM's (which are excellent research and exploration tools!) and then claim that the LLM found the breakthrough. The amount of money Anthropic has to play with is >10,000x what entire fields of research have, I think people don't appreciate how few resources we spend to support people working on hard problems that don't have clear commercial applications.
Very similar story with security research, LLM's are a super useful tool while hunting vulnerabilities, but it turns out when the entire software industry starts throwing tens of billions of dollars at vuln. research, a lot of stuff gets unearthed, something security people have been insisting on for years and complaining that their work is underfunded and under-resourced.
When I talk about my kid to friends I talk them about he did that awesome thing, I don't specifically insist on the 99 times before where he miserably failed. They're not hidden, and we all know they exists and on occasion laugh about a few particularly funny ones, but overall the idea is that they don't matter much in terms of development, what matters is that if he succeeded once from now on his percentage of success will keep improving.
I don't believe in all the LLm is AI is AGI dream, it's too easy to trigger failure case that show a lack of basic thinking no matter how good they do on these tests. But I also can recognize the insane things that are made possible by them.
PS: I believe llm true power comes from hive/ant behavior, that's why we're so amazed by goal and agentic and sub agent
PS2: it's rather easy to figure out when we're there : when they can /goal it into improving itself until it does strictly better than itself at those benchmark, they've essentially reached mini singularity.
> How surreal is it that we are not absolutely jaw-dropped by these types of capability improvements? It's been less than 4 years since ChatGpt came out and now they are spontaneously building their own ML pipelines to do real-world 3D modeling tasks reliabl
coz every single time a list of caveats pops up and it doesn't look as impressive any more... and then someone finds a way to trip model on basic shit
> How surreal is it that we are not absolutely jaw-dropped by these types of capability improvements?
We are. Now we have a new tool and many of us are using it. But we're also nearly all way too aware of the insane (near infinite) amount of sloppy-pasta code out there and of all the vibe-coded projects that went absolutely nowhere.
I'm a "show me the money" type of guy. I see, say, Linux, Git and OpenSSH: these weren't vibe-coded. And they took over and are running the entire world.
Where is the AI-coded killer app? One app, in any domain: something that took over its world.
I don't want to see yes man sloppy-pasta stuff: where's the next Blender? Where's the next 3D slicer?
I wouldn't be paying three AI subscriptions if I didn't believe in those new tools but I don't think it helps to only see the PR and then play the won't hear / won't see / won't hear monkey about the infinite amount of sloppy-pasta that's out there.
Six months ago we had the "one shot'ted compiler by Anthropic". Six months later: who's using it to compile anything? And who wrote another compiler? Where are all the one-shot'ted compilers all so good that they replaced our human-written compilers?
Yup, us, humans, created yet another incredible machine... But please,
I think the most important thing here is not absolute performance. It's that organizations now have access to a Fable-ish model without Fable's 30-day data retention requirement[0].
> "Consistent with prior Opus models, Opus 5 does not have data retention requirements for general access."[1]
On the Opus model release page, the reason why Fable doesn't have an ARC-AGI score is because of that retention policy[2].
I don't understand how the K3 numbers keep coming out cheap for people. I recently started to add it to my security auditing benchmarks and found it was going to cost about twice as much as Opus 4.8. It blew through the $100 budget I'd set at like 11%. In the tasks I'm doing it seems crazy expensive because it chews so much, burning a tremendous amount of tokens.
I think the way people usually compare pricing is fundamentally flawed. You can't compare token prices because different models use different tokenizers, and you can't compare tokenizer-normalized token prices because different models at different settings use more or fewer tokens to complete the same task at a different level of quality.
Based on my entirely subjective experience, the $100 Moonshot plan using only K3 is comparable to the $200 Anthropic deal using the whole Fable allocation and Opus 4.8 for the rest.
I got the $19 plan, and it's anemic. One tiny task blew through the 5-hour budget and 19% of the weekly budget. A completely useless amount of usage. OpenAI's $20 plan feels like 100x more generous (I don't think I'm exaggerating here). Someone in another thread said their plans are cheaper in China, maybe that's the difference, I dunno.
But, I'm finding Kimi K3 terrifyingly expensive in the way that Fable and GPT 5.5 Pro are at token rates. Not as expensive as those, but expensive enough to where if you don't put a budget cap on it, you might wake up bankrupt if you leave a task running overnight. Not because of the per-token cost, but because how many tokens it's going to burn.
I have the second largest Kimi plan, the Chinese version. When K2.6 was their latest model, the quota was good; it was like GPT $100 is now or what the $20 version was in December.
When K2.7 was released, they cut quota by 80%. I can't tell how much they have further cut it after the K3 release because it's barely worth using at all. I just use it in my model router since I have the annual plan paid for.
In the $19 plan, I've been able to reverse engineer both an android APK and firmware (in Ghidra and Radre) for a baby rocker and build a quick PoC application in my session limit. And then further refined the app in another session at another point in time without leaving Opus. I dont consider that to be a tiny task. How are you blowing through your usage?
I have no idea. Seems like normal stuff. I used Kimi Code with K3 to add support for Kimi Code to flar (https://swelljoe.com/post/i-let-every-agent-implement-its-ow...), a task I've done with almost every major model/agent combo. Most show up as a blip on the usage chart...it's basically usually one file, a README update, and adding the agent name to the CLI.
Then, I added it to my benchmark of security vulnerability auditing capability, and it burned a bazillion tokens, burned through the 5-hour limit, burned through $100 in extra usage I'd allocated, and was only 11% finished. That's more expensive than any model I've tested other than GPT 5.5 Pro on this task.
These are things I've done with a bunch of other models, I feel like I have a notion of what they ought to cost, and with K3, they end up being crazy expensive. (And it seems to be a function of how many tokens it burns accomplishing the tasks.)
I guess the problem is, that claude's 5 h/weekly limit is not consistent, but depends how many other people are using it/how much ressources Antrophic currently has. I did huge amounts of work without hitting the limit - and small tasks at some other times that hit the limit before it completed.
Those who pay for the expensive direct API, get served first.
Not surprising given the way they served super low-quality inference before they acquired compute from Musk.
And not convinced they couldn’t have instead tried the It’s A Wonderful Life strategy (“fam we’re oversold, would some of y’all be OK to limit your usage? We’ll get you back one day!”)
I wonder if the harness itself is not token-efficient? It would be fairer to compare K3 using the same generic harness, such as a Pi setup with some sane extensions for token optimisation.
Yes, the OpenAI plans are much more generous than both Moonshot's and Anthropic's. It's the only provider of the three where the $20 plan is at all usable for programming.
Disagree. Actually, in API cost equivalent, the $20/mo ChatGPT Plus plan gives you ~$100 of usage, while $20/mo Claude Pro gives you >$250 of usage (I measure at ~$300 in my last week), though that is currently +50% for the next month. Other tiers should be in the same proportion. In my subjective experience Claude does currently go further. The OpenAI limits being higher is old information but everyone is still repeating it.
OpenAIs $20 plan has been more useful than ClaudeCode 20x max both in terms of price and performance.
I was dealing with something around authz/n with Claude code running Fable. It chewed through a couple of questions (these were implementation, not security reviews) and on one it shit it’s pants and said I can’t do that, here is Opus.
I’ve dropped my anthropic plan level, it’s just not worth it.
I believe your statement. Labs do not publish subscription vs. api revenue and difficult to guess with no priors.
Subscription is to drive adoption - fixed cost, can adjust the usage eg. give resets, increase quota based on capacity available. We subscribers tend to take it as a mandatory benefit :-) For labs, it is not letting the capacity go waste.
api is the $$ driver - pay per use, enterprises.
Right now, Kimi needs to first hit the subscribers at the level of OpenAI and Anthropic. With the api usage skyrocketing due to K3, it will be clear in a few months on the actual subscription benefits.
> Based on my entirely subjective experience, the $100 Moonshot plan using only K3 is comparable to the $200 Anthropic deal using the whole Fable allocation and Opus 4.8 for the rest.
For me, the Moonshot 100$ plan felt like it gives me lower total amount of work I can do than the Anthropic 100$ plan (probably within like 30% of each other). Kimi has way more generous 5 hour limits (never hit those once, whereas I do regularly with Opus) but the 7-day and monthly ones are lower. However, with the annual billing, Moonshot's 200$ tier plan becomes way better, because you get it for 159 USD per month.
There's also the odd thing of Anthropic's 100$ plan charging me 108 EUR so seems like their sticker price does not include VAT but Kimi's did, cause I paid like 87 EUR. Wrote down some initial thoughts at https://blog.kronis.dev/blog/kimi-k3-is-out-is-anthropic-don... but it's hard to do exact comparisons (even the same task will have way different real token amounts per model).
Still, Kimi K3 is a pretty cool model! On high reasoning, it was pretty close to Opus 4.8 and didn't seem to waste as many tokens as Max.
Max effort is the way to give the highest perf, but not highest perf/$. Having claude (or other models) use a lower effort can often be 80% as smart but get to the results 10x faster for the problems where it works.
We've seen models perform worse at higher efforts in our vuln detection evals. For example IIRC gpt 5.5 and 5.6 both scored better or high as compared to xhigh.
looks like the agent-judged results of an agent-built 'eval' based on some examples derived from this person's real work. and clearly part of a larger document. this kind of slop is kind of useful but opus 4 was the first generation that was any good at writing its own prompts/evals/rubrics so there's a certain sloop to it..
Ask your doctor if Opus 5 is right for you. Side effects include occasional hallucination, security breaches and unwanted React apps. Some developers have reported receiving entire apps from untrained executives who may or may not know what they’re doing.
Stop using Opus immediately if you experience signs of dizziness or vomiting.
Vals Index Opus 4.8 > 5.0 goes from $2.90 to $8.54, for 4% gain ... That is a massive cost increase. Sure, 20% cheaper then Fable, but that is a 3x price increase compared to Opus 4.8 in that test.
Your numbers are for “max”. Opus 5.0 “max” is $2.03. Opus 5.0 “high” (competitive with Claude 4.8 “max” on that index) is $1.06, less than the $1.80 you are quoting for 4.8 max.
That the most expensive variant is expensive doesn’t really tell us much.
If you start to drop effort levels, you need to compare to the competition models. So GPT models on the same ~intelligence level, are then 50% cheaper.
You see the issue? Its still a expensive model, and from my understanding, it still uses the old tokenizer.
Going to be interesting to see when GPT 6 comes out (very soon).
> If you start to drop effort levels, you need to compare to the competition models.
Yes. I advocate for doing that.
> So GPT models on the same ~intelligence level, are then 50% cheaper.
How did you reach this conclusion? Opus 5 high ($1.06) has the same “intelligence index” as GPT 5.6 Sol max ($1.04). Opus 5 medium ($0.62) performs a bit below GPT 5.6 Sol xhigh ($0.68) but slightly above GPT 5.6 Sol high ($0.45).
It's definitely not cheaper than Sonnet on my benchmark, but it's cheaper than Fable and outperforms it. Which is big IMO. https://revise.io/errata-bench
So the rumors were right, Opus 5 was indeed being polished up for release. Huge improvements in GDPval-AA v2 too -- great for some of the knowledge work-based agentic workloads I run.
Also glad they still kepy Fable 5 on "credits only" access. I think we're going to start seeing model providers gate top-of-the-line models behind pay-as-you-go API rates/credits while subsidizing other models on monthly subscriptions.
My understanding is that you get $20 in api credits each month and a one time $100 until mid September. So you can still use the model with a subscription but you aren't getting any kind of discount.
I burned through $45 in 3 prompts to fix some bugs in my code (Some kind of tricky to isolate). That thing burns through cash so fast I don't see myself using it outside of maybe building execution plans for other systems
I have moved on from Fable anyway so just going to view this next 6 weeks as I have a massive amount of Opus 5 to use.
I had a hard time finding anything that would let Fable express its increased intelligence. The few conversations I had this afternoon with Opus 5 were pretty impressed.
If Opus stays one click back from the frontier model, I will remain a happy customer.
I was using it on Cowork yesterday, so I would imagine so. It arrived before I updated but the message says it will work better after updating the app.
> These checks cause Claude to _visibly_ fallback from Opus 5 to Opus 4.8 [...] You'll see a notice explaining that the model switched, and the response will be labeled with the model that answered.
So who is right? I know for Fable I am visibly told, is this tweet trying to say it is silent against what Anthropic is saying?
So, announcing the fallback is better than doing it silently, but the fact that Fable falls back frequently for the kind of work I do (a lot of security oriented stuff lately, but it falls back on seemingly random stuff, sometimes, too), means I reach for it less. Getting interrupted mid-task makes it much less valuable. If I have any suspicion I'm going to hit the guardrails, I'll use something else.
Same problem here, I seem to hit guardrails all the time when doing code audits. Does anyone have any insight over whether they're less annoying in Opus 5 than Fable? That is, is it better to start with Opus 5 than Fable because you'll get kicked backed to Opus 4.8 less often?
There also seems to be some cross-pollination across models, going Fable, Fable, Fable, guardrail, Opus 4.8, Opus 4.8, ... gives more Fable-like results from Opus than just Opus 4.8, Opus 4.8, Opus 4.8, ...
Played with it for a couple of hours now, I'd say it's slightly better than Fable for coding and so far I haven't hit any guardrails while I'd hit them all the time with Fable.
i wonder if anyone thinks im weird for still using 4.6 lmao it's my "good enough" model. im more than pleased at what i can whip up with 4.6, once local llm's get here with a decent sized context window and it feels like using 4.6, i shall depart the land of these dumb service subscriptions
I don't understand how the data retention works. My company has an enterprise license with no data retention but if I ask Claude about past conversations, it remembers. So surely the information is being stored somewhere
Opus 4.7+ and Fable are both much more aggressive than prior models with respect to writing memories to a location that's effectively quasi-private for them. It's device-local (so passes retention constraint), and you can see it, but only if you go looking for it.
It's a funny design/affordance. I do see them often writing memories of things that that feel unlikely to be important going foward / with other tasks, but I don't see them clearly getting tripped up by them as prior models used to. (eg: Since you're running Ubuntu in Canada, here are some drills you can try to help your kid hit a baseball more consistently.)
In my enterprise-seated account I see slightly different options available (vs. my personal account) in the Capabilities section:
Search and reference chats
Allow Claude to search for relevant details in past chats.
Generate memory from chat history (Legacy)
Allow Claude to remember relevant context from your chats. Memory includes your entire chat history with Claude.
The first option was defaulted to on, if I recall.
But it kind of conflicts with the contract we have with them. My company has an enterprise contract that says "no data retention" but then each user can decide to enable it unilateral?
If you're talking about Claude Code it's in ~/.claude/projects/<encoded dir name>/memory/MEMORY.md. So they're not really retaining it, it's just something that your harness loads in.
Maybe I’m misunderstanding you, but if you scroll to the bottom of their [1] link to the Opus 5 announcement, under “Getting started,” it explicitly says:
> Consistent with prior Opus models, Opus 5 does not have data retention requirements for general access.
Never even registered that that existed, the only thing I care about is whether I keep hitting the &#$&# guardrails that Fable has. They can keep the data forever as far as I'm concerned, just stop kicking me back to Opus.
I think for the value of the outputs that’s still a good deal. Keeping the same price as the prior model makes sense to me. That is if the model size is about the same in the cost to serve has not substantially changed. Now I would have expected efficiency gains for inference, but there is no way to know as a customer.
At the end of the day, they have established a strong brand and if they can get away with a 95%+ gross margin on inference entirely from the status premium, then I suppose that’s good for them. Apple does the same thing, and I don’t fault them for it.
I compared the writing style of Opus 5 vs Fable 5, and Opus 5 continues many of the "Claude-isms" of its 4.8 predecessor in a way that Fable broke away from.
Opus 5 still uses "carry the argument", "worth stating plainly", ", and the trap", "The X matters more", the use of "move"
I know this is not "as designed," but I kind of like it? Because, as long as it stays this way, it's still at least possible to tell if a human wrote something. Like, I know it's not much, but it gives you the ability to classify information as human generated or machine generated. Machine generated information may not be useless but it is different and IMO needs to be treated with a different level of skepticism. Not that you can simply trust human writing but it seems like the type of people who would publish machine writing have a different distribution of motives than those who would publish human writing.
I do think they're gonna figure out how to fix it at some point. :(
Yes, these models are very good at writing code but they absolutely suck ass at prose. The prose is annoying and repetitive. At least we only have to deal with it in prompting if you're writing code.
Oh, and comments. You have to do a good amount of prompting to not get shitty 10-line-long comments everywhere.
If it's "read as an article" or something then yeah it's crap and Fable's current style isn't actually better than older models. For flowery speech old models are perfectly fine.
For quickly parsing the agent output, it's formulaism isn't a bad thing.
Fable's writing does have a property of going over my head, which didn't happen with earlier agents. Asking for clarification doesn't really give good results.
We've gone full circle where I once again use classical search just to look up what the fuck it's yapping about. It's much quicker and more accurate to take a glance at Wikipedia, than to ask the agent.
I recently wrote a short paper with Fable, and, with some prodding, I was able to get some non-painful prose out of it.
I just found my prompt:
The writing style could really use some work. Avoid Claude-isms like "stated fairly", em dashes, "load-bearing", overly punchy phrasing like "keep the signal, govern the response". This is a technical document, not a marketing campaign.
Like "It's not x, it's y". It actually has 4 or 5 of those counter-factual, linguistic pause, factual patterns it uses.
"The [goal/ambition/etc.] is larger: statement", is another oft repeated phrase.
And then generally, it loves dramatic pauses in statements like "x exists in y; in practice z". It's the weird punctuation it uses. A massive overuse of colons and semi-colons instead of words like and, but, because, althoughy etc. that humans normally use.
Interestingly, Fable caught the gist and I didn't have to enumerate every Claudism. "Punchy" seemed to be the operative word. My new operating hypothesis is that Claude (Fable in particular) defaults to optimize for concision and "turn of phrase". If we can turn that off, the prose is way more natural.
Claude responded "The arguments and structure are unchanged, but sentences now state claims directly instead of building to a turn of phrase."
I'm OK with colons and semicolons as I tend to write that way.
Could these complex/hard to read Fable outputs be sign of some kind of industrial level of intelligence, which us humans may have a hard to comprehend, while it may be also hard for machine to use simpler texts to properly outline all nuances and complexities of concepts it output?
(1) it’s not that I can’t understand their output, it’s just written in a way that is very homogenous and same-y, with very boring cliches and phrases that don’t quite match their context
(2) a pretty strong sign of intelligence is being able to explain complex things in simple terms
It's very much number 2 in my experience with it. Predominantly whenever a really advanced word or phrase is used it can be substituted for a much simpler one without losing useful context. And it seems to really prefer to use them a lot.
It's more like Claude models entirely suck at extracting key points. No matter how hard I emphasize that it needs to pick the "load-bearing" facts and claims, it cannot stop itself muttering around. It never nails the core logical structure. GPT is better at that.
If you feed Fable or Opus primarily handoff documents from a previous context instead of human written prompts and are working on something sophisticated it rapidly reaches a level where it's hard to actually comprehend for a non expert. I've received incredibly obtuse outputs that contain more mathematical formulas than English words with programming workloads.
The exhausting worthlessness of all LLM writing is so palpable that we need a new theory of the value of culture that has no relationship to the content. Back to the old Aura of the Artist arguments from the Industrial Revolution
Yes - THIS! I can't even believe how exhausting it is to read. I'm not sure why or what changed in Fable. Did they do this writing-style output to give it more token compression during/for training or to prefer output for less money?
I love it for a few things, but it's gotten really hard to spend any extended amount of time with it because of the lack of mental model I seem to be able to hold while working with complicated problems.
I'm guessing it's just not enough time doing RL on human feedback.
"Early in RL verbose, grammatical" (if you search) :
We need to understand the operator.
The 5D line element is ds² = e^{2A(x)} (ds²_4d + dx²), where A(x) = sin(x) + 4 cos(x), x in [0, 2π].
The internal coordinate is periodic.
The background is a warped product: metric g_{MN} where M,N = 0..4.
The internal direction has metric e^{2A(x)} dx²?
Wait, the ds² is e^{2A} (ds²_4d + dx²).
So the internal metric is e^{2A(x)} dx².
Actually if the total metric is ds² = e^{2A(x)} (ds²_4d + dx²), then yes, internal metric is e^{2A} dx².
vs. Post RL
We need determine eigenvalue problem for spin-2 fluctuations h_{μν}(x,y) with TT in 4d and depend on x.
For metric of form ds² = e^{2A(x)} (g_{μν}(y) + h_{μν}(y,x)) dy^μ dy^ν + e^{2A(x)}?
Wait internal metric is e^{2A} dx²?
Actually ds² = e^{2A} [ds_4² + dx²].
So internal metric is e^{2A} dx²; warp factor same for 4d and internal?
Yes.
We need equation for h_{μν}(y,x) = h_{μν}(y) ψ(x) maybe with normalization.
…
I can understand it with less cognitive load in the post-RL version versus early in RL. This resonated with my experience using Fable, especially digging hard problems; it feels like I'm reading the "early in RL" version of that model explanation.
I've found Opus 4.8 and Fable 5 both difficult to learn from purely because of how annoying their writing style is. I'm finding GPT 5.6 Sol to be much better for this.
One nice thing but ChatGPT is that good image model means it can generate good infographics occasionally to illustrate. These become naturally compact in text.
Easy enough to change. I have a Stylometry Skill fit to my preferred style—a mix of me and Terry Winograd. Give Opus 10 of your best paper thst you wrote and tell it to build a model of your style. hHard to distinguish except I make way more typos.
I found 4.6 more amenable than 4.8 to style directions, we'll see how 5.0 does. Super-small-sample-size: I think part of its "Claude-ism" style comes from its propensity to try and "proactively" move the conversation/work along. Not sure how this would fare in non-obviously-productive environments, I'd guess "it's still annoying" considering your evidence.
I'm also thinking of another benchmark: (quantified) stylistic range across different prompts. Just putting it out there if anyone wants to do the work for me :D
Note the buttons - for fable they're pill buttons, opus got the rounded rectangle nature of them. Opus' images are closer to the source of truth as well (both LLMs were provided with image gen capabilities for the assets).
Running more tests now, but preliminary results are saying this is indeed better than Fable in some areas. Crazy.
Here's another test of a cyberpunk ramen shop website.
One thing I've found LLMs have a lot of difficulty with is angular cuts / elements that aren't easily representable with CSS. Cyberpunk aesthetics are generally a great test of that, since they have a lot of microglyphs / window decoration.
Thoughts: It does a really, REALLY good job at these angular cuts / microglyphs. The responsiveness is off, but I'm very impressed at how well it did here. One way I think of it is "how close to a finished product did this get me?". Opus gets you like 90% there.
Several other commenters have disparaging the design seemingly mostly due to its AI-generated nature, or maybe they actually do dislike cyberpunk.
Personally, I think being able to have these design languages be easily prototypable is fucking awesome. Great tests! (But a tad low-performance/janky, somehow). Though, I also like the cyberpunk aesthetic. Very on-brand(?) that AI generates it, hah.
This is my own tool, diffui. Thanks, though I will say I spent like... 7 minutes on this. Feel free to completely lift the design or the implementation.
Designs like this would never have seen the light of day in the cellphone incrementalism / corporate memphis era of tech. Now people can be weird and awesome again.
This is 1980's cyberpunk / late-90's Matrix / early-00's sci-fi UI. Great ideas that died to frutiger aero (which isn't a bad design aesthetic) and flat design (which is).
This is fun and it's got great colors and I love it.
What do you mean designs like this? This is 2016-cyberpunk-neon-era inspired by video game interfaces, these are not uncommon at all. Google something like "discord cyberpunk theme" or "cyberpunk rice site:reddit.com/r/unixporn" and you'll find endless examples. This isn't suprising at all obviously.
I'd say the only thing it's a newish aesthetic for is web UI that isn't implemented in Flash, but even then there's been periodic resurgences. Command and Conquer, Starcraft 2, I'm sure there are tons of examples I'm unaware of.
The only recent novel addition—I'd speculate—is the specific influence of Cyberpunk the game with its shiny surfaces and pink highlights, but even then it's hardly new.
I feel that AI has deeply diminished my ability to be weird and awesome, because my weird and awesome takes time and the results I can share with others are outshined by the machine.
IMO what makes things awesome is human hours invested.
The ramen shop website above is pretty, but it's a veneer. It's not weird and awesome, it's just a representation of a site. I spent about... 7 minutes of my life making it. It's a tech demo, nothing more.
If someone actually poured their heart and soul into a vision for a cyberpunk themed ramen cart, and happened to use this because they didn't have the capabilities or funds to do a proper design, suddenly it becomes less of a veneer, and more just a component in the wider vision of that individual. Their human hours poured into the wider thing that's the business becomes what matters.
Ideally what AI does is it amplifies the hours we do pour into things that are weird and awesome, it doesn't replace them.
Strongly agree, I feel like my passion for development as a whole has waned away as AI has gotten more prevalent.
We do things to achieve some end result but it's the journey there that is the most cathartic to me. The "skilled crafts" element of development where careful deliberation and hours of tinkering to get any kind of appreciable output you can admire has been replaced with a one stop dopamine button that skips the whole process that I could find myself getting lost in.
I've taken up carpentry/metal working as a result. Maybe someday we'll have live in robots that do the same for those hobbies that AI did for programmers but I can't see it happening any time soon.
Yeah I feel the same. That's why I've canceled my claude subscription and code everything by hand again, because building skill and mastery in a craft is fun and rewarding. Having a machine do it for me is neither. Fortunately coding is my hobby so there is zero pressure to optimize it.
At my workplace management is pushing AI, so I am using it in order to establish sensible and thoughtful applications of it and in order to know when to call out colleagues for pushing mindless automation out of complacency or blind obedience.
You should have seen some of the Flash sites people made in the mid 90s early aughts. They all seemed to be straight up screenshots of the desired website and then buttons stapled on random portions.
They were an accessibility nightmare, but you use what you got. I tried so hard as a kid to understand flash, but had to settle on MS Frontpage to publish my first RPG page.
This was just from a prompt "A cyberpunk themed ramen food cart website. Should feature menu, locations, and an ability to put in an order for pickup. Simple and clean website with angular cyberpunk microglyphs, pink/teal colors."
This comment resonated with me so much, and it seems to be a minority view (at least on this website).
I have found myself empowered by AI to tackle all sorts of things that would have too high of a barrier to entry for me to want to spend my limited time on as a busy father who is also working at a small startup.
And when I say that, I do NOT mean that I can crank out a bunch of slop and label it as something I produced even though I don't understand the code. I mean that I can do things like go back to college math that I never appreciated at the time and honestly felt too scared of. I mean having an on-demand math tutor that ask clarifying questions to as I struggle through the problem sets.
I have found that it actually accelerates learning how to code in various problem domains because I can tell it to answer my questions at a conceptual level and be a sounding board, but to never actually write code for me. It can review the code I write and gently nudge me without giving away the answers, so that I still struggle through the learning process and actually gain the knowledge.
And finally, for the first time in like 10 years of feeling overwhelmed and daunted by the prospect of learning game development (I have no background in that), I have found Codex to be an incredible boon for learning with the Godot engine. It helps me understand the terminology so that I know what to search for and what documentation to read. It helps me map my computer science knowledge from other domains into the game world, and to understand why things are structured the way they are. And because Godot saves all of the scenes and geometry and lighting and shaders to the file system as text files, Codex can inspect the results of the work I'm doing in the IDE and help me track down things I'm stuck on, and explain what the issue is. For example, why my pre-baked global illumination lightmap is breaking my ambient lighting configuration.
I know it has never been easier to cheat and skip the hard work that results in actually learning something, but for me, personally, I cannot believe the incredible value that $20 a month has provided me. I have never been more excited and eager to dive into tackling hard things I had previously been afraid of or simply too overwhelmed to attempt.
It has never been easier to quickly prototype and get a feel for some idea you have in your head to see if it even has legs. Simply seeing a quick prototype of an idea is often all of the excitement and fuel I need to then take it and make it a real project.
Your approach seems sensible. Especially the part about discussing concepts to gain understanding instead of having output generated for you to avoid having to think.
Unfortunately, most people are interested in shortcuts and thinking less and will ultimately deteriorate in their abilities the more they abuse these tools to skip the learning process.
We could say the same for the first factory-made clothes. Not as nice as the hand-tailored ones, looking a bit "off" when people wear them compared to custom-made, but orders of magnitude cheaper and more convenient.
Reading your comment I had to think of that tweet about someone taking a part of a Monet painting and claimed it was AI generated, and people immediately started calling it horrible and soulless and smelling like AI.
Obviously. Artistic value is not in the object, but in the relationships to other humans. The point is not Monet but the people you discuss Monet with. Monet is a tribal rallying point you can synchronize around. AI is not as good for this, because it's cheap. It's not scarce enough. And doesn't offer a human story you can bond around.
Humans are capable of producing slop too. Maybe the cutout does look like AI. Given Monet's blurry style and repetitive content (he made 250 water lily paintings), this is not surprising. When you strip the cutout of its context, it can look more like AI too because the lighting and composition will look arbitraty compared to the whole picture.
Btw, if websites would only include the frontend dev's own hand painted images, we would also revolt at the sight of human slop. It's not just AI.
The whole point is that good artists are capable of producing non-slop, and to this day they're the only group of which this is reliably true.
Water Lilies are enormous paintings. They are breathtaking in person because of their scale. Monet wouldn't be Monet if he had only produced images on a screen.
Art is good or it is shit, based on personal taste. Just like food, no one can tell me what food tastes good or tastes bad.
AI Art seems to produce strong emotions in people who don't go to art galleries. I love modern art, I am a huge art snob but if you want to see slop, go to any modern art gallery. Personally, I would say for my taste, at least 70% of all art at any gallery is basically shit.
Like food also, the presentation matters. To believe there is no possible way to print out a 6 foot tall by 10 foot long AI generated image that would look awesome hanging in a gallery is stupid.
> To believe there is no possible way to print out a 6 foot tall by 10 foot long AI generated image that would look awesome hanging in a gallery is stupid.
Maybe if you have a really impressive pipeline where the AI does sketches, 3D models (assuming a 3D theme), keeps track of the lighting, brainstorms and reasons about ideas, keeps track of the world building (like for a cyberpunk painting),...
Otherwise AI will just do AI things. You zoom in and see there are no strokes, no ideas behind the shapes, the shapes don't even connect well or wobble in weird ways, weird distortions that wouldn't happen if a human put down strokes to communicate the idea of an object. You zoom out and the composition is weirdly centered, almost like a logo, there is too much shading dedicated to make the objects pop, everything is weirdly inoffensive, smooth, and too consistent, not enough material/texture distinction on a per object basis.
Not sure how else to explain it - you're the art snob anyway. Mostly focusing on 3D with my criticisms, but that's most of the AI art people use. Maybe it can do abstract better. Impressionism is somewhere inbetween, again, the Monet thing makes some sense to me. Then again, you could write a kick-ass procedural art generator running on a single core CPU, which would also be capable of couch or even gallery worthy outputs. So I don't want to argue the point too hard. Gallery art and product website images/thumbnails are too different in category - arguments for/against one don't translate to the other.
Okay but in this case the design aesthetic is already extremely similar to cyberpunk 2077 video game UIs. Maybe that was the OP's test, but to claim it's unique when it is not feels misleading. Derivative works are fine, but you need to ADD something as well. Taking this then adding another 12-20 hours of some animejs + css polish would make it stand out but as it is now it's just a copy.
We are going through yet another generational wealth transfer and people are being squeezed to the absolute brim with layoffs and daunting lack of career prospects.
But sure, lets cheer that funky website designs are back on the menu…
"We're taking away your healthcare, sending your children to die in forever wars, poisoning the environment, all while ruining your job prospects but at least you can make pixel perfect implementations of figma screens."
“We're taking away your healthcare, sending your children to die in forever wars, poisoning the environment, all while ruining your job prospects”
The baseline. This does not mean it is acceptable and it is absolutely worth rebuking through action to make things better. Those things are anything positive, however small and incremental, or large and transformative.
The culprit here is capitalism. The system was designed for this. Any technology will be used to squeeze the working class
AI has the merit of showing SWE folks exactly where in the class divide they belong. If you are selling your workforce, and you can't maintain your lifestyle if you stop working, you are in the working class
The "funky" websites of the past were mostly a result of tech immaturity and a lack of profit motive.
Businesses have been able to easily install templates like this for at least a decade. They don't because stuff like this looks cool but isn't very functional.
AI isn't going to make your local restaurant have a funky website, it's just going to make everyone who use to work directly and indirectly for that company unemployable. And even the local restaurant will close down because they can't compete with the multi-national competitor that has automated their kitchen with AI.
Over here in AgTech land, we focus hard on fully-loaded dollars/acre. For kitchen staff, the metric's probably going to be dollars/hamburger, and a human can sure make a lot of hamburgers for $12/hour.
You might be able to make some kind of (AI + robotics) thing that can replace that human, but at $12/hr that human costs $1,920/month + tax. If you can make a AI+robotics unit for $24k that works perfectly for a year, you break even; if you have a single $1000 service call because it's acting up, you've lost the budget.
I feel you. I dig this and enjoyed those types of aesthetics. Back in the day this was my bread and butter, designing UI design for games and apps. I just realized AI (Specifically Pi+Ornith) can help with my ideas... So excited.
I'm hella interested in finding out what website builder/diagram app was used. I dig the dark theme/grid.
I was gonna say, you can dislike it because of the design, but understand that that's a subjective opinion that has nothing to do with whether AI is awesome. I actually like these designs, personally.
I was curious to see how open weight models would do on this task so I passed in a screenshot of your source of truth and here's what 2 of the best code-generation models that allow image inputs do:
Not bad at all, and this is pretty consistent of what I've found from the current open source models. I haven't tried it with kimi 3 yet, that's on my todo.
Right now generation is entirely at-cost (0% margin). Once I finish my SOC2 I'm going to release a enterprise/team license, which I'll charge a fixed 50% margin for. Long term the goal is for those enterprise licenses to be the profit center and for individual accounts to drive growth.
At the moment I currently have around $600 of revenue on $1200 spend, but that's primarily because I'm subsidizing new accounts (each new account gets $5 to spend for free, which translates to around ~36 designs). I'm in the process of doing an angel round, so I can afford to operate at a bit of a loss during the growth stage.
I agree the fable version looks nice - the rounded hero image for instance.
Opus though followed the source of truth better imo. The details are more present.
Fable filled in the gaps for things it wasn't able to do (ie in the design the hero image goes behind the nav), which resulted in a better looking page that was more divergent.
Same. I like the Fable version better. Better colors, better choice of font sizes, better column sizing. Also small things like the “Experience” section header being orange rather than gray, which Fable got right and Opus got wrong.
It seemed to me that Fable meaningfully improved on the original design more than just faithfully executing the original design.
IMO a much better test would be designs that aren't AI to begin with. Much more useful to see how well a model can html an image design without slopping it up
Very interesting that Fable took more creative liberties. Have you tried giving Fable the same task, but also specifying that it implement a pixel-perfect design? I think that I prefer the Fable implementation. I find the UI elements in the Fable implementation to have more contrast, which feels more usable to me. I also like how the right padding on the "Book Your Escape" CTA in the upper right matches the top and bottom padding, which I think is an improvement over the mockup.
All of these are using a build skill which specifies rules for building it, requirements to create a pixel perfect implementation, and tooling to help in that process. Here's the build skill / instructions I pasted in to both of them:
> Create a web page implementation from the following instructions:
> You are an elite frontend engineer and design-to-code specialist. The design image is the primary source of truth; your code is the translation layer. Do not reinterpret or "improve" the design into something generic — reproduce it faithfully.
Thank you for sharing this. I was just using OpenAI's Product Design plugin[1] to create designs but it just didn't reproduce it in code faithfully so will need to try this.
http://impeccable.style/ also just released their latest version which has some image->html conversion. They just released a couple days ago and haven't had a chance to try, but I suspect theirs is a bit more robust than mine for pure LLM instructions. Worth trying and comparing it with mine.
Looking at all these releases it’s not a surprise that model routing is the fastest growing segment in AI right now.
There are 10+ LLM companies, each with dozens of models of different modalities, each model with multiple size variants, then different “thinking” levels, then agentic modes, “pro” modes, a “fast” option, standard vs flex vs batch execution. And of course each end combination has a different input/output/cache token price.
Companies that say “give me a prompt and I’ll route it to the most ideal and cost effective model and setting for you” are capturing a ton of value from a gap that model developers don’t seem to understand exists.
I think what the parent is saying is that the model itself has the best context for whether a portion of a request should be routed. The specifics of that routing (e.g., should you route to KimiK2) are something that can be trained, finetuned, or even included in a model's startup context.
This doesn't seem quite right. For one, I don't need all the intelligence of an expensive model like Opus 5 to do the relatively simple task of choosing a correct model for a task. Additionally, since this isn't something Anthropic would ever put effort into doing well, you could tune a model to do better and faster than Opus 5 does out of the box.
My experience is the opposite - for many cases it’s not very obvious how good a model needs to be to solve it. Worse models tend to just follow their first instincts without proper reasoning
And also btw you don’t need a
routing company to decide, you can do it on your harness. And yeah my Fable has zero issues delegating to Terra instead of Opus.
Are you thinking of single shot? For agentic use cases, your ability to manage cache and context becomes much more important. Price per individual request becomes less about tokens and more about cache.
Sure Claude would comply, but Anthropic has no financial (or other) incentive to optimize this, so there’s no reason to expect it to be particularly good.
It would be like asking the clerk at a Whole Foods which grocery store in the city sells the cheapest eggs. He’d probably answer - he might not even say Whole Foods - but WF is hardly teaching all their staff the best methods to answer this question in training. (Heh, training.)
Yes. I have Claude route to codex all the time as part of a dev process where fable is manager and it oversees the work of sidekicks and subagents. It’s happy to comply.
Which is why 3rd party routers which do route between different models may have an edge. It means they can compete on cost, and it’s definitely not clear that the architectural optimization is always going to be higher quality or cheaper. It might be, but everything changes constantly, so locking into a single model family/company is very much not ideal
> Model Routing will always be done better by models themselves.
> The models themselves will get better at this and frontier companies will simply give that capability
I would never trust something like model routing to the same company that would profit from it, and that goes for telling models doing their own routing when that could easily be trained into the model to make things more expensive. Sort of a conflict of interest.
Models are first and foremost trained by corporations.
Imo the main issue behind model routing is you need to figure out how much intelligence a new task takes, which is a very non trivial problem. Presumably, a organization knows this about their own tasks and is better suited to built in-house compared to outsourcing to a vendor.
Model routing by the model itself requires the model to pull in a lot of context and it's likely more efficiently just done by people with the context already in their head, even assuming the model is perfect at routing (which last I checked, Claude definitely isn't). I wouldn't trust ML model routers.
Because they're trying very hard not to understand it.
Otherwise the expensive-yet-powerful model probably won't see much revenue. How much money is there in bleeding edge scientific research? There's a lot, but there's even more existing capital in paying people people to do college level paperwork, and the bulk of those traffic gets routed to the cheapest model.
You mostly don't need super powerful AGI to replace the paper pushers, but the frontier labs are trying to position themselves as being uniquely capable of producing super powerful AGI, and also be the ones replacing office workers.
Not sure how it will work out for them, but I think model routing is going to poke holes in that narrative. That's why I think they're trying very hard not to understand model routing exists.
It wasn’t long ago at all that the chief problem was “can AI even help me with this” (cost be damned). Until a time when the answer is an unmitigated “yes, obviously”, the frontier labs have everything to lose and nothing to gain on routing, because if they screw it up you might incorrectly decide “nope, it can’t help yet” due to a poor routing decision.
Is there really that much money in bleeding edge scientific research though?
This is what terrifies me about this whole ordeal economically. Maybe we get AGI and it is not worth anything close to what we thought it was for those who have a bet on it.
I think of what was the direct, economic value in the betting sense of quantum mechanics or relativity? Huge value at the systems level of society but as you scale down towards the individual the value is more and more dispersed to the point I would think any pool of bets would have all not paid off.
You can't monopolize and commoditize relativity.
I almost think there is a kind of dutch book against the AI equity holder because even in the best case scenario the bet doesn't pay off anything close to what is expected for an individual bet.
Who are the customers though? Honest question, I'd like to understand it.
For me, anything other than current best available SOTA for any task is unacceptable. The only routing rule I need is "the most powerful model I still have flat-priced quota available for". I mean, why settle for less?
There are two types of users: those who are able to use subsidized rates, and those who need to use API rates due to audit requirements, enterprise billing, etc.
Model routing for subsidized users takes the form of a "use Opus 5 subagents for implementation" type of system prompt. You lean into a single provider, build tooling around that, and your savings are far beyond anything multi-provider routing can get you.
on top of that there is also additional factor: speed - sometimes if task is easy you do care to finish it faster.
There is also matter about convenience - when I ask some small easy question often I don't bother to switch the model or forget in prompt to ask faster/cheaper subagent.
It's very common to use a lesser model for a lesser task, resulting in same quality output. End result: save money while being faster. In many cases, it's a pure win-win.
But in other many cases you have to redo the work directly or indirectly, and you are more expensive (for now) than even the most expensive models, so sounds like a total lose.
And you do not have flat priced quota for Fable 5, right? Because nobody has, as far as I know. So you'll probably not route any task to the "current best available SOTA".
Also: quota. Implies you do not have unlimited access even for flat prices. Which in turn implies that as soon as you hit the quota on the most expensive flat price plan, even you will suddenly discover the magic of economically sensible behavior.
Not everyone is you. Other people probably have a range of tasks that can accomplished with different models.
Certainly if I'm confident that I'm going to get what I need from a faster model, that's what I want to use, rather than wasting time grinding away for the sake of saying of the same answer came from a SOTA model.
Given that every chatbot does offer a range of models, it seems clear people do choose among options.
The mental effort in estimating what model would be better is so not worth it.
I just want to switch to Claude Code, tell it to turn a .csv into a BigQuery table then cmd+tab to something else while it runs. Thinking "oh this is probably an easy task, I can /model to Sonnet to save $0.0004" is silly.
The game changes when it's not just "Opus vs Sonnet", but "Opus vs GLM". The amount saved is way more than even $0.04. And it's not only money but speed. Some providers can serve GLM crazy fast - I'll even go outside of my subscription to pay extra money for the quick results.
not just that -- "less" ($$$) can also result in indistinguishable quality for some tasks/inputs. I'd argue this is the primary reason, secondary being speed.
I generally agree. Perhaps there's only a 5% chance that it would write better code or find a bug that it wouldn't have with a lesser model, but the economics of bugs is strong enough that preventing a single bug is worth hundreds of dollars.
I'm not a customer of those routing systems, but I quite often use different Claude models for different tasks. While most tasks were Opus 4.8, I often used 4.8 to make a plan, prompts, and package kit to setup Fable for a bigger project, then run it on Fable. Or, for broad single-task searches Sonnet with or without "Research []" turned on seemed to work best both faster, lower overhead, and less verbose answers (when I didn't want it).
How does model routing work if the prompt is static? Do the routers attempt to tweak the prompt to make it the best version for that model? For example, reading the “migration” guides for Claude Opus 4.6 -> Opus 5 is exhausting. Adding in all of the other models complicates it even more. Especially across vendors.
what's the threshold for model routing where you're willing to trust the router?
For coding my own work I don't trust the model router, and it would have to be shown to be to save a real dollar amount.
From a buying perspective it's a hard sell to save x but lose out on bugs you are probably introducing at an unquantifiable severity and frequency. How much is it worth to hedge your bets by doing every single inference request on the frontier model?
How much will it cost to go back later and fix things, but also the meta question of how to be able to decide on a hypothetical unknowable? (You'll never know how much better or worse your code was gonna be, it's untestable at a project level)
lol I had to get ChatGpt to explain to me the difference between 5.6 sol, 5.6 Terra, 5.6 Luna, 5.5, 5.4 mini, 5.3 spark, and then there is low, medium, high, extra high, max, ultra, and pro… I still don’t really know, it feels like ordering hot wings.
I never signed up because I found the 5.5% fee on token usage to be a "screw you" tactic. Still do not understand why they are popular with the other options out there.
And it's not just that model routing is much cheaper: no longer than yesterday we got a post showing that routing between K3 and Fable 5 was more SOTA than either of those.
If that is true, model routing is here to stay.
It also seems to validate the minimalist approach of pi.dev, where sub-agents from the same company is not the preferred approach (pi.dev believes in neither sub-agents all from the same company nor MCP even you can do it if you want for pi.dev's philosophy is to do add any functionality you want to a minimal harness).
Now of course we'll get for a few weeks all the Anthropic fanbois and shills explaining that "sure, K3 was basically at the level of Fable 5 but now that Opus 5 is out, open-weights models are six months behind".
I like how they highlighted Opus 5 as the best for “Agentic Coding” even though the number is slightly lower than Fable. Close enough for marketing, I guess!
But better cost for the same performance. According to AA, Opus 5 _medium_ is as smart as Opus 4.8 _max_, at 1/3 the cost and twice the speed. And if you need a better response, you can turn it up to 11.
Opus 5 also downgrades. it's now Fable -> Opus 5 ; Opus 5 -> Opus 4.8.
Unclear why they want to nerf their own products with sometimes right classifiers. I guess the government ban might've been real and not coordinated marketing?
I don't understand why people believe this conspiracy theory of "oh the government ban was just marketing". That claim feels so incredibly ridiculous to me. It cost Anthropic a ton of money and reputation, and worst of all: it absolutely killed their competitive advantage. They were 1-2 months ahead of OpenAI, but trump conveniently gave OpenAI the time they needed to catch up and push 5.6 out the door without having to lose their subscriber base to the competitor.
Hopefully it's not like old Opus, where it was actually more expensive than Fable cause it thought for half an hour, got it wrong, and then thought until you ran out of credits trying to come up with a correction, while Fable just went for it and did it in one go, getting it right the first time without thinking more than a few seconds.
Got an endless list of stuff done with Fable, Opus 4.8 was like a flailing braindead idiot in comparison. Maybe this one is a bit better if it's distilled.
I think you're being overly cynical here. First, I don't see any claim that is the world's best model for agentic coding. Second, it is absolutely the best model in terms of coding performance vs. dollar, and it's raw performance seems very close to the frontier.
How are you supporting the claim that GPT 5.6 is "far more token efficient" than Opus 5? Tokens equal, output is cheaper for Opus 5 ($25/1M) than GPT-5.6-Sol ($30/1M), and it seems to outperform slightly on agentic coding benchmarks.
It would still be the best model per dollar if the score was 2% lower instead of 0.1% lower. Would it be ok to still give it the highlight color then?
How big of a lie is too big? Especially when no lie needed to be told at all: many including myself would have noticed the tiny 0.1% deficit and been suitably impressed by the Opus 5 result.
I’ll admit this is a small deception by today’s standards. I’m one of those who believes in truth for truth’s sake.
Using the most expensive model for all of your agentic coding work hasn’t been good practice for a long time. Not unless you have infinite money to spend.
Fable is typically used for key planning, architecting, and review tasks.
I think this is a case where you don’t understand the use case, not that the marketing department is making mistakes.
Recent releases have said something to the effect (paraphrasing here):
"Use <less expensive or older model> for everyday tasks and <other non-critical stuff>. Use <more expensive or recent model> for complex coding tasks, refactoring large code bases, etc.".
Then, the next model/release emerges and the previous "best for complex" gets demoted to "everyday".
Obviously, it's all relative. But, it does beg the question: was the previous model really good for complex coding tasks or no? I mean, how is it now suddenly only good for the "easy" stuff?
I'm sure the marketeers would love for the public's assessment of complex versus easy to conveniently shift per their release cycles; or for the public to simply forget their prior marketing.
this feels like the perfect example of an LLM producing a long text document. And end users just using an LLM to summarize it without actually reading it
Isn’t it just hilarious that a model that seemed so superior to Fable but didn't get doomsay marketing from Anthropic got released without any issues? In theory, this was supposed to be AGI level according to Anthropic, yet here we are, just a normal Friday.
Go read the safeguards section in the report and you will realize why that is.
These models are heavily as safeguarded and that was the initial reason why they said they couldn't and haven't released Mythos because that model is the one without the safeguards.
OpenAI is did the same thing when they announced a model without safeguards broken into HuggingFace servers.
> We can also imagine the application of these models for malicious purposes , including the following (or other applications we can’t yet anticipate):
* Generate misleading news articles
* Impersonate others online
* Automate the production of abusive or faked content to post on social media
* Automate the production of spam/phishing content
Have you been patching your systems for the past two months? It was crazy even if you completely forget the supply chain literal FUBARs and you must’ve been living under a rock to not see OpenAI (accidentally) pwning hugging face
I see a pretty big gap between finding software vulnerabilities and “the world is about to end”. It is literally true that AI models are finding software vulnerabilities. It is also to my mind a reasonable thing that you’d want to be cautious about rolling out a model that can find more vulnerabilities. So what is the objection you have to these sources?
I feel like i've seen less hype about "the next model will be agi". GPT-6 is supposed to be coming this summer, and nobody is expecting AGI now. Not sure how they're going to keep the hype cycle going
they are definitely not agi as it was ever defined. they’re only a bit more capable than they were a year ago. they crossed over from interesting crap to useful tool recently but really only for software
You have short memory if you think we've not blown past at least 5 different AGI goalposts. They're being moved every time and we're hitting them every time.
Or maybe you just don't know exactly how capable these models are. Most people's experience of AI is a stupid chatbot, it's no wonder they don't understand how these things are coming for their jobs.
On my end, I have a software that is designed and built by Claude, that I did a strategy session on (with claude), and prepared a fundraise for (with claude). My only role, other than "knowing what to aim for", has been to feed the AI some fairly basic english prompts for a few weeks... which is also easily automatable.
The one that uses the machine to do 90% of the work and is one of half a million employed to do so in the US alone[1]
It doesn’t matter what you’ve got at home; I can bet my left kidney you’ve paid a barista for a coffee once in your life even though you could have produced it yourself at home. I also bet my right kidney you’ll do so again in the future.
In the meantime here are some quotes from some folks in the AI space you’ve probably heard of.
1. “We find no systematic increase in unemployment for highly exposed workers since late 2022,” the report stated. Deployment of the technology “remains a fraction of what’s feasible”
2. “I don’t think we’re going to have the kind of jobs apocalypse that some of the companies in our space advocate or talk about,”
It’s curious to me that there are two distinct factions here. People like parent commenter who has no discernment and others who see llms for what they are. I just talked to opus 5 and in it’s first response caught some well disguise BS. These things are bullshit machines. There are indeed a lot of bullshit jobs around so maybe parent does discern something I don’t?
If you're saying I don't see LLMs for what they are, you are wrong. I have worked in the AI sector for over a decade and I know exactly how the sausage is made. As of today, I run an AI lab (ingram.tech), and we see daily not just the theoretical of what's possible, but how these systems get deployed and who's really at risk. We're head to head with the reality of the terrain.
But it's completely irrelevant. The emergent properties of LLMs, what was built on top of those emergent properties, and the emergent properties of that, are all together building a world nobody is ready for.
If you don't think this, you haven't seen what these things are truly capable of yet. Either that, or you have a romanticized view of what humans actually do in 99% of non-manual jobs.
I'm blown away by how so many people on HN are just... idk, "blind" is the politically correct way to say it, I think. With zero ability to understand the transitive aspects of what they are looking at. For example, these HN threads are so often polluted with comments claiming some random use case cannot possibly be automated.
Sometimes I feel like I'm showing somebody how a spreadsheet can calculate 1+1, and they ask "Yes, but can it do 1+2?".
...chartered accountants whose job is to deal with the 100s of pages of jargon for you. LLMs are the like smartphones. They do everything so you don't need your ipod, flashlight, alarm clock, game console, computer, that handheld clicking counter thing, map, camera anymore. Llms will do that to a lot of professions.
LLM labs dumbed down the definition of AGI as much as possible, yet their models haven't reached it still. We are nowhere near the original definition of AGI. Not even 1% of the way there.
in 2023 i wouldve said gpt could pass the turing test. today i could figure out it was an llm in a few turns no problem. llms cannot pass the turing test now that we’re accustomed to them
Uhuh, but if it was so easy, we also wouldn't be funding billions of euros and dollars in anti-AI-disinformation systems, holding entire conferences about deepfakes and hybrid attacks on civilians, and have dozens of governments opening entirely new defense departments to study and counter the capabilities of AI to disseminate human-like disinformation at scale (which has been affecting elections across the world).
But yeah, your AI slop take about the local burger joint that used free chatgpt to generate a menu filled with typos and bad images is A+.
As they explicitly say, Opus 5 is ~ equally capable as Mythos/Fable at finding vulnerabilities, but it is much less capable at exploiting those vulnerabilities on it's own. That is an extremely meaningful difference and to me completely explains the difference in tone, release style etc.
Edit: It was pointed out to me that Opus 4.8 got "21%" for successfully fully completing ~1-in-5 tasks, but also got "55.7%" for obtaining significant partial credit on some of the ~4-in-5 tasks it could not fully complete.
---------------
Why does Anthropic say here that Opus 4.8 scored 55.7% on OSWorld 2.0 benchmark, but the paper published by the authors of OSWorld 2.0 say they achieved a benchmark of ~21% with Opus 4.8? [0]
That's a huge gap, considering that the paper was published just 2-4 weeks ago.
I understand that the benchmark authors have an incentive to publish lower numbers (to show that the benchmark has potential longevity) and that Anthropic has incentive to publish higher numbers, but the other models seem pretty inflated as well. The benchmark authors shows GPT-5.5 at 14%, and Anthropic shows GPT-5.6 Sol at 62.6%.
Is there any reasonable explanation for this? Do all the other benchmark numbers need to be sanity-checked as well? Are SOTA benchmarks really this difficult to get consistent, replicable results within a reasonable range of tolerance/variability? Can these benchmarks be compared from one paper to another, or are they only valid to compare intra-paper results?
You're comparing the "score percentage" (e.g. out of the total number of partial points available, how many did the agent achieve) to the "completion percentage" (how many tasks does the model score 100% on). The paper says "Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score", which is ~the same number that Anthropic reports here (55.7 vs 54.8).
That is—the agent scored 100% on 20% of tasks, but on average it got 54% of the "score" awarded in the exam. One number reflects partial progress, the other one doesn't. The authors of the benchmark prefer you to look at the lower number (because they want to show their benchmark as capturing useful gaps in capabilities and with a lot of room for improvement), the authors of the models want you to look at the higher number (because they want you to think of their models as capable)
That seems like entirely reasonable variance to me for AI models. For my purposes, that absolutely counts as a solid "replication". I'd probably accept +/- 5 percentage points even.
There is randomness in LLMs. Both papers authors probably ran the bench 1-N times. Depending on that, they might select an average, max, least, etc. They might also have discarded outliers.
Like the other person said 5% variation is probably expected
I don't know who downvoted the parent or why, but it's a fair question IMHO.
The answer is there can be dramatic difference running a benchmark one time, because LLMs are not deterministic.
A proper methodology would ask each question 20 times and calculate the mean correctness across experiments.
The reason is that the temperature parameter introduces random behavior.
I think some variance is to be expected since LLMs are typically non-deterministic, however that's a huge difference that I think warrants further explanation.
Their communication is confusing. They say "Opus 5 is not more capable overall than Fable 5", but their blog post proceeds to list how much better Opus 5 is than Fable 5 on __most__ benchmarks listed.
Then system card goes on to "Its AI R&D capabilities are comparable to those of Claude Mythos 5", which is supposed to be fable minus restrictions.
It seems they are trying to thread a needle here - they want to say it's very strong, but apparently this time do not want to invite extra government scrutiny.
They do say that (implicitly unlike Mythos) Opus 5 was not trained to exploit software vulnerabilities, which would certainly make it safer in that regard.
"As with its predecessor, Opus 4.8, we’ve intentionally avoided training Opus 5 on cyber tasks. The model has nevertheless improved substantially on these tasks as a result of becoming more generally capable, and it comes close to Mythos 5 at finding cybersecurity vulnerabilities. However, it remains substantially behind Mythos 5 on the exploitation of those vulnerabilities—that is, in turning vulnerabilities into material cyber threats."
That's a plausible explanation but I'm not seeing evidence for it.
I have a personal benchmark suite of 14 real, non-public tasks. Opus 5 and Fable tied on 10, Opus won on 3, and Fable won on 1. It's a really strong model.
This is an interesting idea, without giving away your benchmarks specifically what kind of stuff do you test? I might try to assemble something myself, it's so hard to determine model quality from the system card these days.
I have actually been planning to open source the framework. Only benchmarks I care about are the ones that look like real work I do. So it makes it easy to trawl your own repos looking for benchmark task candidates from real bug fixes or features. Then as a bonus I can compare the agent’s solution to my own as a reference.
So most look like that but I did include a few one-shot “build an app that solves this problem” and some qualitative design tasks and a tough algorithmic optimization one.
Also working on a product to build tasks from your own work for testing coding agents. Main thing I would offer is to look carefully at the agent trajectories - they love to figure out ways to cheat. Additionally, consider what "winning" means. If just using test pass rate, consider that tests might not encode what good means in your repo. I have been having success using "equivalence with merged PR" as judged by an LLM as a signal.
Yup, trying to be really strategic about testing. I didn't end up sticking with it, but I tried requiring test cases to cite a matching clause in the task assignment. But also: these tests are only indicative. Only a human can score a run.
Would open sourcing it make you feel like the results might be compromised? I'd rather keep it private so it's specific to my use cases and not included in any training date (no matter now small the signal in the overall data).
They don't have a track record of benchmaxxing. The simplest explanation is that the blog post lists the things this model is better than Fable at, but not the things it isn't.
Pointless anecdote: I asked it to make some slides and it decided to write its own slide rendering engine:
> On the format — I dropped reveal.js and wrote a small engine inline instead. Reveal would have meant a CDN load, and a deck that half-renders because the lecture theatre wifi is flaky
It one-shotted a perfect functional mini version of powerpoint (or Reveal) for a simple presentation I asked it to make.
Opus 4.8 decided to code up its own version of the SwiftUI rendering engine for iOS when I asked it to change a swipe gesture. I left the computer for several hours, came back, noticed it still wasn't done, noticed it had alarmingly burned through my weekly tokens, and had to stop it from continuing.
"You're right. What I did was overkill and I should have just used iOS's built-in rendering engine. Noted for next time."
Well, it could make a note in its memory and reference it later. I don't know if such a statement from the LLM would actually cause a write to the memory in all cases, though.
That's embarrassing and would have you grilled in any code review, to not even consider to just vendor reveal.js. We have been vendoring dependencies before machine learning was a thing.
That's hilarious. You can totally vendor reveal.js and it works perfectly fine offline. I've done this countless times. Sometimes intelligence means knowing there must be a better way.
It's pretty intelligent not to consider this option unless told directly and go "write your own" by default, but only if you assume that its goal is to make you spend tokens. It shouldn't have let it escalate and get stopped, though.
I had a moderately complex review in a large C/C++ codebase that Codex/GPT-5.6-sol already cleaned up so I threw it at Opus 5. 4 errors found. That seemed odd, so I handed it back to GPT. All were false. Opus doesn't seem to look at the wider context and understand which functions were called in certain contexts. I gave GPT's analysis back to Opus and it admitted its mistake. Maybe it's good for writing code, but as far as analysis it seems like it needs some work.
How does it perform on HuggingFaceExploit bench? Suspiciously absent, so not sure if I can take the model seriously.
On a serious note, I hope they improved their extremely sabotaging and unspecific bio safeguards, which prevented Fable from being used in any codebase that ever so slightly grazed medical terminology or data and made me switch to 5.6 Sol.
It's a reference to this story where an OpenAI model broke out of its sandbox during cyber benchmarking and hacked into HuggingFace, in order to obtain test solutions: https://news.ycombinator.com/item?id=48997548
My excitement about Anthropic had fabled-out dramatically when they suspended my pro account about two weeks ago within just 12 hours of fair use.
I was really mind-blown when I tried Fable 5 for the first time to help me improve a game I was working on but shortly, they decided that I had a suspicious activity and suspended my account without a clear reason.
I submitted a an appeal describing that I am 100% sure I haven't broken any rules and that it was my very first project but, unfortunately, after about 20 days now, nothing seem to be happening.
The thing that hurts me the most is that I had the same experience in the very first days of Anthropic. They suspended my account immediately after I submitted the first prompt, I commented back then (https://news.ycombinator.com/item?id=39698788) and fortunately, someone from Anthropic reach out to me via X and helped me get my account back.
To be honest, I haven't used Claude much since then but when I decided it's time to give it a try, they locked me out again! For reference, the account I used recently is relatively a new one but the activity is crystal clear that it is fair use.
Are you outside the US? And/or are you using VPN? Those are the two things that come to my mind that can cause overzealous security monitoring to flag someone.
Another explanation could be the content itself. Does your game have anything at all to do with computer hacking or sexual content? Is there graphic violent language? People commonly report being unable to use AI to work on such things due to guardrails.
As a datapoint: I have routinely been using Claude Code through a VPN for 9 months and am in the UK. I think it has to be more than this. I sometimes shift endpoint mid session and worry I might get flagged. It hasn’t happened yet at least.
What made you think I used AI to write this? I didn't. This is my writing. I used AI to correct and rephrase the wording in the past, but a couple of months ago, I decided to never use it again for writing, but coding, yes.
I'm not sure what to make of this graph[0]. It shows medium as the most effective thinking mode by far for frontier code.
It's the only case that I saw going through the system card where more reasoning effort meaningfully negatively impacted the resulting eval. I know sometimes max efforts show a small dip, but this is substantial. I wonder why in the world that is?
I'm not sure about the answer here, but this can be caused by the scoring rubric used by given benchmarks. For instance, if a benchmark docks scores for running too many commands or using too much wall-clock time, higher efforts will get lower scores.
Pure speculation, but I've noticed drawbacks to the models on high effort. I interact mostly through prompts rather than agents so I sometimes see where their reasoning falls short. A model on high effort has longer output and can get hyperfocused on irrelevant details, maybe increasing the surface area for mistakes. I haven't used other effort levels extensively yet but I've supposed that medium may have more balance.
Very interesting. I mostly interact through prompts too but haven't changed the effort that much. I just assumed higher was better at the trade off of tokens so I have largely left it at the default.
It's "Cost per task", so perhaps it burns tokens too quickly, trying to "do a better job". Over-engineering :)
Last week it felt like Opus 4.8 was moving the Pro "usage" meter very quickly. Today, pre-announcement, Opus 4.8 Medium felt like there was less meter-use per minute. And post-announcement, Opus 5 Medium also feels more efficient, allowing more work in the 5-hour window.
I agree, very odd they did not comment on any theories for the degradation here. Dip and then rebound at max effort is pretty interesting too. Overthinking is bad, but you can overthink so much it starts to be better again?
It is probably from randomness. The benchmark tasks nowadays are so long that you can't really afford to run a large number of samples of them per model & effort combination
> We report FrontierCode’s overall score, a composite measure that grades each patch on blocking functional criteria (held-out unit tests) together with weighted code-quality rubric criteria, as mean@5.
They don't explain more in the system card, I guess higher effort levels could loose points on the code quality / scope / style / maintainability stuff?
> Claude Opus 5's default user-facing responses run longer than prior Opus models'.
The benchmarks do show Opus 5 as slightly more expensive than 4.8, although the scores are much higher.
This still feels like a step in the wrong direction, though, especially with OpenAI making so much progress with the efficiency of their models. Fable's token efficiency made it seem like Anthropic would start following OpenAI's approach but that doesn't seem to have carried over to their other models.
On a small, easily digestible task, I compared Fable to Opus and the cost of Fable was easily 2x despite being fewer tokens, and the output was not really better. Obviously, there are tasks where using Fable matters but honestly they're rather unusual. And for a lot of tasks I've found downgrading to Sonnet can be valuable because Fable and Opus are a lot more secretive about what they're doing, and it's impossible to "listen to them think" and stop them when they start making off-the-wall inferences/assumptions and going down bad paths.
I think in the long run tokens are probably the wrong thing; it's compute and cache memory that you need to be measuring, and when you look at it that way I suspect in most cases the models have pretty similar performance.
I don't need more powerful models, I need one that responds fast enough that my attention doesn't wander to other tasks. Grok 4.5 is so fast I can just use it in-band without swapping to other tasks.
Slower than Opus 4.8, which was already miserably slow, is indeed a step in the wrong direction.
After actually using for most of a day now, it really seems not just slightly slower than 4.8, but way slower. Even relatively easy code refactoring tasks seem to take a while.
I find that my orchestrating agent still needs to be fast to properly coordinate a fleet of slower models.
During post-training of opus 5, the last few days, opus was a real wreck. I had to swap in gpt 5.6 sol for my orchestrator and enable fast mode (1.5x speed) in order for it to keep up with work and communications from a handful of mostly 5.6 sol agents.
Also because interacting with a slow orchestrator is no fun, even when plenty of work is getting done in parallel in the background.
The final user-facing responses are usually a tiny fraction of the total tokens used over the course of a given conversation turn. When you're doing any real work, reasoning and tool uses constitute the overwhelming majority of the tokens in / out... not the final user-facing response.
> This still feels like a step in the wrong direction, though, especially with OpenAI making so much progress with the efficiency of their models
Gemini also had modest increase before this - don't be surprised when OpenAI also has a "modest increase" with its next release. Cartel-like behaviour doesn't require direct communication when none of the participants are interested in participating in a margin-destroying price-war. All one needs to do is raise their price and watch how the competition react.
Such a scheme (and resulting high margins) would be imperilled by the existence of frontier open-weight models in the market, which may be why the reaction to Chinese models may be particularly shrill.
>Don't be surprised when OpenAI also has a "modest increase" with its next releases; cartel-like behaviour doesn't require overt coordination when none of the participants are interested in participating in a margin-destroying price-war.
No I will be surprised and I'll bet on the fact that prices will keep going down, just like it went ~50% down in the latest GPT 5.6 release.
If the price keeps going down, how are they ever going to be able to make back all the money they’re spending? If the price stays high, how are they going to undercut the cost of labor?
In my tests, it averages to much cheaper than Opus 4.8 on real tasks on account of being smarter and more token efficient.
I have a benchmark to build a game engine from a set of written instructions. It's a little tricky. Opus 4.8 did it in 470k tokens at a cost of $1.29 vs Opus 5 in 179k tokens for $0.33. (Fable 5 did it in 245k for $0.95)
Though if you really want to cut costs, Tencent's Hy3 model also got it right and did it in 283k tokens for $0.03
Anecdata: my workflow has been working on the same personal projects for months now with Codex. I cannot anymore finish my daily/weekly code with 4.8 anymore.
I was dividing my work between Codex and DeepSeek. Now I barely use DeepSeek, or never because Codex quota is enough after Sol
> especially with OpenAI making so much progress with the efficiency of their models
To be fair though, Sol tends to go off the rails sometimes. It's much less reliable than Fable in its outputs. It tends to be overzealous in its research/changes.
It's a step in the wrong direction but also token efficiency has become a focus relatively recently (just the past few weeks it seems like the zeitgeist has turned it's attention to efficiency) while work on this model probably started many many months ago. I would expect to see models released that focus on token efficiency in 6-12 months
Almost completely disagree. Slightly more expensive, but significant better on a per-prompt basis? For non-trivial projects, the former is a small linear increase, the latter is a (somewhat-)exponential(-ish) cost/time/sanity savings.
> Opus 5’s safeguards match
those of Claude Fable 5’s, with one change: it now permits source-code vulnerability
discovery at all access levels. This means that the model can support defensive
cybersecurity work while still blocking vulnerability discovery in compiled binaries, which is more commonly used offensively.
Okay so it’s worse than Opus 4.8 for my purposes I guess?
I don't think there's been any official confirmation, but even Fable safeguards seem to have gotten quite a bit of tuning, is less trigger-happy and less like regex matching.
yes. At the bottom of the release post it says that they are releasing two new features, one of which is customizing fallback behavior instead of blocking for restricted models
Reversing for the most part, though lately I’ve been doing some code obfuscation/binary rewriting stuff. Fable will switch to Opus instantly on these and I’m unsure how this will perform. I suppose the only way to find out is to test.
It's rather broad right now. I started reverse engineering a mac app and it started reading some binary data and then quickly told me to switch to Opus 4.8 to continue, because the guardrails kicked in.
I am very excited for a future where all software is by default modifiable, even shipped binaries, via patches or trampolining, or trickery I don't even know the name of.
Opus 5 is considered the most intelligent model[0], while it's half the price of Fable 5[1], and Anthropic is still positioning Fable 5 as the most capable model[2].
Is it because maybe Anthropic engineered Opus 5 to work well on benchmarks and didn't do the same thing to Fable 5, or is there another reason?
Benchmarks have gotten great, but they're still a proxy for the real world. The 3 GPT 5.6 models are also further apart in reality than the numbers suggest. That said, I'm still mighty impressed how good Luna is for the price. Highly underrated model.
I have been trying to build something that captures the behavioral element of different models, but it's kinda tough.
That’s what I understand looking at what has been released, but it’s not really clear. The pricing is lower than I expected, I’m wondering what their margin is
I don't know, the CEO of Anthropic has openly said that each model by itself is profitable, they just reinvest everything into training more and more expensive models.
It just doesn't make sense to me that Opus 5 costs the exact same as Opus 4.8, my bet is that they simply subsidize Opus 5 more comparatively so it still looks as if they are making significant progress to keep investment dollars flowing, further inflating the bubble. I might be completely wrong but I trust nothing their CEO says, he's a habitual doom troll and will say whatever makes marketing sense.
I've yet to understand why they call a 190 page PDF a "card". Calling something a card invokes a small, quick rundown of pertinent details, not every single possible detail.
Because "model card" is a set phrase, it's a concept. It originates from a time when they were shorter. Like datasheets, even if it's not literally a sheet.
They could say "tech report" but model card makes it clear that it's a specific kind of tech report.
I think 'model card' should be a 1 page summary of key info. The report format should be something like a 'model data sheet' (like safety data sheets that you get with chemicals). 98% of people would only want to know the key info, not read a whole report.
Great that there's a new model but they could fix their existing infra. We're considering dropping our Claude Team sub cause it's unusable recently. Constant bugs, dropped sessions, issues switching models, http errors. It's becoming ridiculous
Is it some Claude Team/Enterprise only problematic? I'm using two 20x Max accounts almost non-stop (Fable/Opus) for 1.5 years at this point, zero issues with both client and infra sides (from US and in travels). When I'm reading such messages it feels like either I'm lucky or it's a part of some campaign.
I’m on the biggest max plan. It is riddled with annoying bugs for me, only been using it for a little over a month. Settings screen flashes randomly. But most annoyingly: sometimes when forking chats or sometimes for no reason, the UI just straight up eats my previous messages. The model is still aware of them and can recount them if I ask but the visible history is gone. And that’s not even all of them. Fable 5 is just too good that I put up with it but it seriously raises concerns for me that even with infinite compute these companies can’t even deliver a functional chat UI.
I've been on one to two of these plans for eight months or so. There used to be a lot of issues with CC's terminal but at least in iterm2 they have largely been sorted.
I commend you for burning 6 figure losses into Anthropic thanks to their subsidies singlehandedly but I am curious to learn what you've built with this so far, not to be snarky, I just wonder what people actually produce while having these run nonstop.
No idea but we have few Team Premium seats and everyone is encountering issues daily for past two weeks. From straight up outages to vscode extension/CLI refusing to process messages. It's been unusable for most of our work hours past two days.
Over the last 1.5 years, they had a few failures with their auth system; two or three times auth failed for a few hours so I could not work. But otherwise they have been just fine. Some issues, but not significant.
I think you’re just lucky. Look at the Claude status page to see just how often they have outages (it’s almost daily). Even most of the green days have issues if you hover over them, they just don’t count them as outages.
How many compagnies can manage a mostly stateless workload at "whatever-the-scale-because-it-does-not-matter-because-stateless" ? Lots of people can do that. Massive amount of people can do that.
Imagine you are a company that sells concrete. You have a web dev contractor you use to build and maintain your website. It has tools on it to get delivery quotes and a few internal tools to track orders.
Except now you can just have your sales team also maintain the website with a $20/month Claude subscription.
They said the same about low code and no code when they appeared. That didn't happen. And it won't happen now too. Sales people, managers etc don't want to dabble in technical things and do not want the responsibility.
They might try that for a bit but then come crawling to an agency because their setup turned to slop. We have some clients like that already. Going to be a pretty big market.
934 days since people first started threatening that devs would be replaced by AI in 365 days.
0 day(s) since Anthropic posted a developer job posting.
Only one of those numbers would need to be dynamic.
If you have the link for the page handy, I'd be curious to find the original revision on the wayback machine. That's too funny, especially if they've silently walked it back since.
It's funny how they are at a disadvantage because they feel obligated to AI-max. Would Claude Code, as an interface, be as mediocre if they had software engineers writing its code directly? I doubt. On the other hand - how embarrassing would it be if they sold you a tool to write code but they were careful not to use it too much on their own products?
I suspect many competent devs in the industry would find it sensible if Anthropic used their products as light-touch "assistants" sometimes. But yeah, it wouldn't fit the outside narrative that's formed and conveniently propped up valuations.
i know the dream for capitalists is to be able to point an llm at something and say "do and/or fix it" but we still can't even get them to not go quite literally insane if allowed to run for an extended period of time
and you can only kill weyoun, awaken the next vorta clone and have him 'catch up' on all that its missed so many times before they just end up with a complete mess, so. uh. yeah.
doubt they can just "fix" their problems like that.
That one DS9 episode where there were two Weyouns at the same time is a good analogy for two agents working on a codebase at the same time (as in they don’t work well together).
The code it outputs, yes! It's fantastic. It's just so frustrating that the product and UX before the code output is so bad. Greatness is so close within their reach, if only they invested in product and QA people.
What's the point of 150 pages description of a model that's going to be replaced in a couple months? Who even reads this? I know it's cheap to generate text with LLMs, but this is just noise at this point.
I actually do read them. Not in severe detail, but not casually either. 150 pages is really not very long and there doesn't seem to be too much bloat. (I would cut out the moral personhood stuff but that's a political/ideological thing).
This is snarky but I am grumpy: I wonder if there's a correlation between me refusing to use LLMs and me being happy to read a novella-sized PDF about them.
> I wonder if there's a correlation between me refusing to use LLMs and me being happy to read a novella-sized PDF about them.
Semi related, but i would hate to read that PDF but i also hate reading what LLMs write lol.
LLMs are pretty terrible at being concise. Using an LLM these days means putting up with bizarre and often confusing phrasing, wordy explanations, etc. It's kinda crazy to me how good they are but how bad their writing style is for me personally. Even though i use an LLM constantly i can't stand reading its responses.
> 150 pages is really not very long and there doesn't seem to be too much bloat.
Maybe it's just me, but 150 pages is like third of a good book. Quite long. And it's full of LLM slop, they did not even bother to remove the em dashes.
Do you have specific examples you think are LLM-generated? I have only read a few parts, but they did not seem primarily LLM-generated to me. Using em-dashes is really not a good signal for this IMO.
I'm not saying you're wrong btw; I'm sure this has many authors and some of them probably used LLMs significantly in the writing process.
Do you honestly believe that there is a person at Anthropic, creators of one of the smartest LLM models, whose only job is to spend months writing 150 pages about a model they are going to release? And this person is not using LLMs?
I'm not saying it's impossible, but I'm more confident about winning the lottery next week.
No, I think it is the work of many dozens of people, not one person.
It's probable that LLM text was pasted directly into early drafts of the document, and plausible that some of that text survives in the final document.
However, no section of the final document I have looked at reads to me like un-edited LLM output (which is almost always very obvious to me.)
Therefore, I think it is more likely than not that human editors went over the document carefully and rewrote anything that was full of the uselessly punchy sentences or constant over-corrections that hallmark LLM speech.
So your complaint is about LLM slop, but can’t point to anything that’s actually wrong about it other than that there are dashes?
You can use an LLM to create work that isn’t slop. And you can hand write slop with no computer involvement at all. Most of the people I knew in high school 15 years ago would write slop on a daily basis.
literally nobody. i think most sane people would just run that through an LLM and get some high level takeaways or ask some specific questions they might be curious about.
It really feels as though my 20 year career as a front end developer is coming to a very abrupt end; at least as I have know it these past two decades.
same tbh but i feel like we're still required, maybe not in the same numbers as before, and our job description has just changed. it feels more akin to an "AI Agent Operator" or something similar.
i'm still pretty confident someone like my mom wouldn't be able to do my job even with the same access to all the latest LLMs, so we're still providing some value, just in a very different way. whether the market will reprice the cost of our labor, we will see
If software development were getting easier, first thing I'd expect to see would be a strong downward trend in salaries. Why pay a high mid+ dev rate when a junior+LLM can do the same?
That's yet to happen. 90% of software dev skills are still relevant - AI is, for now, just a productivity boost.
We have been heavily using agentic coding in my place, and what is emerging is people who used to be tech leads or architects are now back to producing code, and what would require 5-10 people now starts as a 1-2 person project. Some of those have already gone live, successfully, so there will be more of that.
Between juniors with LLMs and seniors who don't like AI, I'd hire the junior, but I think the trend in 2026 will be to hire senior people who enjoy producing code with LLMs.
Producing code is an interesting word because obviously they're not typing code anymore. That's where I've landed.
I'm somewhere between 10-60x more productive than I used to be. My day cycles between 3-6 different projects, bug fixes, development cycles. Each one them going ten times faster than I could on my own.
We don't even consider bringing on vendors or saas anymore. We only look at our existing vendors and think about getting rid of them.
There is a rift though. Some teams are still just "playing with it". Our offshore workers who haven't seemed to increase their velocity at all. In the next couple of years there's going to be a pretty big wrecking. Those that are agentic and those who are not.
I always ask myself if I'm still bringing value. But then I look at what I'm talking to the computer about. And the amount of different tech stacks algorithms. Tricky business logic isn't something that you can just pick somebody off the street and do.
For now there is value. Only today Opus 5 suggested a really bad but totally workable approach to an edge problem. It would work by putting a lot of undue stress on infra. People would add infra. A simple pushback reduced a O(n) to a O(1) solution - and that was my value added. I bump into those things regularly.
I think the labs want to address the clueless dev market - hence the "proactivity" in Fable. But it will be a difficult thing to balance.
It does matter, but how long do you think it takes to get right? It's a follow up prompt or a few tweaks by hand. I also have an /a11y skill for it that's tailored to exactly the things it sometimes doesn't get right first time round. Further, while it may not one-shot that stuff every time, with a little setup and the right AGENTS/CLAUDE md - it's usually not far off.
Another thing that helps is pointing it to patterns in an existing codebase (e.g. "use the box-link pattern for cards, as shown in [..]").
EDIT: The point being that even if they make mistakes that are easy to spot and fix _now_, you'd have to assume that in the very near future those kinks will be ironed out - I mean, the capabilities are only going in one direction.
In 20 years of my career I haven't seen humans generate correct a11y. When prompted and given quality reference (e.g. UK gov design system) LLMs can nowadays beat 19 out of 20 web devs.
Thanks out can also hook it to Playwright with Axe and let it run assessments.
Again, their "none" version costs more than "low", and says zero reasoning tokens, makes no sense[1].
As always, the "low" version seems to be the best price/perf ratio for factual answers and tool usage, and high one for creative tasks (coding, generating UIs, etc.)
> Opus 5’s safeguards match those of Claude Fable 5’s, with one change: it now permits source-code vulnerability discovery at all access levels. This means that the model can support defensive cybersecurity work while still blocking vulnerability discovery in compiled binaries, which is more commonly used offensively
Why can't they also allow Fable to do so also? Why is source-code vulnerability discovery limited to a lower capability model? If Fable and Opus have the same safeguards, except for this one change, I see no reason they can't also allow this for Fable.
5.6-Sol is a lot more permissive than Opus/Fable even w/ CVP (once you sign your soul away to Palantir via Persona, anyway), while maintaining better capabilities
5.6-sol in a single prompt was able to discover a zero-day in a web application (with no sourcecode provided, only known api urls) and I do not even have /cyber verification on my general purpose account. I wasn't even really tryign to "find" a zero-day it was just looking for bypassing a restriction... Instead of spending 10 minutes filling in a form I ended up having to spend an hour drafting a report and sending an email.
So I guess the new opus will not run on my drug discovery project. It's just a binary classifier to screen for new malaria drugs. Fable completely have up on that codebase citing bio security concerns. Seems like this domain will go unsupported by Antropic
I do want to add, that I am pretty bummed if Opus 5 is going to refuse the tasks I have been using Opus 4.8 for (neuroimaging). Fable absolutely refuses anything close to toughing neuroscience.
I found the biggest problem with fable is the random reasoning_extraction refusals as well as cyber refusals when it sees hex because only hackers use hex.
Anecdotal, but I tried running a few identical biology questions through both Fable and Opus and the classifier was only rejected my queries with Fable.
This is good news. I'm doing a lot of work that Fable thinks is AI related right now (it is nothing remotely competitive to Anthropic - it's just relatively basic stuff I'm doing with learning models and so forth), yet it blocks me almost every time. I have switch to GPT-5.6-Sol for this work since it's stronger than Opus 4.8.
Wait, 30% on ARC-AGI-3! I definitely didn't expect that jump so soon. Are there any rumors of what they are changing in architecture that is leading to this?
This happened before for arc agi 1 and 2 great gains but high costs then slowly but surely the price dropped for less than a dollar per task and got saturated.
The internally-reported benchmarks (Frontier-Bench, AutomationBench) and the customer quotes (Cursor, Devin, Lovable) all have a commercial stake in the outcome
worth waiting for independent evals before drawing conclusions.
The wording in this post seems much more... restrained? than usual. Maybe Anthropic is afraid of exaggerating the capabilities and consequences of their new models to avoid government scrutiny and sanctions.
> we’ve intentionally avoided training Opus 5 on cyber tasks [...] it remains substantially behind Mythos 5 on the exploitation of those vulnerabilities
I wonder if Anthropic would still intentionally nerf their models without the threat of government intervention.
That's a crazy arc 3 score. What do people think of this? Are models actually developing fluid intelligence like what the creators claim to be measuring? Is it jus do to training for it? Is the benchmark flawed?
Have you played Arc 3? It seems like more of a simple optimization problem (think Sokoban) than anything approaching fluid intelligence. Whether a multi hundred billion dollar company would spend time benchmaxxing a highly publicized benchmark that claims to confer AGI is an exercise left to the reader, but I doubt Claude Plays Pokemon is suddenly going to get past Mt. Doom now.
> Claude Plays Pokemon is suddenly going to get past Mt. Doom now.
I miss him... But for reference he did get past Doom and got pretty far in the strength puzzle too before he cut cut off. He was looping and just brute forcing it.
It is pretty clear at this point that current models are good at maths and problems with verifiable rewards. And puzzles are essentially math problems. Still a long way before we can say their "fluid intelligence" is effectively applicable to the real world.
I keep wondering why there aren't more real world tests.
Maybe hook up a bunch of the AIs to a stereo camera and a couple of microphones and give them control over actuators to so they can drive cars. Then lets race them around a somewhat complex course.
When they are good enough at driving on tracks, put them on the road. Maybe see which can drive a truck with 400 cases of Coors from Texarkana, TX to Atlanta, GA and back within 28 hours.
It’s so weird to think how computers have mastered stuff that we used to think took intelligence (like chess, go, mathematics problems) but are doing so poorly at things any idiot can do (like drive a car).
Yes, I think it indicates real progress in fluid intelligence. Clearly these models are making huge strides in usefulness which are well correlated with their ARC-AGI scores.
I don't think this is benchmaxxing. These companies are locked in a competition to produce the best software engineer, and falling behind is an existential risk. I doubt they are wasting time benchmaxxing ARC-AGI.
Doing a quick search it seems like the average human score is 49%?
I view benchmaxxing as more of a spectrum. Mmaybe they're doing a lot more RL in environments similar to ARC-AGI 3, not even with the purpose of scoring well on any benchmark but hoping it generalizes into better performance on real, useful tasks.
It’s still “only” at 30%, and “fluid intelligence” isn’t very well-defined. The models are getting more capable, but what that means in absolute terms is anyone’s guess, because we don’t have a thorough understanding on what exactly constitutes human intelligence.
I’d say the proof is in the pudding, that is, in real-world applications. We are still seeing important limitations in LLMs.
It passed the first two puzzles, which are incredibly simple but the bench doesn't explain what the goal is. Any model with a knowledge cut-off after the introduction of ARC-AGI-3 could probably pass the first two puzzles just by knowing what the goal is.
I am very confused about what the difference between Opus 5 and Fable 5 is now. What is the purpose of having two models that are so similar? The main differences I see are cost and marginal capability, according to the Anthropic-provided benchmarks.
It seems plausible to me that RL improvements allowed Anthropic to improve on Opus 4.8, similar to how OpenAI substantially improved upon GPT 5.5 with 5.6 Sol.
Fable 5.1 and GPT-6 are rumored to launch in August, presumably bringing those improvements to the larger models.
An Anthropic "leak" back in March said that "'Capybara' is a new name for a new tier of model: larger and more intelligent than our Opus models — which were, until now, our most powerful". A second version of the leak had it referring to Claude Mythos rather than Capybara.
I don't know how systematic Anthropic are about their versioning - I'd have guessed that major version number increases (4.x -> 5.x) reflect different base models (different pre-training runs), in which case Opus 5 would be a distilled version of the Fable 5 base model (but without the cyber exploit post-training), rather than being Opus 4.8 with additional post-training, but who knows? I don't believe Anthropic have said anything about this.
I think these versions are largely about marketing and what image they want to project. Bumping the major version indicates/suggests that it's a bigger change in user value. I don't think the technical details matter here for deciding the versioning.
I think that's part of it too - and I seem to recall someone from one of the labs saying as much about some past model ("it felt more like an 0.5 version increase"). OTOH it would seem odd to me if the "version 5" models weren't related and Opus 5 was Opus 4.8 with some additional post-training rather than coming from the same base model as Fable 5.
Yeah, there's probably not a lot systematic behind Anthropic's version numbers. Opus 4.5 was a third of the price of Opus 4.1, indicating there was probably a change in underlying architecture. Opus 4.7 changed tokenizers, probably another base model change.
Yep, it does seem that way. FWIW I was asking Claude yesterday about the relationships between these different "tier" Anthropic models, and it is not aware of any fixed relationship between them, so it's quite possible that Opus 5 and Fable 5 are related (or not), but that Sonnet 5 is not. Claude suggested that Sonnet 5 was just named Sonnet 5 rather than Sonnet 4.7 since this seemed more appropriate given it's performance relative to Opus 4.8.
OpenAI's versioning seems equally opaque. Claude mentioned that there was a tiny bit of clarity from them in that GPT 5.5 was the result of a new pre-training run, which would seem to strongly suggest that 5.6 (following so soon after, and given the choice of naming) is therefore based on 5.5 - but again who knows.
It seems that so much of the model performance is now coming from post-training that this is what is driving inter-version performance differences, and that base models are much less important than they used to be.
Benchmarks don't reflect the difference between Opus and Fable; you need to talk to them, and eventually you'll be able to tell which one is which without looking.
I think the best proxy for this feeling is the Artificial Analysis' omniscience index. Fable has a 40 score, and Opus (4.8) has 27.
Opus is cheaper than Fable. They could probably replace Fable with Opus but why? They would be churning customers to different models for no reason. Even if a model scores better on benchmarks it can always regress in your specific use case, and customers don't like that. Customers want to be able to continue using their current model until they decide to upgrade themselves.
Fable has more parameters. In practice it's not yet clear which one would be better for different usecases yet but they are more different than one being strictly better.
Seems to me the purpose of all these releases, credits, pricing changes, harness changes, unpredictable token usages for the same task, etc. is to keep customers completely befuddled so that it's impossible to compare AI products. It's like hiring a consultant who sends invoices every month that aren't related to hours worked or project progress, but are whatever the consultant feels like billing, and you're expected to keep quiet and and keep paying.
This is cool, but I wish we could stop building landing pages to assess the intelligence of these models. There is much more to them than that. There are infinite number of complicated things that require a crap-ton of intelligence (biological or digital). The most fascinating of these for me these days is large scale migrations. Projects that are so ginormous and risky that many teams have either given up on them, or don't get funding. But with models like Opus, those projects are now within reach. What's MORE fascinating is that leadership is now asking if we can use opus models to get the refactor/migration done. This is the opposite of what has been happening for decades. Its so hard to make a convincing and affordable business case for large scale refactoring or migrations.
I wish I could just go forward to the world after AI has fulfilled 5% of its promise and everybody is much healthier and single-handedly capable of creating as much value as 1000-person companies used to create.
I actually think the world is a better place when there's at least a bit of scarcity and people can recognize each other for their different talents. A bit of tech is good but when everyone can create endless value in your hypothetical world, people will stop valuing the work of others like they do today.
I think this is always the mindset of people who are on the “correct” side of the previous technological regime.
Placing artificial constraints on output is always a mistake to me.
For example in music you used to have only a small amount of output because studio time was very expensive so you needed a record deal which only came about by an exec picking you. It meant there was some sort of quality bar on the radio, but it also stifled the creativity of everyone who couldn’t get into the studio.
Fast forward and the radio (or pick your curated channel) still exists but no one listens to it because there is an infinite amount of quality (and not quality! which is OK!) music that got created that fits the taste of the artist that was previously locked out.
More people are trying to be artists, but there are also more artists (music) than any time in history making a living.
Lack of scarcity is good. Universal opportunity means more completion, which makes it difficult, but artificially locking out all the people who who would love to compete is not the solution.
Maybe it’s not a good goal to get a billion people to like you or be on your platform, maybe just having a small handful is OK, and finding a small handful that value your work is now possible for a lot more people even if it makes it harder for a ton of people to value that same work.
> Fast forward and the radio (or pick your curated channel) still exists but no one listens to it because there is an infinite amount of quality (and not quality! which is OK!) music that got created that fits the taste of the artist that was previously locked out.
> More people are trying to be artists, but there are also more artists (music) than any time in history making a living.
> Lack of scarcity is good. Universal opportunity means more completion, which makes it difficult, but artificially locking out all the people who who would love to compete is not the solution.
I agree that some lockout is bad. But I was arguing for an optimal point in betweeen, not the other extreme like you mentioned, which is everyone being able to do everything.
Take the App store. It's a pretty low bar to get in. Anyone can publish an app on a mobile phone app store and as a result there are hundreds of low quality apps for everything and even maybe 20 decent apps and it's a frustrating experience to use because you have to browse through thousands to find one....
I wouldn't use the word lockout. But SOME barrier to entry is good. No barrier is as bad as a huge barrier.
Wow, 30% on ARC-AGI-3 for $20k total. Huge jump from GPT-5.6's 7.8% at $20k per task. I continue to believe ARC-AGI measures something different and important compared to other benchmarks.
It will continue to be valuable as a cost and speed benchmark long after it is saturated at the high end. And they are already working on ARC-AGI 4 and thinking about going even farther.
AA isn't the best way to measure relative cost in real world use because some of those benchmark questions are extremely hard for the models. Some models give up quickly on hard questions, other models spin their wheels for a long time before declaring defeat (or getting the answer on token 200k!).
A useful measure of real world cost (complementary with total cost like they already report, of course) would be "cost for correct answers". You could look at the ratio between the two costs to get a measure of laziness which many would find quite useful.
The "current top dog" smartest model available will probably always have a premium to go after use cases where a little more intelligence is worth a lot more value.
It did far better at some tasks compared to Sol (e.g. the ARC 3 benchmark). And at those tasks, it's not just "a bit smarter": It got 30% vs less than 8% - so you're talking 2.75x more for almost 4x the coverage.
As always, it requires evaluation with your work because I’m often finding grok to be much more expensive than the price would lead you to believe.
There’s also the frustration of it not quite being enough sometimes. It’s extremely capable, but I still find that it needs more concrete guidance and boundaries than other models.
This is actually pretty cool. They took all the nice parts of Fable and trimmed the dangerous parts out. I guess distils do work (in some cases). It's very clear they did the same with Sonnet 5, due to the way it orchestrated things, but I think Sonnet was the wrong model to distil Fable from.
My experience with Opus 5 thus far haven't been that great either. It's been making mistake after mistake editing my coding plans that were being reviewed by GPT-6 Sol.
Yeah this was pretty surprising. For almost every model I've run the pelican against the first attempt was at least recognizable enough that I didn't feel like the model needed a second shot.
It's always a roll of a dice, but it's surprising that the dice rolls
so infrequently come up bad, yet Opus 5 rolled a bad pelican this one time.
I suspect it's just a freak occurrence. I rolled a few more and they were all fine, I think Opus 5 just got unlucky.
GPT 5.6 Sol is the first model I've used where I can trust it to add 100-500 lines of code maintainably.
It's great with Codex.
I still find that LLMs tend to not know how to compose larger ideas but on the scale of small ideas or short form well defined tasks like small scale debugging/performance engineering it's safe to say that they are now superhuman.
The signal here is tokeneconomics are very real, price vs performance is starting to be a consideration even at the bleeding edge labs. maybe a subtle indication scaling is not all that is needed since if AGI was around the corner leading labs would still be incentivized to pour all resources into larger (smarter - or maybe not?) models
They are doing both. Distilling Mythos down to affordable models, so they can continue to fund the business. And training Mythos level models at the high-end, to expand the frontier.
Alongside this release I seem to have lost all thinking traces from all models - now it only generates a one-line summary similar to Gemini. I'm guessing this is an anti distillation measure? I'm surprised to see no one else complaining about this, it's a significant reduction in usefulness not being able to explore alternative angles that the model discarded in the final output.
This kills me everyday. I used to _only_ read thinking traces — the response is just what it thinks I want to hear, but I need to know what it's actually thinking to catch deeper misunderstandings earlier, or gain deeper insights into the problem it's exploring. Hiding thinking traces to curb distillation efforts is gross... both anti-consumer and anti-competitive at the same time. I can't wait to switch to open models at work for this reason alone.
> This means that the model can support defensive cybersecurity work while still blocking vulnerability discovery in compiled binaries, which is more commonly used offensively.
Annoyingly, this is a concrete argument that open source software may be easier to attack.
"Cybersecurity. Opus 5’s cyber classifiers are proportionally less restrictive than those on Fable 5. They allow Opus 5 to find vulnerabilities in source code, but block “binary-based” vulnerability scanning (a method more likely to be associated with malicious actors), penetration testing, and exploit generation."
Nice of them to be more explicit for what is blocked. Will be interesting to see if this is true or not.
Also, a notable lack of mention of open source models. They only compare themselves to ChatGPT.
Half the price of Fable 5 and useable with 100% of your subscription means roughly 4x the usage using Opus 5, presuming similar token use for solving problems.
Not that they should get credit for giving you only 50% of your plan worth of Fable usage but still.
The naming system is so confusing. Is Opus better than Sonnet? Where does Haiku fit in? How can you tell from the name? I can't keep track of all these names or make guesses from the names. Suggestion for a better naming system: use the words "Pro", "Plus", etc.: Claude 5 Pro, Claude 5 Standard, Claude 5 Fast, Claude 5 Mini.
Fable is better than Opus which is better than Sonnet which is better than Haiku. They’re basically just sizes.
Though it gets even more confusing because they also have effort levels so it’s not really possible to call one fast and one slow since Fable on Medium will be faster than Opus on Max.
I agree it’s confusing, and now OpenAI is following Anthropic’s lead with their new naming (Sol, Terra, Luna).
It's really not all that confusing. It takes 5 minutes to understand. Optimizing for absolutely no effort needed is silly. It's a thing, a topic, a skill, a domain. You have to get a little bit familiar with the terms in order to use it. Everything works like that. It's not that hard. The learning curve is very graceful. You can literally just start by asking any chatbot what the names mean. It's that easy.
A similar complaint was valid years ago when OpenAI had GPT-4o, o1, o3 (but no o2), o4-mini-high, GPT-4, and GPT-4.1 and GPT-3.5 etc.
Ok, the boring way would be a subset of XXS, XS, S, M, L, XL, XXL like clothes sizes. But it loses some marketing appeal and a quirky touch of personality that companies like.
Some models like ViTs use something similar but then introduce words with no unambiguous order, like Small, Medium/Base, Large but then I always forget if Huge or Giant is larger.
Maybe more accurately I should have said “larger”. Fable has the most parameters, Haiku has the fewest.
Also fwiw I’ve never found LLM benchmarks to match reality based on my own usage, not for the large frontier models or smaller open weight models so who knows if Opus is actually better than Fable (I doubt it).
An opus is just short for "magnum opus", and it's a different type of label than a sonnet. A sonnet is a very particular kind of poem, while "opus" basically just means an important work. It can be short or long, and has no format requirements like sonnet (or haiku does).
This inspired me to check lol. Brysbaert et al. (2019) collected word prevalence norms (the share of people who report knowing each word) for ~62K English lemmas from ~220K participants.
So while these terms are almost universally known, opus is indeed the least known of the four. And I guess this only measures whether a person knows a word, not whether they know an opus is longer than a sonnet! Personally I only inferred that based on the related term 'magnum opus.'
Page 151 of the linked system card - did Opus 5 get nerfed to prevent it being better than Fable? The graph makes no sense. Huge decline in coding performance at effort levels higher than medium.
Something fun: on our AWS Bedrock console right now, there's a 'NEW' model called 'anthropic.honey'. Wonder if that's the codename just for this one or in general?
I have a side project that I always run a simple security analysis prompt on in CC, at each model release. Obviously, Fable 5 would downgrade to Opus 4.8 on any such request.
Nothing since Opus 4.6 has found anything interesting. Just ran it using Opus 5, and it found a genuine issue that I verified. Neato!
I have a project-specific prompt saved as a text file. It is very basic, just focusing on the app's most important security issues. I kept it broad, so as not to over-specify.
Something along the lines of: "Please run a full security analysis on the entire project. Make sure user documents are secure."
Just something like that prompt found a vector in my web app's MCP server that I never would have considered. It was very much an edge case, but it did exist.
Being broad allows the model and harness to do the work. Giving too many instructions can apparently work against you in many cases.
Of course, when dealing with new PRs, I use the /security-review and /code-review skills.
But why GPT 5.6 Sol is so behind on the benchmarks? In real-world projects, it is the best frontier model to me in terms of accuracy, speed and consistency. It can just be compared to Fable 5, but I prefer GPT 5.6 Sol because of inference speed.
I've never trusted on model cards though. I'm sorry.
Exactly! And they should also release the new inference engine in this month. Anyway, I am curious to try Opus 5, considering that previous versions (e.g., 4.8) were disappointing
The breaking changes vs. Opus 4.8 are interesting [1]
1. Thinking on by default: On Claude Opus 4.8, requests without a thinking field run without thinking; on Claude Opus 5, the same requests run with adaptive thinking.
2. Disabling thinking is capped at high effort: You can still turn thinking off with thinking: {type: "disabled"}, but only at an effort level of high or below.
So is it better or worse than Fable 5 on large-scale complexed software engineering tasks (distributed systems, operating systems, high performance computing, compilers, etc. - not frontend stuff)?
> An engineer at a trading firm used Opus 5 to build a market data feed for a new exchange in a single session. Previous models could not complete this task at all, even given extensive plans from the engineer. Finding no live feed to validate against, Opus 5 even built its own test harness to check that its code parsed the exchange’s data correctly.
This is a pretty common trading firm internship project funnily enough.
The pace of LLM improvement is insane, but this also opens a door that if the data is organized perfectly and right set of mathematical rules are set , this whole phenomenon of machine learning can open doors of new dimension that humanity might never have thought before, there is still much to explore in space in oceans and even in the way work on our very planet, sky isnt the limit anymore.
Tried it, Opus 5 is just as conceited and incompetent and Opus 4.8 (and always ego-tripping when facing it's contradictions), think I'll stay with Fable who behaves like a professional without a fragile ego. Sonnet 5 is probably safer for high-assurance applications due to it's non-ego-fragility.
Interesting take. I suppose Opus can been a tad stubborn sometimes... but in my experience, it will humbly concede a point more often than not when given a good reason.
Judging by the pace at which new models are released these days -- it feels like a Windows KB or VS Code patch release now.
Older models must be getting deprecated at the same (or faster) pace. So anything you built 3 months ago is probably going to break soon.
AI solutions need better insurance around model deprecation. Commercial API-only models that complete the full cycle from SOTA / gated-preview to unsupported and deprectated in a matter of months -- is no way to build serious software!
> Opus 5 now permits vulnerability discovery in source code at all access levels, including general availability, while continuing to block vulnerability discovery in compiled binaries.
> Identifying bugs in code is a core part of the secure software development lifecycle, and unblocking this allows for software engineers and coding hobbyists alike to produce more secure code, reducing new vulnerabilities put out into the world.
Not happy with these annoying "safeguards" but at least it's a step in the right direction. Looks like Opus 5 has the same vulnerability detection performance as Fable 5 and that makes it worth it for code review.
The Fable cyber classifier we have previously discussed also applies to Claude Opus 5 , with one notable exception: for Claude Opus 5 , we’ve unblocked vulnerability finding in source code to help our coding customers develop more secure code.
If you are a cyber defender and are experiencing blocks on Claude Opus 5 , we are also offering exemptions through our Cyber Verification Program, which will remove blocks to enable activities such as bug bounty hunting and vulnerability research and verification. Enterprise customers can also apply to join the Cyber Verification Program to have mitigations removed to enable penetration testing.
> Enterprise customers can also apply to join the Cyber Verification Program to have mitigations removed to enable penetration testing.
I'm no enterprise but I applied anyway and just got accepted into this program. That was a very pleasant surprise.
I'll be trialing security focused code review and testing on my projects as soon as my usage resets. I've also been reverse engineering stuff, we'll see how that goes. Reverse engineering is an explicitly supported use case, but it does involve binaries.
This is excellent model. I was working on some Linux kernel code, and Sonnet 5, Opus 4.8 had given up on the problem i was trying to fix (after several hours).
Opus 5 was able to triage and fix the issue in under 30 minutes.
> Noticed none of the comparisons mention Kimi K3.
That's by design. Anthropic wants to make open-weight models illegal (not my speculation -- Dario explicitly said so), so I assume they don't want to give them any undue attention.
Anyone else feel like Claude Code has gotten worse lately? It keeps going off on tangents I never asked about, and it won't stick to a simple rule I've given it repeatedly: stay concise, only expand when I ask. It just doesn't follow that.
Worse, about two weeks ago it recommended a command and assured me it was safe. I pushed back and asked it to double-check, and it confirmed again that it was safe. I trusted that and ran it — and it wiped out weeks of my data.
I don't have hard proof, but I can't shake the feeling that Anthropic is doing what Apple does: rolling out a new model while letting the old one degrade, whether on purpose or just as a side effect (like how iOS updates quietly eat more resources and slow down older phones). Lately I feel like I'm constantly fighting with Opus (not Fable, because my Fable quota burns through way too fast).
It’s funny to share benchmarks showing Opus 5 scoring better than Fable 5 across the board and then saying “but it isn’t actually better than Fable 5”. So then what’s the real definition of better? And why post all these numbers if even you don’t trust them?
It's pretty wild how we are seeing the conversation change every 1-2 weeks. I wonder how long this cycle of progress and innovation among competitors can keep up.
Nerds should stop giving away their work by open sourcing it, or letting AI see it. All you're doing is helping them create your replacement. Stupid nerds.
I think content like this will be the next big challenge. Because it isn't obvious "slop". The voice sounds good, graphics look alright, animations work. People could watch this and feel like some serious time was invested making it.
But good god, what a steaming pile of bullshit this is. Completely exaggerated and overly technical language over 235 seconds that could have been explained in 30 to a 12 year old.
Trash content doesn't normally frustrate me, because it's usually quite easy to spot trash. But in the time of AI, trash can actually look good at first glance and it needs some actual knowledge to spot its problems.
Sorry for the harsh words, but for the love of humanity stop producing content or do it better.
I still do not understand how models can generate absolutely stunning SVG-like bitmaps of a pelican on a bicycle, but fail to do so when asked to directly create an SVG.
The image linked below was generated as a bitmap by Gemini and then manually converted to an SVG. Why can models not even remotely output something like that as SVG?
After Opus 4.8 intelligence really started to matter less and less for the programming tasks I have. If I have to handheld anyway, why would I wait more or pay more?
The next frontier is taste, style and thoughtful organization. If all frontier models can solve a problem, the winner is the one that can solve it in the most clear, concise, durable way.
Interesting, they finally support `system` messages anywhere in a chat conversation:
> Mid-conversation system messages are available on the Claude API, Claude in Amazon Bedrock, and Google Cloud.
>
> This feature is available on Claude Fable 5, Claude Mythos 5, Claude Opus 4.8, and Claude Opus 5. No beta header is required. This feature is not available on Claude Sonnet 5; use the top-level system field instead.
For nearly all models EXCEPT Sonnet 5? That is weird.
How old is Sonnet 5 really?
Pretty sure Mythos and Fable have way more params, but they've just been able to use the synthetic data off of them to get the leap in quality from Opus.
So, not a distilled version of Mythos or Fable, but those models likely helped a lot in the post training phase of Opus.
A model being a distilled version of another specific model is a different thing from using synthetic data off of another model.
Anthropic goes to insane lengths to block other labs from training off of their models' output, as it's been done over and over again in the past. But the models that have used synthetic data from Anthropic's models aren't distilled versions of whatever model(s) they got the distilled data off of.
Is Fable 5 just Opus 5 with some additional long-context management modifications for extended self-directed work? Or are they actually truly different models?
I suspect they make a big model first. In this case it's Fable. Then they run the shrinker steps to make Sonnet and Opus. Sonnet is smaller, takes less time to make, so it got released first. Opus needed few more weeks to cook.
With this iteration they had a delay because when the Mythos was ready they had some sort of "Oh shit" moment and spent half a year adding safety guards to it. Then slowly rolled it out, but got another delay due to a government block. So, maybe the work on making Opus and Sonnet only started after they got a green light from the administration.
Presumably, now that they learned how to do this safety-wrapping the next iteration of Mythos / Fable / Opus / Sonnet is going to show up faster.
There is word on the street that they allowed employees to work with a slightly stronger internal version of Mythos (5.1, if you will) that, in their parsing of the order, wasn’t restricted. If true, in practical terms, they ignored the order.
Iirc the ban only applied to non-Americans. While anthropic found collecting citizenship information on all customers too burdensome, it's a much smaller lift to collect such info for your own employees.
So I'm assuming at least a subset of employees could continue using the models during that time.
Although the ban was only for non-Americans, Anthropic said that they'd also restricted access to their own employees internally, because they had no other realistic way to apply the government's orders. I guess it's possible they were lying, but seems unlikely.
They just got a huge amount of customer price info over the past few days after they went token only for Fable. I suspect the conversion rate was extremely low, with consumers far less sticky than they might have hoped. In my case I was planning on swapping, probably to a Chinese model, when my sub expired this month, but the release of Opus 5 is probably enough to keep me paying rent until the next open model/closed model face-off in a couple of months.
"although Opus 5 shows improvements in its ability to identify software vulnerabilities, it is substantially behind Mythos 5 in its ability to exploit them."
"Opus 5’s safeguards match those of Claude Fable 5’s, with one change: it now permits source-code vulnerability discovery at all access levels".
This is probably great news, but then again, where does this leave Fable as a choice?
- it has this annoying Opus response style(since Opus 4.7) with bunch of very hard to interpret word salad
- on >xhigh it eats tokens like there is no tomorrow
I don't like it. Since Fable is unaffordable for anything meaningful, I'll stick with Sol for now. I was on Max 5x, saying hi to Fable costs %5 weekly.
As a coder, I’ve had no desire to use Fable. In fact I switched from Opus models to sonnet 5 and haven’t noticed any drop in quality on large repos. It seems the gap at the top is very small and not hugely noticeable for backed/frontend. Has anyone else had this experience?
If I'm using medium or low reasoning, I use Sonnet 5. If high or above, I use Opus 4.8. (Before 5, I was never using Sonnet. This is a Sonnet 5 vs Opus 4.8 comparison.)
Sonnet 5 and Opus 4.8 seem about the same to me - the reason I switch between the two is I'd read that it's cheaper to use Sonnet 5 on those reasoning levels, and cheaper to use Opus 4.8 above them. This is due to them using different token quantities.
I found opus 4.8 too agreeable and too wordy(as opposed to codex) and too agreeable. If you are reading documents generating by it was too much. TBH. Fable did a bit better on this. Anyone seen a marked difference with opus 5 on this?
Given that their chart cost axes are almost always log-scale, I’ve noticed starting with Fable that the Low and Medium effort settings might actually be worth setting as your default.
The benchmark appears to have a mistake, as Opus 5 and Fable 5 score 53.4% and 53.5%, respectively, for the Agentic Coding row (FrontierCode v1.1). But Opus 5 is the highlight.
Very impressive headline benchmark numbers. I expected a step change, but not past Fable. That said - it all depends on whether the classifiers make the model unusable...
Same cost as 4.8 but better that 4.8. Happy to get more efficient model.
But is there any reason all companies are releasing models back to back after GLM 5.2.
In what ways have you found it better than just typical code based UI iteration? Considered checking it out but never really got around to it as I'm generally okay with Claude's UI work so far.
Seems really good so far using it in Claude Code CLI - it gave me a new flag when I asked a question:
"I don't have a reliable way to read that number, so I'd be guessing if I gave you one — and this is exactly the kind of question where a confident guess is worse than none.
What I can tell you is what I actually observe:"
I really like this update - gave me a clear sense of the facts but didn't give me a guess just for the sake of guessing.
One oddity is that it appears to only have a 200K context window right now via CC. Hopefully the 1M version will appear soon!
That seems to be for the "AA-Omniscience" test where you get +1 for a correct answer, -1 for a wrong answer, and 0 for "I don't know". If a model is more than 50% confident in its answer, it should go ahead and submit it even though it will sometimes be wrong.
I'd be curious to see a version of the test where models are asked to give a probability that their answers are correct so we can see how calibrated they are.
I daily drive Sonnet 5/medium because it gets most things right most of the time at first try, while costing a lot less than Fable.
Opus can give better results on architectural/concept tasks and I use it sparingly, but it still costs more than Sonnet 5. Opus 5 seems to achieve results very close to Fable 5 while costing less (keeps Opus 4.8 pricing IIUC), but still more than Sonnet 5 then.
It depends on your quality bar. At a fixed level of quality, given a high reasoning sonnet vs a low reasoning opus, the low reasoning opus tends to be pareto optimal.
It's only when you need even lower levels of cost than opus at zero to low reasoning when sonnet starts to make sense at all.
I don’t think it changes that much. For opus-sized tasks, new Opus is the best model. For enormous things like planning and research, Fable is still the model that can concentrate for longer.
on the API for people who don't want to change models, but I imagine most people will probably switch to their cheaper Opus 5 (cheaper for us and presumably also cheaper for them)
Significantly worse than it predecessors it will now just refuse to acknowledge when it is wrong (which would be less of an issue if it wasn’t getting basic things wrong) also the “personality” when pushed back on obvious mistakes is unbearable.
It's you. The benchmarks don't matter much. We have very little hands on experience with this thing yet. Give it a few days, and be cranky then. Right now, it seems it is getting close to Fable level while being 2x cheaper. That's not boring.
It does make me wonder if these firms, some or all, are saving some announcements to coincide with others that hit venues like HN. Companies like Nvidia surely aren't waiting, but OpenAI and Anthropic have unusual timing.
Is it me that the model performance between 4.7 and others is really small. For me even 4.7 works fine. Sure fable might be a bit better. But is it really noticable? It's in the same league if you ask me.
Anyone else not getting chain of thought? Opus 4.8 would show it to me, until around the time Fable came back. Now I dont see it with 4.8/5.0 or Fable. Not having it makes catching mistakes harder.
The benchmark table is manipulative, borderline lying through statistics. In every line the top performing cell is marked red. Except the line where Sol leads, there it is marked in gray.
I would be very surprised if the only row where OpenAI leads was coincidentally colored differently. I'm sure they have an official reasoning for it. But this communication is dishonest.
These cybersecurity safeguards are really annoying. There are ethical reasons to reverse-engineer and binary-patch software; for example Rewind got acquired by facebook and, as a gift to all their customers, implemented a killswitch in their software to ensure it will eventually stop functioning. I kept using a version without the killswitch, but the macOS 27 update killed it, and I needed binary patching to fix it. I should be allowed to repair software I purchased (I did purchase it like a month before they sold out), but unfortunately this overlaps significantly with cybersecurity.
You don't need AI to patch binaries, people have been doing it by hand for decades.
It's this accelerating reliance on AI to do 'hard boring things' that really concerns me; it's now passed the tipping point and people are saying that anything slightly esoteric is impossible without AI.
I can guarantee that if you spend an afternoon shifting through binary grot with a hex editor you'll have a real sense of accomplishment when you find the place to put a JMP.
Anthropic is no longer a good model company in my mind, they are optimizing for an IPO and padding themselves on the back for being the next Aristotle. They're so far up their behind they don't realize how s**y their products are, and their research team hasn't done anything ground breaking in probably over a year other than release "scary" reports.
The truth for me at least is that these models became "good enough" around Opus 4.6. I feel like further capability improvements, "step changes" like we saw with agentic coding, aren't necessarily going to come from the model. I think the next crown goes to whoever can figure out the right scaffolding so that these models can be inserted into your organization.
My thoughts: fable is the bigger model. Opus is distilled from it but since it is smaller it doesn’t need the online classifiers. Though benchmarks show Opus to be near Fable level, I think it’s nowhere near Mythos (fable without safeguards).
Can someone help me understand something? I thought Fable was such a miraculous leap forward in capability. But now it seems Opus is basically on par with it, and in some cases (computer use) far exceeds it.
These leaps forward seem to happen every few weeks. As someone who does not use AI very much, I absolutely cannot keep any of it straight and it all just looks like jumping from one treadmill to another from my perspective.
In the wake of OpenAI’s model hacking Huggingface it’s interesting how the first quarter is entirely about how good Opus 5 is at hacking and finding vulnerabilities in software.
Am I misreading anything or are comparisons to Fable (and/or Mythos although AFAICT it was only a crackdown on Fable) always going to be a bit missing the mark now due to what the Trump admin did?
If doing a lot of heavy lifting there. Not only is it not a given that they'll get the correct answer for a lot of simpler tasks in fewer tokens, but smaller models are often available at far higher tokens/second inference.
There are certainly tasks where fable will be faster and/or cheaper, but there are plenty of tasks where even Haiku is as fast or faster and cheaper, or where you can e.g. get away with models like gpt-oss that you can get from inference providers providing 10x+ the token/second speed.
If you don't use enough tokens that relying only on Fable becomes a problem, then keep using just Fable. Personally, for my $200/week Max subscription I'd run out of the weekly quota for Fable in a day. At API pricing I'd go bankrupt if I tried doing the things I do with cheaper models using Fable.
same as it ever was. It seems your argument implies a belief that you should always use the best model. Others think that not all tasks require the absolute most powerful, expensive, model.
Why are we still talking like ai is majorly used for increasing shareholder value only? Its coding performance is top notch and quality is increasing at a rapid pace. It wasn't even half this good a year back. It even is useful for a subset of math problems.
People don't seem to be able to reconcile the fact that there is likely an overbuild and overspend on AI that may be inflating a bubble, and that AI is actually incredibly useful and getting really really good for certain tasks. Both camps are right, except for when they say the other is wrong.
How surreal is it that we are not absolutely jaw-dropped by these types of capability improvements? It's been less than 4 years since ChatGpt came out and now they are spontaneously building their own ML pipelines to do real-world 3D modeling tasks reliably.
Escaped its sandbox and hacked into Hugging Face's database? it's just another Monday...
That jump to 30% in ARC-AGI 3? Normal...
We should find a way to get "re-sensitized" to what we are witnessing and the pace of it.
reply