Perhaps I'm not understanding it correctly, but here's my take on what the paper is doing.
Imagine you have a problem you want to solve (let's say, identify an OCR'd handwritten character, e.g. the MNIST Dataset). You tell 3 agents "Hey, each of you take a stab at getting really good at recognizing characters from this dataset. You can take 10 refinement steps to continue to improve ". You can't give each agent unlimited steps of course, because you have a finite amount of compute.
So each agent goes off, and by the end, Agent 1 got to 90% accuracy, Agent 2 got to 80% accuracy, and Agent 3 got to 89% accuracy. Agent 1 wins, of course.
But then you look at the refinement steps, and after 2 steps, Agent 1 was _already at_ 90% accuracy. So the agent spent the next 8 steps basically not moving at all. Agent 3 on the other hand, perhaps was continuously climbing in accuracy at every refinement step, but hit step 10 and had to stop.
Now because you recorded every step from every agent, you know what you'd do differently next time -- you'd not allocate as many steps to Agent 1, and give Agent 3 more steps, because perhaps that might result in Agent 3 coming up with a better answer.
From my understanding, that's what they built in the form of a "search" controller -- a way to evaluate automatically and reapply how you could allocate resources more effectively, when applied to a new problem.
But I guess my misunderstanding is how applicable the search controller is when applied to new problems -- just because one pathway stalled early for one problem, doesn't mean it would work for another?
Your understanding is basically correct.
"how applicable the search controller is when applied to new problems". We need meta-agent thinking pattern. Self-evolving agent has been very popular and we want to use agent to design a perfect agent. This is the problem that the "search" controller employed in this paper aims to solve.
Could use the shorthand of 85%, 10%, and 5% as the way to divide the workload; 850/1000 computes, 100/1000, and 50/1000. Brute sprint, clean up & revaluation runs, then checksum and presentation.
Intuitively I wouldn't readjust how many steps they each do, but instead add another run afterwards, that get the same amount of steps as the previous, but now also with a concise description of what the previous attempts did and what they achieved, and ask it to improve. The amount of compute you have available, would dictate how many full iterations of this "fan out search > consolidate" workflow you can do.
In the paper (section 5.1), they actually tried to abstract high level directional insights into the prompt in order to see if that helped, and they basically found it underperformed a prompt that didn't have those insights at all, implying that directional guidance perhaps over-constrains things.
I've stopped waiting for sites to do what I want in terms of features and functionality -- their incentives and mine very rarely align.
I think the solution here eventually will be a search engine that lives on your personal computer (it's also what I'm building currently). Ingest content from the web -> sort/filter/view it locally as you choose.
Note -- if you do this, watch out for requesting access to "all tlds". They send you two emails per TLD -- one for your pending state, and one for your approved/rejected state. I suddenly had 1k+ emails flooding into my inbox, until I found the setting on their website to disable emails.
This is actually where I see software going in the short term -- cloud moving to local.
A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search.
But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it.
The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally.
It's more of a pipeline than a back-and-forth. New abilities happen in the cloud first because they require specialized, higher capacity resources and then move towards being local as the resource usage gets optimized.
Outside insane GPU appliances, I really see cloud based things as a lock-in to subscription based offerings. The fallacy of always using the latest code isn't worth never ending subscription fees. I'll install it locally. I'll keep my data locally. I don't need to wait for things to upload/download. If there's an update that feels worthy, I'll purchased and install locally.
This post does not feel like it was made in good faith. The general premise ends up being:
- Matz is nice -> DHH / Tobias are "not nice" -> Ruby is doomed (because of two not nice people, also perhaps because of AI)
I think if they had focused perhaps on DHH (DHH's writings have been very polarizing), and also dropped the AI bit at the end, they might have had a stronger point, but Tobias ends up being judged guilty by the author almost purely through association:
> [...] Lütke doesn’t say anything nearly as incendiary as DHH, but the actions of the company he controls tell their own story, and his posts on X, the Everything App, document his turn to the right, along with all the crypto and AI stuff you’d expect to go along with it.
The author ends with a hope that a fork of rails will happen, quoting a user on mastodon:
> [...] we're building a community to execute a bigotry-free #Rails fork [...]
Here's my issue: let's imagine the author gets exactly what they want. Rails is forked, the fork becomes more used than rails, the community is saved!
The problem is that this forked code now becomes two things: a product of software (what it was before), and a product of ideology.
When a piece of software now also represents an ideology, people that want to contribute (but are not coders) end up contributing to the ideology instead. E.g. in this case, they'll try to help define what constitutes a bigot, find potential bigots in the community to root them out, things of that nature. When the "easy" bigots are found and removed, the work does not stop -- the definition of what is a bigot changes, and new bigots are found.
To be clear, the above happens regardless of ideology -- it's just human nature (See: Levari et al., Science, 2018). For example, many people on Reddit who claimed it had gotten too left-wing/right wing tried to create their own social media platform, only to realize that targeting an ideology as their main differentiator ended up eventually only attracting the people who are so deep in that ideology that it pushed away anyone casual.
He compared an entire race to wild animals that were, according to him, rightfully brought to full extinction in the middle ages.
What does he need to state in order for his ramblings to not be referred to by euphemisms? Call for concentration camps and gas chambers? Or will this also be an 'edgy' blog post? A 'spicy' tweet?
DHH was not the point of my comment (in fact, I said if the author had focused on him they'd have more of a point), so this unfortunately feels needlessly attackative.
About the same thing I saw. I kept having to increase my bid every day, and even at $4 CPC basically saw nothing converting.
As you said, it's also really hard to get visibility: metrics are poor, and targeting seems more like a whim/suggestion than anything else (their label for context hints basically tells you they only _might_ take your guidance on who they will show your ad to).
Audiences consists of hashed emails or phone numbers -- no targeting demographics, job titles, etc.
Overall it's just not there yet -- I paused my ads today.
> Can you imagine a venue refusing entry to former clients 2% of the time just because they’ve “improved their experience”?
On the flipside, if a client enters enough venues that refuse entry to them because of something the client can fix on their end, eventually the client will probably change themselves -- "If you meet one asshole during your day" and all that.
To bring the analogy back to browsers, if a website works fine for a client, they'll have no pressure to change anything on their end -- why upgrade from Windows XP when the site looks fine in IE6? Eventually the client is forced to upgrade -- normally by their operating system. That works, but what if the operating system adds another 2 years to their end of life -- do you just hang on and hope the shim / hacks you added hold?
Gauntlet AI I believe is correlated originally with Lambda School (YC S17). YC founders I believe are able to post job postings on Hacker News, although this might stretch the definition a bit...
Austen Allred started GauntletAI after his Lambda School bootcamp (now BloomTech) was fined, banned from participating in lending activities, and became too toxic to escape their old brand.
It's not clear to me why he gets to post privileged ads on Hacker News. Is GauntletAI a division of BloomTech, and therefore considered a YC portfolio company?
These aren't even job ads. GauntletAI is a recruiting play. They make money by getting strong candidates to apply and then collecting recruiting fees from companies for placement. They really do have people travel to Austin for some disorganized vibecoding classes with their vibecoded output used for resume building to increase their odds of getting placed (and therefore GauntletAI getting paid).
It's just the evolution of their bootcamp model updated for AI and the fact that their founder was banned from participating in lending agreements due to their deceptive practices. Now they're trying to collect money from the companies instead.
So this isn't even a job ad. It's a recruiter soliciting candidates. I didn't think YC companies were allowed to use their postings to advertise services.
As one of Austen’s haters, he’s fun to follow. End of last year he announced that he was going to relaunch a bankrupt company called Marin Software and have a documentary crew follow them and do it all with AI in a month or something.
A few weeks later, wow, they’ve booked $1.2m of revenue! And then he never mentioned it again. Documentary never surfaced. Website doesn’t work.
I feel we need a "proof of work by human" for emails. Something that could be signed that attests that someone took the time to write the email, not just sent a template / used AI to auto-generate a personal looking email, etc. Sure that could be gamed as well (have an AI write characters one by one to look more human-like), but taking more time usually is a fairly good blocker for spammers / salespersons / etc.
I would love for a proof of human work to exist, but how would you even do that? It would need to be monitoring the user activity in their email client, which isn't something that can be trusted by a server (and is pretty shady).
But that makes me think of Hashcash, that was developed to limit email spam via proof of work, but I don't think that has ever been used in practice: https://en.wikipedia.org/wiki/Hashcash (and of course wouldn't work for the proof of humanness you're talking about).
Imagine you have a problem you want to solve (let's say, identify an OCR'd handwritten character, e.g. the MNIST Dataset). You tell 3 agents "Hey, each of you take a stab at getting really good at recognizing characters from this dataset. You can take 10 refinement steps to continue to improve ". You can't give each agent unlimited steps of course, because you have a finite amount of compute.
So each agent goes off, and by the end, Agent 1 got to 90% accuracy, Agent 2 got to 80% accuracy, and Agent 3 got to 89% accuracy. Agent 1 wins, of course.
But then you look at the refinement steps, and after 2 steps, Agent 1 was _already at_ 90% accuracy. So the agent spent the next 8 steps basically not moving at all. Agent 3 on the other hand, perhaps was continuously climbing in accuracy at every refinement step, but hit step 10 and had to stop.
Now because you recorded every step from every agent, you know what you'd do differently next time -- you'd not allocate as many steps to Agent 1, and give Agent 3 more steps, because perhaps that might result in Agent 3 coming up with a better answer.
From my understanding, that's what they built in the form of a "search" controller -- a way to evaluate automatically and reapply how you could allocate resources more effectively, when applied to a new problem.
But I guess my misunderstanding is how applicable the search controller is when applied to new problems -- just because one pathway stalled early for one problem, doesn't mean it would work for another?
reply