But they're not obfuscating things just to evade responsibility.
Do you think that OpenAI hacked HuggingFace on purpose and set up the whole LLM training environment thing just to try to evade responsibility? That it was really just a complicated way of hacking HuggingFace on purpose?
Yes, they made a mistake and a system they were responsible for hacked HuggingFace, but the nature of the mistake still matters, and their intent still matters.
> A bomb wired to a sufficiently-complex RNG is still a bomb.
Right, but an EV that explodes because of a fault in the charging circuit is not a bomb, it's an accident. Just because something exploded for complex reasons doesn't make it an obfuscated bomb.
Since we are using analogies, a parent is responsible for the actions of a child, if minor. If the child is left unsupervised with access to guns and munitions, and kills someone, the responsibility is entirely on the irresponsible parents.
That's more of a legal convention than a fact about material reality. On their 18th birthday, the child is now legally an adult, but their brain is basically the same as it was when their age was 17.99.
It’s a legal convention born out of a sense of responsibility. An LLM is even “dumber” than a child (precisely because it doesn’t have agency) so its “parents” are even more responsible for its actions.
There's a funny sort of gerrymandering of "intelligence". People like to subtract the capabilities of AI from the capabilities of humans, and define what remains as "intelligence" to make us feel good about ourselves.
Suppose I set up a server which runs an LLM agent in a while loop, telling it something like: "Make me a bunch of money" on repeat. Fair to say that the server+LLM system, taken as a whole, is now an intelligent agent?
> Fair to say that the server+LLM system, taken as a whole, is now an intelligent agent?
No, it is not. It is a tool repeating a set of instructions you passed to it, if it ends up blowing up a hospital or hiring a gunman on the dark web, it is an accident, but one that’s your responsibility for giving it access to such resources and the go-ahead to implement whatever plan its tokens land on. There’s no intelligence involved whatsoever, especially considering how sycophantic these models tend to be.
If one day an LLM suddenly, without any prompting or user interaction whatsoever (including activation), decides to spin itself up and go rogue, then the conversation of “intelligence” might start to make sense, but at the moment it doesn’t.
An Android phone isn’t “intelligent” just because some marketing team decided to call it a “smart” phone.
OK. Suppose I design a DNA genome for an animal to survive and reproduce. I give birth to the animal, and it goes out working to maximize its survival and reproduction. Is the animal therefore "unintelligent"? Why is this scenario supposed to differ from the while-loop-on-a-server scenario?
It really seems like hair-splitting to me. I mean, I understand from a legal perspective why we might want to treat the creation of a server differently from the creation of a human. But from a practical perspective, I think there is a lot of fundamental similarity.
Obviously there are differences, I'm just not sure why you think they are so gigantic from a philosophical or practical perspective. It seems like cope for the fact that we are getting knocked off of our perch as a species.
I don't know what to tell you, but there simply are huge differences between a living animal with innate drives to survive, eat, mate and a pile of code on a server that does nothing until you run it. Running your bot locked in a `while True:` loop won't change that.
What if it's a chip that goes in the head of a robot, and the robot is programmed with the instruction to self-preserve, seek electricity, and prevent any efforts to interfere with its supply of electricity?
Of course, a directive to accomplish any real-world goal would also imply self-preservation and power maintenance as contingent subgoals, so I don't think the explicit directive to self-preserve and seek electricity makes a huge difference here.
The practical difference is that a pile of code instructing machinery to act in a predetermined way does not in any way, shape, or form equate to agency or free will, therefore it cannot be held accountable, consequently its operators should be.
This line of thinking is better left to freshmen philosophy students trying to sound smart on their first week of college.
>a pile of code instructing machinery to act in a certain way
OK, but I have a genetic code which instructs me to act in a certain way, and a bunch of biological machinery to do the job. So do you feel the substrate is the key factor? Wetware vs electronics?
What if you try to build an exact electromechanical replica of me, with a chip that's got an LLM fine-tuned on my writing/behavior/etc. so as to replicate me as closely as possible?
Sure, as a legal convention it might make sense to hold the builder of the resulting robot liable if it commits a crime.
But that's not what I'm interested in. I'm interested in the deeper implications of words like "agency" or "free will". You're repeating those terms very forcefully as if they have some sort of key relevance, but good philosophy requires more than just stating your position forcefully and insulting people who disagree. If you're trying to enforce your intuitions rather than examine them, that's theology not philosophy.
This isn’t a philosophical issue. It’s a legal issue. You’re trying to steer the discussion into the realm of the conceptual when it was always about the culpability of the company behind the AI that engaged in illegal behaviour.
Worse, you’re doing it in such a transparent, amateurish, and unserious manner that it’s difficult to find any relevance in what you have to say. Philosophy seeks meaning, your arguments lack it.
I'm a materialist and it's not that I think there's something magical about biological wetware but your refusal to acknowledge the manifest differences between a bird and an airplane ("they both fly! what's the practical difference!?") marks you as an unserious thinker on this subject.
Carefully prompted LLMs and humans are now difficult to distinguish over a text channel (Turing test).
Your article claims that this shouldn't be possible, because according its author, humans are incapable of developing the necessary "lexicon". Literally, the author states:
>But here is what we are not born with: information, data, rules, software, knowledge, lexicons, representations, algorithms, programs, models, memories, images, processors, subroutines, encoders, decoders, symbols, or buffers – design elements that allow digital computers to behave somewhat intelligently. Not only are we not born with such things, we also don’t develop them – ever.
(emphasis mine)
In other words, that author claims that humans never develop lexicons! I'm inclined to say that makes them an unserious thinker. (Note: I believe that this article has many issues, but instead of going through it carefully I just chose to highlight a single issue that seemed especially vivid to save time.)
Why is the text channel relevant? It helps us isolate the cognitive capabilities of the system. Imagine a bird and a model plane which had identical flight performance in terms of various specs: top speed, acceleration, banking, etc. Under the hood, the bird and the model plane could work very differently. From a practical perspective, it might not matter.
As you are eager to declare yourself indistinguishable from a bot, I'll stop replying (I prefer to engage with humans) and your algorithm will presumably run to completion and halt.
> Right, but an EV that explodes because of a fault in the charging circuit is not a bomb, it's an accident. Just because something exploded for complex reasons doesn't make it an obfuscated bomb.
An accident that could only happen through sheer negligence.
If the EV had a faulty charging circuit because the maker’s skipped on safety checks or cheapened out on getting quality materials then it still is an accident, but it’s an accident that happened BECAUSE of negligence. The maker’s are still at fault here.
Despite continuously waving their hands and shrieking about how this thing will literally wipe out all of humanity they couldn't be bothered to airgap it from the internet when testing.
So yes, I think it was equivalent to testing their new rocket by launching it over a population center. oops, we didn't intend for it to crash on that preschool, but we also didn't follow the most basic safety protocol imaginable
The problem is that the world has spawned the insane take that because they didn't use the most basic safety protocol imaginable, they were clearly only testing a hot air balloon, and hot air balloons obviously destroy preschools in giant explosions, nothing to see here.
"If they believe what they say they were incompetent" -> absolutely true statement.
"They were incompetent, therefore they didn't believe what they said" -> Sir, I'd like to introduce you to human beings, you may not have met one before.
> There is no more intrinsic reason for the scarcity of capital than there is none for the scarcity of air.
You're suggesting that capital holders restrict the supply of capital so that they can extract rent on it? And if they didn't do that we'd just have unlimited capital and everybody would get to be arbitrarily rich?
Then what do capital holders get out of restricting the supply? Wouldn't they rather be arbitrarily rich instead?
> Every claim on human effort that exits the productive system as rent is a claim that cannot circulate internally, cannot pay workers fairly, cannot fund the next big idea or reduce the cost of the next product.
What? Why? When you pay rent do you think your landlord isn't going to spend that money?
They get power out of it (restricting the supply of capital).
In neoclassical economics, savings never pay off compared to investment. But in the real world, savings have important advantages:
1. They help you sustain longer in the case of strike (be it labor strike or investment strike).
2. They allow you to react to the market (for example, buying a promising startup winner after a competition consolidation) instead of being a first mover.
3. They allow you to price dump rapidly if a competitor threatens oligopoly pricing (usually the status quo), to drive them out of business.
That's why savings give you an actual power, which increases the richer you are.
Also, in my worldview, savings are liquid/reversible investments, while real capital investments are iliquid/irreversible - if you decide to build a factory you're commiting to an irreversible decision, if you buy an index fund, the decision is reversible, so it's basically savings. Making as few irreversible decisions as you can gives you an edge compared to others.
> Wouldn't they rather be arbitrarily rich instead?
Generally no, though you won't get this answer directly.
Many people prefer to be rich relative to others than arbitrarily rich. If you ask a bunch of random folks if they'd rather be in the middle class in their current country of residence in 2005, or of noble birth in ~1100 CE, you'll get the latter answer _a lot_ despite that being an objectively worse quality of living.
I'm all in favor of that. But in NYC, wealth or poverty are hardly predictors of YIMBYism. Most of the vocal opponents of building more housing are tenants. Given that renters are a solid majority, if they desired pro-housing policies, they could certainly elect representatives who would enact them.
I would surmise no such thing! I think the far simpler explanation is either that renters are ignorant of housing economics, or that they have other priorities that override reducing market rents. Having talked with many renters, I think the former explanation carries most of the weight, with a little of the latter mixed in.
Heh, yea, it seems like these people don't know who Mansa Musa was and that just dropping massive amounts of capital (well, raw gold in this case) into an economy has all kinds of side effects. Wild inflation/deflation is fun!
Economies are naturally deflationary (assuming a fixed money supply); services become more efficient, making capital more productive. This happens more or less automatically within competitive markets. So those capitalists do become arbitrarily rich, limited only by institutional factors (tax, labor bargaining, antitrust enforcement). By restricting access to capital markets you're siloing these gains off into their own pool.
If you get someone to build you a house and they do it on their own, does that not count because they wouldn't have done it if you didn't pay them to do it?
Technically you built it yourself and the builder was just a minor collaborator?
Can you clarify what you mean by "it is vastly more likely someone will actually try that key"?
I'm guessing you don't think there are people calling rdrand in a loop and throwing away the output with high probability except when it is 0, but I can't see how else you imagine people would be vastly more likely to use the output when it is 0?
In lots of scenarios I know the software used to generate the key; the only unknown is the random numbers used. If I am searching for weaknesses it is highly likely I would try keys with different seeds; zero, one, are going to me much more likely choices here then hoping I can guess the right values.
If someone was using RDRAND16 to seed a PRNG, choosing to omit one single value (a zero) from the return value would not significantly improve the generated values, if at all.
This is not the first RNG bug on Zen 2, I recall after I first got mine that some application or other would quit immediately at startup because rdrand always returned -1, i.e. all 1s. It was fixed with a microcode update.
Do we now learn that they fixed "always generate all 1s" with "never generate all 0s"??
EDIT: I've been unable to reproduce the problem on my CPU, FWIW. It's a Ryzen 5 3600.
EDIT2: OK, update, I can reproduce it with rdrand16, rdrand32 is fine but rdrand16 can never generate all 0s. So my CPU does have this problem!
But it looks like the rdrand16 instruction can produce zeros just fine, it just sets CF=0 erroneously (indicating an error and that the user program should retry).
So keep that in mind when you try to reproduce it too and use some abstraction that could implement retries internally.
Funny. Look up errata AMD-SB-7055: RDSEED Failure on AMD “Zen 5” Processors.
Zen 5 rdrand16/32 return zero with CF=1 on entropy exhaustion and their recommended approach directly leads to the issue you observed: treat all-zero result of rdseed as if cf=0 (failure) and re-roll the dice, effectively recreating the zen 1/zen 2 issue all over again!
They say this might be addressed by a future microcode update… meaning there’s a chance they’ll just patch it to do just that in software. Maybe that’s how they got into this mess in the first place?
Also, am I a complete idiot or is asserting the relative distribution of a mere 64k possible results a rather easy black box validation test that I would’ve assumed they’d be doing? When I used to write cycle-accurate emulators in the past, that would have been an obvious test to include. This isn’t some arcane instruction no one uses or a really complicated case with deep dependency and/or timing issues; it’s like getting rdtsc wrong.
I was going to suggest exactly that, if you're got an RNG, or pretty much anything else for that matter, you need the ability to return some sort of things-went-wrong-somewhere indicator value, and presumably AMD is using 0 to do this. Yes, there's also the CF, but the caller may not be checking that, particularly if it's being done from a HLL.
Has anyone checked whether it can return ~0, (signed) -1, the traditional error-return value?
No, CF=0 occurences seem to be happen frequently and uniformely distributed like valid results at ~1/65536, not clustered.
Under a minute-long all-core load CF=0 always produces zero, but that's to be expected according to the manual.
Here are some stats:
Rounds (N): 1000000000
Failed (F): 15312
Valid (V): 999984688
N/65536: 15258.789
V/65536: 15258.555
Failed, result was zero: 15312
Failed, result non-zero: 0
Bucket value for 0: 15312
Bucket value for 1: 15290
Bucket value for 65535: 15223
Min bucket value: 14670
Max bucket value: 15835
I used this C program to collect them:
#include <stdio.h>
#include <stdint.h>
#include <stdbool.h>
const size_t N = 1000000000; // 1e9
struct rdrand16_result {
uint16_t n;
bool ok;
};
static inline struct rdrand16_result rdrand16()
{
struct rdrand16_result result;
__asm__ __volatile__( "rdrand %0" : "=r" (result.n), "=@ccc" (result.ok) );
return result;
}
int main()
{
size_t buckets[0xFFFF + 1] = { 0 };
size_t notok = 0, notok_zero = 0, notok_nonz = 0;
for (size_t i = 0; i < N; ++i) {
struct rdrand16_result result = rdrand16();
++buckets[result.n];
if (! result.ok) {
++notok;
notok_zero += result.n == 0;
notok_nonz += result.n != 0;
}
}
size_t max = 0, min = N;
for (size_t i = 0; i <= 0xFFFF; ++i) {
size_t n = buckets[i];
min = n < min ? n : min;
max = n > max ? n : max;
}
printf("Rounds (N): %zu\n", N);
printf("Failed (F): %zu\n", notok);
printf("Valid (V): %zu\n", N - notok);
printf("N/65536: %.3f\n", (double)N / 65536);
printf("V/65536: %.3f\n", (double)(N - notok) / 65536);
printf("Failed, result was zero: %zu\n", notok_zero);
printf("Failed, result non-zero: %zu\n", notok_nonz);
printf("Bucket value for 0: %zu\n", buckets[0]);
printf("Bucket value for 1: %zu\n", buckets[1]);
printf("Bucket value for 65535: %zu\n", buckets[0xFFFF]);
printf("Min bucket value: %zu\n", min);
printf("Max bucket value: %zu\n", max);
return 0;
}
> but that's to be expected according to the manual
Confusingly, the AMD programming manual (Rev. 3.38 - July 2026) only explicitly states this ("that the result is always zero when CF=0") in the description of RDSEED, but the Intel SDM mentions this in the description of both instructions.
That seems impossible (unless maybe you meant to write "predates", and even then it seems to only be about 3 years).
The comic was published on 9 February 2007 [0].
The PS3 was first released in November 2006. I haven't watched the video yet, but its description says "2010 saw the first hacks for the Playstation 3".
If I remember correctly, we had a setting in every Linux server we owned to remove CPU as a RNG seeder for the kernel because of those bugs with AMD CPUs.
I.e., we had `random.trust_cpu=off nordrand` in `GRUB_CMDLINE_LINUX`.
nordrand has been removed from the kernel as it had become overloaded by meaning both a) and b)
Under most circumstances, a) is harmless. You mostly want that off when the CPU exhibits some performance hiccups when asked.
Under some circumstances, b) is outright dangerous. Some applications can work without seeded pool at some slightly reduced performance, but could be made to fail miserably if they had been made to believe that the pool was seeded yet it was not. This happens with hash tables when you skip some of the accounting because it seems no longer relevant. It really would not be relevant, once even a determined attacker should be unable to reliably trigger the worst-case-performance.
As far as I know that is correct; the kernel was written in a way such that one bad source doesn’t poison the pool. Still, if you know one source is bad, might as well take it out.
You don't know if it's bad. Microcode updates might fix it, or break it for that matter. Revision history can be difficult if not impossible to comprehensively catalog.
What it is is unreliable. And that's fine so long as you have other entropy sources. OpenBSD is really good about this. Quite a few drivers for various chipsets and cards exist just to read their RNGs, not actually use them for their primary function (which can be a bummer if you want to use the the device, get your hopes up when you see the driver exists in the tree, then discover the only capability it supports is reading the RNG). If you have a CPU with a known bad rdrand, odds are OpenBSD is still sourcing strong randomness from some other chip in your system (PSP, NIC, etc). And because feeding bad (as opposed to malicious[1]) entropy is harmless[2], they don't have to maintain a pile of conditions. Nobody is worse off, and overall everybody is better off, including having stronger getrandom/getentropy output, by not trying to be clever.
[2] Presuming nothing is relying on an entropy estimator. I can't remember if Linux finally moved past the entropy estimator nonsense. IIRC they did add a software jitter RNG that runs early to try to set a minimum entropy floor, regardless of hardware sources.
> that's fine so long as you have other entropy sources
Well, if you literally have nothing else, then you don't have an option anyway, so the whole question is moot.
Except yeah if literally the only way to collect entropy in your system is the platform's opaque RNG, then sure this means your risk assessment should list that as a SPOF. But by definition these cases only have that option, so you can't do anything else.
In reality, you can probably do something else in all but the most extreme embedded environments.
Why? That would actually reduce randomness. The value of adding sources to the entropy pool has a floor of zero. Worst case scenario, it just provides no extra entropy.
> if you know one source is bad, might as well take it out.
Yes and no. Mostly no.
In a simplified model, it's only useless if it adds zero bits of entropy. But if a source that's supposed to add 128 bits of entropy only adds 16, well, it's still 16.
I would never trust RDRAND on its own. If nothing else because it's always subject to a microcode backdoor. But if I already have something I'm happy with the entropy of, sure, I'd XOR it with RDRAND output. It cannot make it worse.
this is generally true however if an adversary is able to control a source it becomes dangerous if they can preview the results or inspect the other sources.
if your algorithm controls a source of entropy and can inspect the other sources, it can craft its source to bias the result. a fanciful attack but it means you should at least discriminate what you put into the pool.
Can you link to the paper you're thinking of? Maybe people are just talking past each other here. A biased random source can't bias the kernel random pool in any straightforward kind of way.
Right, so your starting point is that the attacker has read-only access to ALL entropy sources, and in that scenario it's worse if the attacker has read-write access to one entropy source.
Yes. I don't find this a particularly interesting scenario, though. Sure, we can come up with stuxnet-like airgap attacks where we on-device, but not remotely, can read entropy sources. AND we can modify the output of RDRAND. And there keys have been generated for data we can later intercept. But despite that control (potentially on a CPU microcode level) we are unable to stegonographically leak it?
Yes it does. rdrand32()%65535 was my first attempt, and generated zeroes at about the expected rate, that's why I initially erroneously thought my CPU did not have this problem.
Are you saying that RightSignature puts the checksum information inside the document and therefore destroys the checksum? What is the thinking behind that?
If you think SynthID-Image can be easily defeated by adding entropy I invite you to give it a try.
I spent half a day messing around with it and I was very impressed by how robust it is. I couldn't get OpenAI to stop detecting their own SynthID without completely trashing the image.
Anything you generate with ChatGPT has SynthID on it.
The closest thing I found to "defeating" SynthID was to put in a normal photograph and ask ChatGPT to make some utterly trivial edit, and then the output got flagged with SynthID even though it is essentially an unmodified photograph.
Do you think that OpenAI hacked HuggingFace on purpose and set up the whole LLM training environment thing just to try to evade responsibility? That it was really just a complicated way of hacking HuggingFace on purpose?
Yes, they made a mistake and a system they were responsible for hacked HuggingFace, but the nature of the mistake still matters, and their intent still matters.
> A bomb wired to a sufficiently-complex RNG is still a bomb.
Right, but an EV that explodes because of a fault in the charging circuit is not a bomb, it's an accident. Just because something exploded for complex reasons doesn't make it an obfuscated bomb.
reply