I think the biggest architectural change here is them doing different compute for prefill & decode, with pretty much architecture from this microsoft research work from 2024 https://arxiv.org/abs/2405.05254, very exciting stuff!

For production loads? No one will run DS without expert parallelism so it’s not important for model to fit on one gpu.

Do you mean people will run multiple GPUs, or that streaming a small % of experts from disk won't completely kill performance?

Time for AMD to improve their directstorage story I guess. Is there a good writeup on how one would use it there ?

Same for gpudirect (more useful for scale-out or training).

Everyone is focused on AI but these are two interesting techs trickling down from the NVIDIA tree, relatively "easy" to use there, which maybe exist in AMD world but I somehow missed the docs and APIs and demos on how to use them...

The issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.

Totally, edited my comment to specify "based on the assumption above." The main takeaway I was going for was 0.1% is not a small number in this context

It's very unclear what's special in Rubin to be optimized for inference? I can see disaggregated bit (with having separate prefill and decoding nodes), but what else?

Lot more SMs & Tensor Cores for NVFP4 going by the looks of it.

What does it mean it's better at nvfp4 training? What's different between training and inference to make this true?

We're getting to the limit of my understanding, but I believe most Blackwell users still usually run FP8 passes through the transformer engine - they'll just store weights at NVFP4. Nvidia has model-specific stabilization recipes for NVFP4 end to end, but they're taking fixes all the time.

Nvidia says Rubin should have fewer stability problems training with FP4 because of hardware changes - "adaptive compression". There will still be outlier instability inherently, but something they're designing in reduces the cost of managing it.

But yeah, grain of salt - we haven't seen this in practice.

I'm also puzzled by that statement. The issue with training is (as I understand it) one of precision and the associated numerical stability. You need enough bits in order for backprop to function correctly.

Of course there are techniques such as quantization aware training but I don't understand why a datatype would work for inference but not for that.

You can also abandon backprop entirely but that comes with a whole host of tradeoffs and again why would it work for inference but not for whatever alternative training regime you selected?

See my reply to the GP comment!

It’s crazy something which basically changes the way US government works on a such a deep level is not even on a main page of hackernews.

The core insight there is to separate value semantics (no identity) from reference (itself) semantics (nullability). While this particular change can bring very limited amount of improvements it’s still does some - probably smaller to no object header + more guaranteed optimizations for variables on stack.

It’s when they land next part (nullability) it will shine fully - particularly on the intersection of not null and value. Alternatively if they introduce tearable semantics it will also shine - it would be possible to still optimize array of value classes, even if they are nullable (for example by having correspondent nullability mask).

So they are taking right step in a right direction. They are just trying to land this incrementally.

It seems to me that the "and a null flag" is adding a tremendous amount of unneeded complexity. Supporting null seems to be about avoiding C#'s ref/in/out solution. I suppose that, from a JDK dev's perspective, that's a reasonable requirement. I just hope that we actually get non-null.

I'm not sure the no-tearing rule is particular helpful either. Like, this is something folks get wrong all the time in regular java. There's plenty of places where we use unsynchronized classes and expect synchronization to occur in the containing class or other explicit lock. If atomic operations are a requirement, and non-atomic value types get turned into references, then value types seem pointless.

I fear we're getting something called "Value types" with none of the actual benefits of value types. Like "we heard you want something called value types so here you are". No, we wanted a way to declare arrays of structured values without having to deref pointers, or to store structures inline as a field within an object. What I've read seems to be not that unless the structure's total size is 63 bits...

There is obvious proxy to the amount of training data - revenue. And I think anthropic is way ahead of them.

DOJ puts an accusation with clarifying text in semi private document? They don’t do this, they do much worse things (and get, rightly, much worse response).

This document isn’t great, but comparing it to Trump administration actions isn’t great either. As well as focusing on it rather than on a substance of the article in question (which is, about Garry Tan accusations in a first place).

Thank you for your comment though it made me go back and reread the linked text more critically.

> Thank you for your comment though it made me go back and reread the linked text more critically.

Yeah he was just making stuff up.

I somewhat agree that the whole document is sloppy, but I also think this conversation overemphasizes what is wrong with it. It is not as if the document simply accuses the journalist and leaves it there; it actually elaborates on what it means by “violated HIPAA” in that context (and please don’t make me start comparing it with current administration behavior).

There is also a broader question of how to properly handle what are, quite honestly, likely bad-faith actors — the journalist in this case. Should the office simply ignore the smearing campaign and the lies? Maybe. But this is already an issue even before accounting for the significant amount of money — pushed by a very small number of people — being spent against the attorney’s office.

OP didn't say the office should ignore lies or libel. OP said the attorney general should be very sure before making severe accusations against the press.

I think both are right - shouldn't ignore crime and should be responsible in addressing it.