just heard on Risky Business pod that they apparently have taken it down, I distinctly remember reading it two days ago, came here to find the link, found your 404?
is there a realistic way to distill 2 consumer hardware friendly models with max ~200B and ~20B? Qwen did it, but would it be possible for 3rd parties (unsloth etc)?
Yeah, why not. Toughest part is running the hardware so you can create the traces for downstream training, but once over that hump, nothing would stop you from doing that no.
wanna chime in, recently tried vLLM to consume a NVFP4 Gemma4 safetensor model and see how the batching can show up in nice t/s numbers.
it's slow to start, it's Linux only, it doesn't like WSL much, ended up with either old or nightly container builds, I more or less have given up.
Appreciate how llama.cpp simply works and does things fast and obvious
my latest experiments with local LLM (mistral coder variations) fitting in older 6 GB GTX1060 were disappointing as long as you try to hook Copilot (CLI or VScode) to it and are used to provide a lot tooling. this seems to bloat initial prompt to 20k and more which seems the bottleneck if I did not completely misconfigured things. output tokens/s are more than fine, but PP is frustrating / unusable.
e.g. if on an annual plan? 0x will be gone, but there are okay 1x and 0.3x models left. I am pretty much curious how the early may test invoicing will look like. current setup of tools etc. is way too chatty eats up 1+M token per PRU easily. not sure how much is cached.
I had some 3x request that I did the math for fun on long running task, and at API price it would have been ~$260 that the in/out and cache. All that for $0.12.
not sure how much resources it took from my WinARM Win11 as store app, but it's buggy in either release or beta since months as it loses network connectivity after some time...
I think it's more Dave that "sucks"...click baiting titles, no need to and ever and ever repeating that he was at MS.
He should do Podcasts as there is no value in videos only speaking (feels like 1.5x btw)
