I think it’s possible AI job disruption (or the threat thereof) could lead to a new labor movement capable of uniting people across the isle. It remains to be seen, but I think if the threat of a “permanent underclass” isn’t enough to get people to compromise and cooperate, nothing will.
People need to update their priors. The “permanent underclass” concern arose from a time when people thought AI was going to dominated by the first company to achieve it. But now we see that AI is becoming commoditized faster than almost any technology in recent history. IBM and Microsoft had a monopoly on personal computing for decades. AI is becoming a commodity before any of the AI companies have even turned a profit.
AI is going to be like electric motors. If you had a monopoly on electric motors you could control the world! But instead they’re commodity products mass manufactured around the world
At least for me, I think half the value in building to learn was that the knowledge and skills acquired in the process, especially cursory skills and knowledge, might be useful in the future, even if there was no obvious path to application at the time.
I built several projects at home, many involving learning e.g. graphics programming and rendering, that would never be useful in my professional work, but which were intrinsically interesting and enabled me to build other, more useful projects later on. It also gave me greater confidence in my abilities as an engineer, and cursory skills I learned in the process did help in my professional work.
Now it feels like what’s the point. The machines can or will be able to build anything I could want, useful or not, faster and with less frustration. I probably won’t be able to be employed as an engineer long enough to build a career on said skills. And I can’t mentally justify not spending that time with friends and family, when the expected return is basically zero.
I still find math, science, and engineering interesting and intrinsically rewarding, but in a closer sense to how one might feel about playing video games. The information is or will eventually be useless, so it isn’t worth spending a significant amount of time on.
Wow thank you this was actually very helpful for me in understanding why I’m feeling so demotivated by ai. Gaining knowledge, even if not immediately useful, to become a better overall developer was a huge part of why I enjoyed spending so much time building things in my free time. Now it seems pointless, because with ai, will that knowledge really make a difference? Probably not. Bummer, anyways I appreciate the comment.
I've felt the same for a while now. On my free time I have completely stopped building projects where the main motivation is professional learning. I now only build stuff I want or am genuinely interested in, and utilize AI in the process to save time.
Eventually I plan on learning some physical skills, like renovating, working with electronics and plumbing. I doubt robotics will render that useless very quickly.
On-prem AI is another solution. I’m surprised more companies aren’t leaning in this direction due to security/IP concerns. My employer is planning to spend several million on local AI hardware.
I’m not exactly sure what distinction you are trying to make, but a compressor that generalizes from currently observed day to potential future data _is_ a better compressor, as it will have better predictions as it observes more of the source data distribution. Maybe you are making a point about whether compressors can “learn” after compressing the input data, but this also isn’t a fundamental property of compressors. Also compressors do have to deal with non-stationary data sources, so it also isn’t correct to say a compressor only models stationary distributions.
But your definition of compression as prediction is tautological. How would you operationalize that into a measurable metric of compression?
The crucial idea is that once you do that and start optimizing for compression, at some point more compression on the training dataset can lead to worse predictions on the futures you care for. Which is very incompatible with the poster statement.
I agree it is tautological in some sense. It is more precise to say compression requires prediction, and rather than sampling (as in a generative model), we use the probability distribution to efficiently encode observations. A better predictive model of the space you want to compress gives you a better compressor.
To your second point, it depends on what you mean by “futures you care for”. In my view, if you optimize your compressor for a set of particular inputs at the expense of all other inputs, you are explicitly making a judgement about what future data you expect to see (and in some sense what data you care about). If the space of data you want to compress well includes all truthful, valid English sentences, then in the limit you need a model of the world to make accurate predictions to compress well.
If you want further evidence of the predictive ability of compression, Marcus Hutter has a talk [0] where he presents a few papers that used a compressor (gzip) to approximate the Kolmogorov complexity of a string. Using this computable approximation you can construct the algorithmic information distance between two string by compressing the concatenation of two strings (pieces of text, DNA sequences, etc.) and measuring how well they compress together vs. their independent compressed sizes. The idea is that the Kolmogorov complexity of the concatenation of two similar strings is lower than the complexity of two dissimilar strings, since you can learn the structure of the data from a portion of the data, and use the learned predictive model to better compress other parts of the input data. Using this idea, the paper’s authors were able to perfectly reconstruct phylogenetic trees and linguistic family trees _simply by measuring the compressed size of concatenations_ of DNA sequences and a piece of text available translated to many languages.
This is a lot less surprising when you learn how non-LZ compressors work, that is, by modeling a probability distribution and using those probabilities to encode information in the minimum number of bits required to transmit the data. A less obvious conclusion is that LZ compressors do this to implicitly, the length of each symbol they could emit (literal or match, etc.) can be converted to the probability distribution the LZ compressor induces, since the number of bits to encode the symbol is related to its probability by the information content.
A common design in compressors is to use LZ as a first step, but to then represent the constant data and/or offset-length pairs from LZ using an entropy coder.
Deflate (as used in gzip) uses a Huffman coder. LZMA (as used by xz) uses a predictive range coder. Zstandard can use either Huffman or FSE. Some high-speed compressors like LZ4 skip the entropy coding stage entirely at the expense of compression ratio.
Bzip2 is an interesting aversion of this pattern - it uses the Burrows-Wheeler transform as a first pass instead of LZ. Unfortunately, this is one of the major reasons why it's so slow.
If doesn't correspond cleanly. I can see why you draw the link, because LZ compression will replace words with symbols but BPE is a non-contextual entropy encoding while LZ is contextual and adaptive and that makes it very different. I think BPE actually has more in common with Huffman encoding.
Really cool project. I wonder what the future of LLM inference will look like. The Talaas demo is promising, but using an ASIC with weights in ROM means you can’t update the model (weights or architecture) without replacing the entire chip. SRAM isn’t dense enough to store model weights, but DRAM has bandwidth issues unless you use HBM which is expensive. Maybe novel memory technologies are the future (there are a number of emerging technologies in R&D), but they likely require breakthroughs to become commercially viable. Systolic arrays could work, one could imagine architectures where routing (architecture) is fixed but weights are programmable, or architectures where the weights and routing are programmable but the compute units are fixed function (coarse grained architecturally reprogrammable), or maybe the weights are in ROM but can be hot swapped easily with into fixed compute elements. Definitely an interesting and emerging field.
Once you have a mask of the positions you want to compress, you can generate a shuffle index vector from that mask to place the desired elements in the low part of the vector. You can expand the mask into nibble-sized indices using pext/pdep and some magic constants, then expand those nibble-sized indices into a vector of indices to use as the shuffle indices.
Yes, that's one approach. Another reasonable approach is to get out a mask from the comparison using `vmovmskpd` and use that to look up a shuffle constant, since there are only 16 possibilities. This also works well on NEON, although I wonder there whether it'd make more sense to find the shuffle dynamically rather than loading it.
I think this is one approach to AI safety and interpretability that could work, but would require labs to slow down to figure out how to extract circuits/algorithms out of trained LLMs rather than deploying the opaque artifact.
reply