Or, we could introduce a software building code, the way we have codes for every other kind of safety-impacting product. But apparently software is never unsafe, we never need to protect people from software systems, and definitely shouldn't pass a law requiring those systems be protected adequately, with legal consequences for not doing so.
It pays for itself very quickly if you do 24/7 generation. Use an AI agent that orchestrates other agents working on many things at once constantly. If speed is a factor, you'd not buy a Macbook, you'd buy dual RTX 3090s. About the same price, but at least 6x faster than M5 Max. The benefit of constant generation is you can do a lot more research, coding sub-agents, experiments, etc in parallel when you're not "at work". You end up getting a lot more work done than if you only sit there babysitting sessions.
Yeah I'm surprised no one pointed this out, if something like persistent agents gets more popular/useful, the local option pays for itself surprisingly quickly.
> Maybe we need "quality certifications" for AI agents - essentially eval suites that demonstrate those agents won't cause harm under reasonable patterns of usage
Based on how LLMs work, this is impossible. You cannot predict how they work, it's literally based on a combination of random seed and a mostly-unpredictable path walked based on every token of input.
You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it.
> the "industry standards" common in, say, electrical engineering and other disciplines are sorely lacking here
100% agreed. We have ignored SWEng's lack of discipline for too long. Now that the SWEng isn't even a human, we are looking at total catastrophe (on the scale of improperly built buildings falling down on people or catching fire) if we don't adopt a software building code.
> You don't blame a knifemaker for somebody getting cut by a sharp knife. AI is a knife. Very handy, very dangerous. We have to use them safely, that's all there is to it.
If I grossly neglected to maintain live deadly bacteria in my containment facility, am I absolved of blame? Since, you know, the bacteria is the real bad guy who should be put in jail?
I agree that we should basically be requiring hand-written-only design docs, because it should force people to make sure they know what they're getting someone else to read. But there's two problems I run into:
1) A lot of people who write design docs, RFCs, etc, don't write them well. I end up needing to get them on a call and explain their entire idea to me because it's the only way to pull the details out of them.
2) Regardless of how much I write by hand, I still have engineers who are so incredibly lazy that they just don't read the docs at all. They can't be arsed. So I have to get on a call and basically explain the whole doc to them.
This is starting to lead me back to what other people hate: meat puppeting. Telling Claude my idea, Claude writes it up, and I ask that engineer to ask their Claude to read my Claude output and summarize it for them. I really want a better solution, but our engineering management is almost nonexistent, so nobody does anything they don't feel like doing.
Your second paragraph is the impetus behind Agile Software, and we've all seen how fantastically that failed. Lots of code pushed out quickly, but also a lot of really shitty products, uncertainty, never-complete projects, dysfunction between teams, etc.
> Software is not like a nuclear power plant or offshore oil platform
No, but it does impact people's lives significantly. How many times has your personal information been leaked by a company making products by people who didn't care? Who would have predicted that a security company's terrible QA would lead to 8.5 million crashed systems, 42,000 delayed flights, 10,000 cancelled flights, and over $10B in economic losses? I'm sure the developers just said "not our problem". But their lack of concern, and "just throw shit at production" mentality, had real world consequences.
- One of the things about design you notice as you get older, is the bigger a software application is, the worse your options are later on. The design doc can be considered a proxy for the application: the bigger your design doc, the bigger the application will be. You should still try to think about as much of the app as you can in your design doc (don't intentionally try to make it small and miss things). But if you start to notice this doc is getting pretty heavy, that may be a hint that the application is trying to do too much.
- Next suggestion: make C4 diagrams and split up your design doc by each level. You don't need to make perfect C4s, or even have a good handle on how to define them. You just need four levels of abstraction by which you'll give the reader a picture of how your application works and fits into larger concepts. This will help you and the reviewers understand it better, and will likely also elicit more questions, which helps you better define the design.
SLOs encompass everything else because they're the objective of your product/service/application. Monitoring & Alerting comes from the Interfaces & Logging because the purpose of the Monitoring & Alerting is to "consume" your Interfaces & Logging, in order to meet your SLOs. Security / Privacy / Legal are under SLOs because the extent to which you perform all those things again comes from SLAs (which are basically a bunch of SLOs combined with legal stuff). Infrastructure comes from SLOs because you can't meet your SLOs/SLAs without Infrastructure built to fit them.
You might as well ask why knives are sharp enough to cut you, why hammers are heavy and blunt enough to destroy things, or why guns fire bullets so quickly that you can't react to them. These "behaviors" are not strange side effects, they're inherent and necessary.
You can't trust an effective AI any more than you can trust a sharp knife. If somebody asks you for one, it's probably not a good idea to throw it across the room at them. You will have to figure out how to get it to them safely.
> if OpenAI can’t use the compute, someone else can
How? The hardware is in OpenAI's datacenters. Does Nvidia have a couple hundred semi trucks, contractors, and IT technicians, to repo the hardware and resell it to someone else before it's lost most of its value? These chips will be replaced approx every 3-4 years. So if OpenAI tanks, after Nvidia pays for and waits for the process to collect the hardware, they then have to sell it for pennies on the dollar. They lose almost all the investment.
Also consider that SpaceXAI already had datacenters full of gear that they basically weren't using because nobody wanted their product, so they now rent it to Anthropic. The demand for hardware isn't really there at the scale of OpenAI.
>Does Nvidia have a couple hundred semi trucks, contractors, and IT technicians, to repo the hardware and resell it to someone else before it's lost most of its value?
reply