Are you sure that will be enjoyable? Lots of articles out there about people opening coffee shops with idyllic dreams of what it will be like, only to find the reality is a daily grind (hehe) of dealing with delayed suppliers, broken machines, absent staff. All that not-fun stuff falls on you while the customers sip their cappucinos.
This 100%. I just requested a hackathon for performance improvement (might be wasted effort).
We are pushing tons of code and now our CPU usage has grown exponentially over the past year because the bad engineers just ship whatever Claude gives them and do not think about the consequences.
Our biggest consumer of CPU right now is HTTP connection churn because engineers are creating new clients every request we handle. If the engineers would just think for a second, push back on Claude, even Claude would tell them this is bad. But they don't... Platform engineering is now 10x harder with terrible engineers and unlimited code machines.
Don't even get me started on ffmpeg usage, engineers act like the resources are unlimited.
When I worked at a particular fruit company in Cupertino, my boss had been asked repeatedly by the developer teams to provide a tour of the data center we'd recently finished building out, which housed the servers their code ran on.
He gave them a lengthy, grueling, hyper-detailed tour of the entire facility, encompassing the HVAC systems, electrical systems, network and computing systems, finishing with about 20 minutes where he had them stand inside a hot aisle that he was just outside of, giving a fantastic soliloquy on the importance of code efficiency, and the consequences of ignoring it. It was hilarious to watch from the comfort of the cold aisle, knowing full well what he was doing.
The scale of a cloud datacenter is pretty awesome to witness. I had the opportunity to visit a large cloud company’s data center years ago, it was immense in scale and (mostly) very well thought out.
I’m personally of the opinion that everyone who works in software should have to rack a server and bootstrap it. Physically mount it, cable it, and get the *nix distribution of your choice running on it, serving Hello, World.
Everywhere I’ve worked, there is a marked difference in the quality of engineers who had played with hardware - even those who merely had expressed interest in it, and maybe had an RPi - and those who had not. There is something about physically touching the thing that runs your code that makes you better at it. Maybe it’s a correlation between “wants to play with something unnecessary but adjacent” and “curious enough to ask why more frequently,” but I swear, it exists.
I think it maps to the proclivity of the engineer to want to understand the multiple layers that apply to whatever they're working on. The more curious, the more likely that engineer is to gain a better understanding of the whole, and therefore presumably be able to do better engineering.
Busy servers generate more heat. Inefficient code makes servers busier. There’s a real physical outcome for bad performance code at scale.
But you don’t really want lots of idle or underused servers, aside from burst capacity management. So if they universally and consistently wrote more efficient code I’d expect a smaller datacenter. Full of still busy servers but just less of them.
He was showing how complex and expensive a data center is and giving them real-world visceral experience that inefficient code has real-world implications.
Exactly. For most of my career, bad engineers (or juniors who were earnestly learning) would struggle to even create output that compiled, let alone ran. And it would take them a long time to implement something poorly. So the blast radius from their incompetence would be limited. Today, a bad engineer can churn out ten thousand lines of garbage that actually compiles and runs, before my first cup of coffee. If not on a tight leash, they'll sink the whole codebase or inundate every senior person with garbage code review requests. The blast radius is unlimited!
> Today, a bad engineer can churn out ten thousand lines of garbage that actually compiles and runs, before my first cup of coffee. If not on a tight leash, they'll sink the whole codebase or inundate every senior person with garbage code review requests. The blast radius is unlimited!
The term for that is "10x AI engineer." Anyone who has anything negative to say about such people is just jealous of their insane productivity and speed.
FYI, your comment showed up as [dead] like you were banned until I vouched for it, but I don't see any reason for that. Just thought I'd let you know in case you want to follow up.
> You can make your own numbers look incredible while reducing the throughput of the entire team.
Taken how some companies award promotions and bonus this is actually a double win. You not only improve your own numbers but also make this of your competition worse! /s
I just left a backstabby stack ranked company and so many people were sabotaging colleagues or teams they were in competition with. AI psychosis has been enabling terrible management practices as well as terrible engineering practices.
Sadly AI also amplifies both the good and the bad managers and tech leads. If you have managers who fail to set a clear direction or tech leads who fail to define a good framework for developers to operate within then AI just means the poorly directed developer effort goes further in the wrong direction faster. Meanwhile teams with well-specified goals and better structure and processes can exploit the benefits of AI tools much more effectively.
In the before times, a small dedicated team could hold back against bad engineering culture and intentional enshittification. Now the balance has changed.
> We are pushing tons of code and now our CPU usage has grown exponentially over the past year because the bad engineers just ship whatever Claude gives them and do not think about the consequences.
Sometimes this can be a death by a thousand cuts. Any individual change may not impact performance to a noticeable degree but when they're pumping out a 10x increase in commits it can be a slow decline.
Just look at how they're merging ~300 commits a week into bun.
That really sounds like the kind of people who will complain in university about being forced to study calculus.
Natural consequence: Then they never grasped the concept of computational complexity.
O(n) Vs O(n²)? They have n, what's the difference? Python is fast enough. The only thing that matters is shipping features fast! Features! Our competitor will have this next week, we need to write code fast, everything else is a matter of adding more compute, which we will pay with revenue!
As a side note; There is plenty of useful math that many people need (or would benefit from) in the real world. Much math curricula is arguably poorly aligned to needs, as it emerged from various historical flukes of design-by-committee. Popular math curricula is not perfect by any means. I recommend looking at high school or college textbooks and considering whether it is well prioritized.
The missing part of mathematics education, IMO, would be to focus more into developing the intuition of what something means, instead of the current focus on getting some (numeric) results.
That was one of the biggest changes that happened with common core. Common core itself didn't really mandate this, but it took a reset of the curriculum to insert the new ideas like an emphasis on intuition rather than rote arithmetic.
The problem with common core is that it replaced building arithmetic fluency with less rigorous ways of building intuition, leading to a generation of kids who never gained the basic math skills that are foundational to actually understanding the intuition.
You can't just skip arithmetic drills in favor of intuition and still come out with students who are prepared for further study in math. There's a reason Kumon etc are so popular - parents are replacing the lack of mathematics drills in schools with after school options, leaving behind all of the kids whose parents can't afford it.
Montessori. My kids’ school has kindergarteners solving trinomials [0]. My kids learned multiplication and division by 3rd grade. They’ve also memorized powers of 2 up to 2^20, but that was more me telling them about exponents one day, and then giving those as examples. However, they did learn binary trees (and binary) from their school.
Kids can absolutely learn complex concepts early, they just may not be able to formally write them out until a few years later. But they’re kids, so who cares? By the time they need to be able to write out math equations, they’ll understand the underlying concepts on such a deep level that it doesn’t matter how long you make it, they’ll plow through it.
I was a skeptic, and am aware that I sound like a cultist, but damn if Montessori isn’t something special. I highly recommend it.
You don’t learn Big-O in Calculus because it requires CS algorithmic concepts.
But the core question of “how does this behave as N -> \infty?” is asymptotic behavior (ie: limits) which were developed for calculus and are very much part of the foundational calculus canon.
Big O notation predates Computational Complexity. It was originally used to guesstimate asymptotic behavior (limits as they approach infinity). The related Little o notation (even more specifically limits of already large values as they approach infinity) is directly related to early attempts at defining differentiation and differentiability and still sometimes show up in Calculus next to derivatives to explain how derivatives work.
The visual interpretation/intuition of Big O notation can be a useful way to build a visual intuition of what a function's derivative and integral "shapes" may look like, so some Calculus books teach Big O notation, too.
If your company has sketchy process and minimal checks on the software quality, of course it naturally follows that they hire bad engineers as well. fortunately, that also opens up an opportunity to be a emergency leader with an eye towards promotion.
> If your company has sketchy process and minimal checks on the software quality, of course it naturally follows that they hire bad engineers as well. fortunately, that also opens up an opportunity to be a emergency leader with an eye towards promotion.
AI disease is encouraging "sketchy process and minimal checks on the software quality." QA has been eliminated from my team, and the QA engineers that are left have been declared to be developers now.
Gotta move fast, and I guess making sure the stuff we ship works was "slowing us down."
The bigger issue (I find) is that the pipeline for finding those engineers is completely fucked.
Hiring pipelines that tested the wrong thing have existed for years but the problem is magnified 10x when you test for something that weakly correlates with ability at best which an AI can do better than a human.
This is leading to stuff like incompetent junior-level engineers being hired as principals.
My company's hiring process is basically: Can you code a product quickly with AI? Can you understand the generated code?
That's basically it. It is surprising hard to find people that can do both, but engineers are becoming much, much better at the first gate while flaming out on the second.
> a emergency leader with an eye towards promotion
i've gone this route a few times in my career, it's very stressful and involves angry/panicked people and many all nighters. Also, the glory fades fast. would not recommend.
The actual issue is not the "bad" engineers but the bad organization. While people are allowed to push and merge whatever crap is generated there is no point to do otherwise. Even if you do care about performance, "good" code your teammates don't and close more tickets and are better by many metrics.
If you are closing one ticket per week with "good" code but your teammate does three with "bad" code - it's actually you are a bad employee. Also they may say you are a toxic one.
It means the company is measuring the wrong thing.
I would much rather have someone on my team who ships less but whose work I can trust than someone much faster whose changes leave me wondering what problems we’re going to discover later.
And when production breaks (and it will), I need the person who made the change to actually understand it well enough to help fix it, instead of showing up with no idea what is going on.
You can obviously be an asshole about how you do it but I don’t think pushing back makes someone toxic.
You need to be flexible and compromise when the business trade-off makes sense. But you also need a backbone. If you think something is going to cause real problems, bringing it up is part of the job.
Do you have SLO/SLAs defined? Are you monitoring performance? CPU usage? Cost increases?
You should have plenty of data to review regularly and push back on any teams that are causing problems. That's a process problem and you need a process for it.
Trace the increase back to specific deployments, call out those teams, and make them fix their shit.
Yes to all, but it's just me responsible for monitoring performance/cost. I either push back or fix it myself, but I am behind. So this is an attempt to give/push more ownership to the teams.
A hackathon for performance improvement screams to me "you'll be re-interviewing for your own job." I don't foresee this going well for the company. Best case scenario: you wildfire churn all the chaff engineers which limits the company's throughput until they're rehired... with no expectation the new hires will be any better. You also lose A LOT of domain experience and are self-inflicting brain drain.
I'd try and push for some "lunch and learn" meeting where the engineers get lunch catered and in exchange sit in on a meeting where you explain your point of view. Without monetary incentive it'll be hard to change the culture, but not impossible (and food goes a long way in greasing the wheels).
The handling of backpressure is such a good example to distinguish bad from good engineering. A good implementation even presents all the typical "clean" code indicators: DRY, KISS etc.
My company got rid of line management. I now report to a director with over 20 reports and barely any time for career development. Our performance reviews are AI generated and we’re losing engineers. The industry has gone completely insane.
To a first approximation, basically no one in the industry does. Users regularly report performance regressions in basically every major piece of software, most of which would have been caught by performance testing.
It can sometimes be difficult to predict what will cause performance regressions when developing an application or web server with test data. So you do your best with what you know at the time of release, and then deal with performance issues when you know what is causing regressions (e.g. saved searches) and can then make measurements and profiling to see what exactly is causing the issue.
Likewise, improving performance pushes the limit at which performance regressions happen allowing more data (documents, triangles/pixels, etc.) to be processed. This allows things like more complex game graphics. That in turn makes it harder to improve performance for the next round (more advanced triangle/face culling and pre-processing).
There can also be trade-offs with things like data layout, memory usage (caching and memoization), or implementation. For example, when processing XML/HTML data you could use DOM (more memory and upfront parsing, but easier to perform complex queries across the data), SAX (less memory, but more complex to process due to tracking state), or reader API (similar to SAX but a different processing model).
There can be challenges with various features, such as type checking with higher-order generic types. Others add various levels of overhead, such as parsing a program, constructing an AST (Abstract Syntax Tree), generating an IR (Intermediate Representation), the optimization passes and final code generation.
JavaScript evaluation for example is complex. Before Chrome the approach was to use a slow interpreter. IIRC, Chrome was the first browser to introduce JIT (Just-in-Time) compilation, leading to faster JavaScript and eventually more complex applications running in that language. Modern browser JavaScript pipelines are complex in order to achieve and maintain performance:
1. start interpreting the code on the AST so it is run immediately (or only rely on (2));
2. generate a machine code equivalent of that interpreted code (for faster baseline performance);
3. run more aggressive optimizations on known types based on profiling/analysis for even faster performance, including special handling of things like asm.js.
Startups have limited resources, we prioritize the best we can. As a staff engineer it's my responsibility to bring this to my company, but it's not an overnight process.
why wouldn't you use those limited resources to test things before they go to production, instead of creating more crappy things that go to production?
What do you test? How much time/effort do you spend testing each of those things? How do you know those will be performance issues?
There's only so much you can do -- making reasonable assumptions, choosing suitable data structures, database indices, testing various workloads, etc. -- in a limited test environment.
You may spend a week optimizing a feature that only 10 people use or that never gets to the point where it becomes an issue. Or you may have something that can't easily be tested at scale, such as various counts or other dynamic data that are determined by complex queries that you may (likely) find you need to cache but don't necessarily know which values will become issues until you start using the system.
We have a test team that runs various test suites(automated) and manual testers that test functionality. They spend their entire working day, every day testing.
For my small team we are not creating our versions, we're just more likely to find a self hostable version and use an LLM to help setup in production. For us it's a middle ground between the build your own SaaS and paying $10ks. But in some instances we do still pay for the other company to own the on call.
Usually when I'm working in one part of the codebase and I have sample data or something at a specific path on my local machine and Im testing the same thing over and over again will I make a Makefile or something and info/exclude it to help me keep focused. That's one way I use it.
I use git worktrees pretty heavily in my own workflows (I worked like an AI agent before AI agents made worktrees cool). I like to track my ephemera/utility scripts in git, so what I do is keep a private ephemera repo for those, and then use `git worktree add` from the collaborative repo to check out the branch I'm working on there into a subdirectory of my ephemera repo.
git-home/
company-project/ <-- git repo with main checked out
ephemera/ <-- my private repo
my-data-script.py
work/ <-- gitignored
company-project-feature-X/ <-- worktree on feature-X branch
company-project-feature-Y/ <-- worktree on feature-Y branch
This way, too, I can easily use the same ephemera scripts across multiple branches, or even multiple repos, concurrently.
How do you manage the lifecycle of the worktrees in practice? Is it mostly manual git worktree add/remove, or do you use aliases/scripts/some other tools?
I would ask for references for the suicide claims, so others can assess the impact themselves. That's a very serious claim to provide without any proof, especially to a group of people who very well be going through the same thing. I am not saying it did not happen, only it's the right thing to do.
Any other engineers just living life frozen at this point. I am unable to make any life decisions because it seems like I won't have a career in the near future. I am unable to purchase a home to settle down for my family, because dad might not have a job next week. I know I am fortunate to have a job, many don't, but fuck if this career isn't the worse thing ever for my overall health and happiness.
Yeah I'm there with you. I got lucky as a kid with delving into this as a hobby and it turned into a professional career. Thought we could change the world for the better, what we made instead was social media cancer and LLMs that can pretend to make everyone 10x more productive. I loathe it.
Absolutely. News like this is so hard to ignore. Nervous as hell to drop big money on things the family needs right now. Grateful to have a job, but life overall was just better in almost every sense before AI became part of our daily vocabulary and layoffs occurring every couple of weeks.
I've been out of work since almost a year ago after getting laid off and the same is true for a lot of my coworkers; the job market is absolutely broken in half for a lot of different but related reasons. Thankfully I have significant savings and low costs so I can just coast and do stuff in my own time, but the same hasn't been true for others I know.
Frankly I fully expect people to get even angrier once they become unable to meet the bills and companies still tout the whole AI line.
It's great, I'm in my 30's, two kids, relatively high income family, but can't make any financial decisions because it feels like on a dime we will be living paycheck to paycheck all while having zero debt. It's a stress I never thought I would have.
Feels like the most important thing we can do is document it all. Save multiple encrypted copies. There is so many things worth documenting, but don't let others tell 'you' what happened. Write it down.
reply