FWIW Incus is really good here. I don't quite have it fully working for headless gaming containers sharing a single GPU, but I'm very confident it will work at this point.
If you're going to expose static secrets to your app, it kind of doesn't matter whether they are in a single vault or in a gpg encrypted file or something. The real benefit of a vault IMHO is to distribute dynamic certificates instead of a static secret, so that way it's time-boxed, and the static secret (private key) never actually leaves the vault.
Not directly related to openbao, but I've thought about this for awhile that I'd like to use ssh certificates instead of ssh keys to allow an agent a time limited SSH access to a server, and that seems to work for gating the initial connection, but I haven't yet figured out how to enforce the established connection dies after a certain time. I could maybe hand roll a solution with ExposeAuthInfo and ForceCommand wrapper, but it feels like this sort of thing should be handled delicately..
NixOS has transformed my use of Proxmox. I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all. Proxmox is just a target, and I've abstracted things enough to where I can deploy the same machines to libvirt on a local machine too. Why would I need to let my agent into my PVE box? I haven't looked at incus, but if I wanted to run the full stack declaratively, nixos and LLMs are so powerful now that I would probably just say to run libvirt and ZFS on nixos natively.
> I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all.
I would love to know more about how you do this, particularly the deploy part. I'm considering moving away from Ansible, but haven't had the time to dedicate to exploring a similar Nix experience.
This is what I've built [1], and I'm using it daily for things like agent sandboxing when I deploy it builds a new image completely replaces the OS disk - on upgrade it only keeps its data drive mounted to /var.
I'm happy to see this, and I have lots of thoughts about this. Building declarative services on Nix is a far superior way of distributing Linux to VMs than most any other way I've tried. I am working [1] on very similar things, but I've been leaning more on the self-hosted path, my VM template targets libvirt and Proxmox VE with a single CLI api. I even have an experimental branch that targets DigitalOcean. For VMs especially, I want my OS to be immutable. My VMs should contain no state other than my application state. Upgrades should be a full image replacement and reboot.
So in my template, I have created the VMs with two disks: first one is for NixOS and is built from an image, and it is read-only. The second is mounted to /var and is used for all system configuration as well as application state. If I have multiple VMs, they can all share the same base image (thin provisioned). That's the mode that I want for my deployments of services, immutable and as stateless as possible. For agent use, its different, you actually want a mutable NixOS root so that the agent can do what it wants.
I built three modes: immutable, semi-mutable, and mutable. mutable removes the read-only lock on the root, and just lets you manage the VM as a pet. semi-mutable adds an ephemeral overlayfs that gets wiped the next time you upgrade the base image. So that gives you kind of the best of both worlds: an immutable read-only base image and the ability to "nix profile add" whatever you (or your agent) wants, but with the contract that these imperatively installed things will disappear the next time you upgrade. Are you planning on adding a LICENSE to your machine0-nixos repo?
I tried out NixOS a few years ago but recently transitioned back to Rocky Linux and Ansible. I know that Nix is treasured by some but it always came across as an esoteric tool for functional programming idealists. I found the community to be split between people who were genuinely helpful and people who were just... not.
I found Nix just really hard to work with. The documentation was just so poor and every aspect of Nix just seemed to be divorced from pragmatism.
An example of this, years ago, was that I wanted to do something VERY simple: codify the creation of a directory in NixOS. It took me 6 HOURS to find the relevant code for doing that. I couldn't even get an answer out of the Discord server.
I don't know if I'll ever pick it up again. The learning curve was incredibly steep and it's just not on job descriptions and I've never worked in a shop that has used it. I tried it out as a curiosity, found that it was hair pullingly frustrating to use, and moved on.
Yea, I totally get it. The thing is agents change the game. You no longer need to worry about the learning curve or how best to implement.
Just point your agent at a machine0 VM and say "make a machine that does X", then you get code you can use to build on any nix box and you'll always get the same result.
Once you experience this, it's hard to go back to a "traditional" OS, you'll want to nixify everything :)
I could say the same about Atomic distros and reproducible builds. Nix no longer has an exclusivity here. I find Ansible w/ Alma 10 Atomic (there's an rpm-ostree driver, cargo, pipx, etc.) gets the job done with less tokens, and less cognitive load on my team, that already doesn't want to work on configuration. Does Ansible have perfection? No, but neither does posix. Papering over this bump with a perfect abstraction isn't as useful in our imperfect world.
All that being said, I'm glad people like you are pushing Nix forward. Competition here is a good thing.
Ansible is a different tool than Nix. It's been specifically designed to fit your exact use case. If ansible works for you and your team, there's no reason to drop it.
I've never had a moment using nix where I've felt "wow I couldn't do this with any other tool". That's not really the benefit of nix. The benefit is the flexibility of the idea and the tooling.
It's a flexible enough platform where I can use one config to manage 5+ computers in my house with various operating systems. I can use it to easily setup all my dependencies for my work projects. I can use it to build my work projects. I can use it to create vms to test those projects on. I can use it to build 3rd party projects without any instruction.
All that required was a few months of pain learning the fundamentals of nix and I've received the rest for free pretty much. It's the epitome of upfront work for long lasting benefit in my opinion.
I do think the community recognizes this to be an issue and is steadily working on improving beginner-friendly docs. I am about seven years into using Nix for various things, and can mostly solve most problems, but I won’t deny that the learning curve at the beginning was brutal. The real and most meaningful unlock is learning to read the nix language well enough to follow what is happening, then checking out nixpkgs locally to look at crate derivations and such to understand what idioms exist in “real code.” The module system also took ages to click for me, but was a big unlock.
Anyway, I hope the community continues to make the onboarding process more welcoming and easy. Personally, I am hopeful that guix will really take off at some point, because even though I get it now, I’d way rather read lisp than nix.
I am learning Clojure this week, and my test project is a calculator / unit convertor [1]. I wanted it to run in the CLI and on the web, so it targets several hosted platforms: Babashka / JVM / ClojureScript. It's a single code base written in cross platform .cljc files. I already have about 250 tests written for the abstract calculator API, run as a test matrix across platforms, so the project is already in a good place for testing a new runtime.
I just learned about basilisp from the parent comment, so I asked Claude to add Python support to the same .cljc files I have, and we finished the port in about 30mins, and then fixed Python specific test cases for another 30 mins, but now all of the existing tests are passing. That's impressive in several ways.
Portability is achieved by testing. You have to put the platforms you want to support into your test harness, and the earlier the better. A calculator is purely functional, so this is a fairly straight forward port and really easy to test for. I'm not sure about larger projects, but it seems like there is something seriously right about Clojure's design that makes this easy to do.
Olives are a fruit and yet they are disfavored by the movement. Coconuts are a seed but they are considered acceptable. And within seeds, maíz, soy, and safflower are all very different botanically.
Coconuts are fruit, actually. Also while corn kernels and soy beans are technically seeds, they are, at least in my opinion, pretty far from the vernacular definition of seed. Part of this is that soy is a legume, corn a cereal, unlike safflower and rapeseed, which might matter nutritionally.
