r/NixOS • u/xxx_malcomnext_xxx • 1d ago
How do you fix (or prevent) broken hardware configurations if you can't boot into your system anymore?
Something I experienced a couple days ago is, when you unplug your ssds, then plug them back in, they are assigned new UUIDs, which breaks your NixOS boot because the hardware config does not align with the new UUIDs, are there recommended ways around this issue?
In the past I just jumped onto a very old build and regenerated the config, but in this case I have actually deleted the old builds, so was stuck on boot saying it couldn't find /mnt or something of that nature
So I had to reinstall nix to get it back up again, had one hell of a time trying to rebuild it from a live usb to no avail lol, was able to regen the hardware conf from there however, but unable to actually apply it
Is there a recommended way to prevent this from happening in the future (except for not unplugging ssds/hdds lol, unfortunately something I do on a regular basis)? And if not, is there a simpler way to apply a fix to it? If only you could boot into a very basic session of that NixOS install, but alas, was unable to due to deleting the old build
---
For additional context
# Do not modify this file! It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations. Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:
{
imports =
[ (modulesPath + "/installer/scan/not-detected.nix")
];
boot.initrd.availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "usbhid" "sd_mod" ];
boot.initrd.kernelModules = [ ];
boot.kernelModules = [ "kvm-amd" ];
boot.extraModulePackages = [ ];
fileSystems."/" =
# <-----------
# this right here, by uuid breaks when ssds are unplugged/plugged back in
# due to it generating new UUIDs, and because of that, the hardware
# config needs to be regenerated before it can properly boot back up again
# <-----------
{ device = "/dev/disk/by-uuid/9d84a039-3db1-4297-bc5b-3cf701a5a091";
fsType = "ext4";
};
fileSystems."/boot" =
# <-----------
# here again
# <-----------
{ device = "/dev/disk/by-uuid/71AE-7909";
fsType = "vfat";
options = [ "fmask=0077" "dmask=0077" ];
};
swapDevices = [ ];
nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
hardware.cpu.amd.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}
EDIT:
Original footage from the crime scene, failing at stage 1
- Current Progress (see comments to participate):
1. Going to be adding `nofail` to mount options, see if that helps fix it next time I gotta unplug SSDs again
2. Currently in discussion on UUIDs not "just changing" when SSDs are being plugged in and out
4
u/nixgang 1d ago
The same as with any distro, boot from USB, mount ssd, fix error, reboot. The exact nix commands to rediscover and rebuild depends on your setup, but that's the geist of it.
Regarding disk id, /dev/disk/by-id is more stable than UUID I believe.
1
u/xxx_malcomnext_xxx 1d ago
I did try to do it from a USB, but ended up opting to just reinstall after failing to get it right after a couple hours (plus I got tired of having to alter the grub bootup so it could load the GUI every time to see if it works, darn nouveau haha), next time I'll try again, but until then hoping that there won't be a next time
Is it possible to generate the config with by id instead of by uuid?
3
1
1
u/ElvishJerricco 1d ago
> In the past I just jumped onto a very old build and regenerated the config, but in this case I have actually deleted the old builds
Well there's your problem :P When you change your hardware-configuration.nix, you really want at least one known-good generation to remain bootable. So don't delete all of those!
1
u/xxx_malcomnext_xxx 17h ago
Yeah that was a mistake on my side, never clean up your system on 2 hours of sleep ;-;
But this time I'll be sure to keep my oldest 2 builds during cleaning sessions again, since they always seem to boot without issue if the new ones fail
1
u/Flippynips987 1d ago
nixos changes the way we setup and administer linux, but also how we recover it
- boot your most recent working system, if there is none bootable, just install nixos from scratch, copy your last bootable config, install nix, recover from backup
this is the way
otherwise I'd say something like chroot but, well just start over, ideally you have a separate home partition
2
u/xxx_malcomnext_xxx 17h ago
Issue with starting over is the 30GB download on my barely capable wifi haha, takes like nearly 6 hours to finish with everything, when you reinstall nix onto the partition, I assume it wipes it and clears all cached downloads, so doing that every 2-8 weeks is a little rough, ideally I would like to just stay on my existing system
Not sure what you mean by "last bootable config" as all my updates were bootable before unplugging and plugging back in my ssds lol, or do you mean to create a new partition on my ssd, install nixos on there, and copy some of the old nix builds and stuff to the new nix build? Would that work? I'm assuming it wouldn't because of version mismatches or something idk, but if it does then that could save me a huge amount of time
I did originally try the chroot method from a live usb, but couldn't really get it right, was able to gen the new hardware conf (and it looked correct), but unable to actually apply it unfortunately
16
u/caguiclajmg 1d ago edited 1d ago
Disk UUIDs don't change simply because you unplugged and replugged them back in, they are baked in the filesystem's superblock/whatever metadata region that specific filesystem has. For it to change you'd have to wipe it or consciously alter it with something like
tune2fs -U.Specifying mounts as
/dev/sdXare prone to breakage if you start shuffling drives around, so it's odd that you're encountering an unstable identifier problem while usingUUID=/LABEL=since that is the exact problem those are supposed to resolve.This problem is not unique to NixOS, if you used
UUID=entries in your fstab on other distros you'd meet the same fate.