nice! for anyone with postgres in the mix, what bit me wasn’t the backup itself but the disk. a table in my side project kept one snapshot per minute and never deleted any, the disk hit 100% and postgres went down with a PANIC for like 40 min. now the nightly pg_dump writes to a .tmp, checks it with gzip -t and only then replaces the old file, and the app warns me if the newest dump is older than 26h or suspiciously small. restoring is just loading it into an empty db and starting the app, boring but it’s the part you actually want to know works

one or two services first is the right call, the scope creep is real lol. good luck with the move, and do a full restore of the photos while the box is still at home, that’s the one thing you really don’t want to debug remotely

yeah that matches what I see, most of what we run at work is one app and one database, so the slow query log plus looking at the plan gets you like 90% of the way. I never really had the multi service case where otel pays off, I guess at some point it does, but for a small team it felt like a lot of plumbing to end up finding the same n+1 query lol

I’d add that most of the time nobody actually measured anything. at work we inherit slow sites all the time and the first guess is always “the stack”, then you profile it and it’s one query running per item in a loop or something like that. I had the same with my own side project, was sure I needed a bigger server and after measuring the problem was somewhere else completely. measuring first is boring but it’s cheaper than any rewrite I guess

Plan sounds fine to me, debian + immich in docker + tailscale is pretty much the boring setup and boring is good for a box in a shed. one thing I’d do before carrying it out there, install tailscale and ssh, reboot it a couple of times and make sure it comes back on its own without a screen, because walking to the shed with a monitor gets old fast. also something like uptime kuma on another machine (or your NAS) pinging it, so you know it’s down before you’re on holiday and the photos stop uploading. I run a box at home I only reach over tailscale and that’s the part that saved me most.

good to know about the limit, thanks. for one domain plus DNS-01 that’s all most of us need anyway

yeah that sounds like exactly it, a snap that never worked still sitting there. “snap list” will show it, then sudo snap remove <name> and you’re left with only the docker one, so whatever you open is the right one

spain, so CEST till tomorrow actually, the 13:45 UTC slot is 15:45 for me. 7:45 with people around sounds way nicer than my timer lol

on the fear part, in a healthy team the author still owns the change, the reviewer is a second pair of eyes not an insurance policy. what helped my people a lot was asking the author how they tested it and what happens if it goes wrong, if they can’t answer that’s your review comment right there. and if something breaks after you approved it, you fix it together and add a check so it doesn’t happen again, nobody serious blames the reviewer for that

They don’t go away on their own, every installed snap is mounted as its own loop device and snapd keeps an older revision or two so it can roll back, so the number moves around after updates. snap list --all shows which snaps they are, if one says disabled that’s the old revision and you can remove it with sudo snap remove <name> --revision=<rev>. about libation, if it’s installed as a snap itself that could be the actual problem, snaps are confined and can’t read folders outside your home or on other drives unless you connect the removable-media interface, so I’d check that before worrying about the loop devices.

Good to hear it’s been smooth, the caddy plugin was the main reason I looked at it in the first place.

oh, didn’t know that, thanks for the heads up. deSEC still hosts dns for free if you bring your own domain I think, so a cheap one plus deSEC for the acme dns challenge still works, just not the free subdomain part

that’s actually a nice idea, updating with people around makes it way less of a chore, and someone always knows the weird fix. I’d probably steal it if I had more than one box to look after.

same here on the reboots, I never let it reboot on its own either. for me the reboot wasn’t the problem, it was needrestart bouncing nginx and postgres right after the upgrade, which you only notice if something long lived is running at that moment, like a stream lol

512mb and getting killed every morning at the same time sounds like a fun one to track down lol. a fixed weekly slot like your tuesday evening is probably the sanest setup honestly, I just know I’d forget.

yeah exactly, no reboot, it was needrestart restarting nginx and postgres after the systemd update. you’re right that the timer is kind of the blunt fix, the cleaner one is probably telling needrestart to only list instead of restart ($nrconf{restart} = ‘l’ in needrestart.conf), or excluding nginx from it. I went with the timer because I still want things restarted, just not while I’m live

it’s a single cheap VPS that only I look after, so I’d rather get security patches the same day than remember to log in and do it, I guess that’s the tradeoff. and yeah you’re right about scheduling, that was the whole fix, the default timer runs around 6 UTC plus up to an hour of random delay and that happened to land on top of the stream. moved it to the afternoon and it’s been quiet since

nobody said desec.io yet, it’s a non profit from berlin and they give free subdomains under dedyn.io, with a proper API, so you can do the DNS challenge for let’s encrypt (there’s a caddy plugin for it too). I haven’t run a domain there myself so can’t say how it is long term, but for a “try some things” domain it looks like the least shady of the free ones I guess.

What I do for that is turn it around, instead of checking for a failure I alert on the absence of a success. backrest has hooks, so on snapshot success it pings a push monitor in uptime kuma (healthchecks.io works the same), and if no ping arrives in ~26h it alerts. That way it catches the backup that never ran, which is the one that scares me the most I guess

money wise no, not even close. it’s a bitaxe gamma doing around 1 TH/s solo, so it’s basically a lottery ticket, the odds of hitting a block are tiny. it costs me like 3 euros a month in power so I treat it as a toy, I like checking the best difficulty it found and it’s a nice way to learn how mining works