You need monitoring
Kubernetes? New Relic?
It makes me start looking for the next thing. Got my jellyfin, got my pi hole, my retro console and just recently home assistant set up. (Just a few more buts to add to that). Next i think i am going to look into self hosting a cloud storage solution. Like google drive/photos etc. Would be nice to make my own backups and have them offline
I should do some breaking network changes… While tunneled in.
“Yes, while connected to my wireguard server through port 123 here from my Chinese office, I should probably try to upgrade the wireguard server. That’s a great idea!”
Ask me how I know.
I stopped the tailscale service…
… while ssh’d through the tailscale interface.
Luckily, it was my home server and I had to drive there anyway.
I used to make nginx changes while vpn’d into my network and utilizing guacamole (served via said nginx). I’m not a smart man.
Backups. You’re forgetting them.
Pro tip: If you’re using openwrt or other managed network components don’t forget to automatically back those up too. I almost had to reset my openwrt router and having to reconfigure that from scratch sucks.
Let’s tinker around and accidentally break something.
and debug it until you have to reinstall your entire stack from scarch
GET OUT OF MY HOUSE!
Scarched arth
Are you implying it’s possible to debug without having to reinstall from scratch? Preposterous! 😂
When’s the last time you checked if your backup solution works?
What’s a backup solution…? (I’m only being half sarcastic, I really need to set one up, but it’s not as “fun” as the rest of my homelab, open to suggestions)
No mercy for you, then. ;)
I at least have external backups for important family pics and docs! But yea the homelab itself is severely lacking. If it dies, I get to start from scratch. Been gambling for years that “I’ll get around to a backup solution before it dies”. I wouldn’t bet on me :|
But if my backups actually work then I miss out on the joy of rebuilding everything from scratch and explaining to my wife why non of the lights in the house work anymore.
If it’s stable, it’s not a lab.
That’s infrastructure.
Now try migrating all your docker containers to podman.
Just did that last weekend. Nothing to do anymore. 😢
Did you do Quadlets?
Yes of course. Had to spend a couple of hours fixing permission related issues.
I had problems getting apps with multiple containers working in quadlets (definitely a knowledge issue on my part, but didn’t feel the time learning it was beneficial, but will probably revisit during kubernetes learning) so went back to podman with docker compose.
I think it’s kinda better using quadlets, because I wrote some custom scripts, and quadlets made the process better. But podman compose is probably file too.
Have you tried introducing unnecessary complexity?
If you know how your setup works, then that’s a great time for another project that breaks everything.
Saturday morning: “Incus and podman seem interesting. I bet I could swap everything over while the family is out this afternoon”
Sunday evening: “Dad, when will the lights work again?”
Actually, one thing I want to do is switch from services being on a subdomain to services being on a path.
immich.myserver.com -> myserver.com/immich jellyfin.myserver.com -> myserver.com/jellyfinI’m getting tired of having to update DNS records every time I want to add a new service.
I guess the tricky part will be making sure the services support this kind of routing…
Wildcard CNAME pointing to your reverse proxy who then figures out where to route the request to? That’s what I’ve been doing - this way there’s no need to ever update DNS at all :)
I find the path a bit clunky because the apps themselves will oftentimes get confused (especially front-ends). So keeping everything “bare” wrt path, and just on “separate” subdomains is usually my preferred approach.
Why are you having to update your DNS records when you add a new service? Just set up a wildcard A record to send *.myserver.com to the reverse proxy and you never have to touch it again. If your DNS doesn’t let you set wildcard A records, then switch to a better DNS.
Don’t worry, you’re one Docker pull away from having to look up how to manually migrate Postgres databases within running containers!
(Looks at my PaperlessNGX container still down. Still irritated.)
https://github.com/pgautoupgrade/docker-pgautoupgrade
Or if you are on k8s, you can use cloudnativepg.
