Things I worked out once and refuse to work out again.
Finding the right MTU the slow way
Spent an evening on a link where small requests were instant and anything larger simply stopped.
Classic symptom and I still did not recognise it. The answer was path MTU: something upstream was
dropping the fragmentation-needed messages, so nothing ever learned to send smaller packets.
Two things fixed it. Allowing destination-unreachable through the firewall instead
of dropping all ICMP, and net.ipv4.tcp_mtu_probing=1 so the kernel works it out when
the messages never arrive. The lesson I keep relearning is that blocking all ICMP is not
hardening, it just moves the failure somewhere harder to see.
journald quietly ate my disk
A small instance filled up and I blamed the application for a while. It was the journal, with no
size cap, keeping everything since install.
SystemMaxUse=200M in a drop-in under journald.conf.d and the problem
has not come back. Worth setting on day one rather than at 94% full.
Backups I actually test
I had backups for years without once restoring from them. When I finally tried, two of three
were unusable, for boring reasons: one had been excluding the directory that mattered, and one had
silently stopped after a key rotation.
Now a restore runs on a schedule into a throwaway directory and shouts at me if the result is
empty. An untested backup is a guess.