Spent a while reading about and understanding how to reliably write files to disk and I'm dropping the best resources I found here so it doesn't all just evaporate. I found I needed to read all this stuff and a bunch of mailing list posts to really get my head round it all because the information is kind of scattered but this is plenty to be getting on with.

Writing data consistently using buffered IO (I didn't really consider direct IO at all):

Totally related but a bit different is stuff around fsync errors. PostgreSQL realised after 20 years that they had been handling this wrong (got named "fsyncgate"). Summary: errors from fsync cannot be relied upon at all and is essentially unspecified behaviour - different filesystems and kernels do different things (page cache may or may not be cleared, marked clean, etc). PostgreSQL eventually went with intentionally crashing and relying on their write-ahead log to recover from these failures: