I could say I saw a everthing but not, here is the case:
There is a VM Host with ZFS Storage, direct attached, and there is a Linux VPS with ext4 filesystem, running mySQL Database.
Without sign of error, suddenly the VPS kernel log started spamming:
[23564.244561] blk_update_request: I/O error, dev vda, sector 960227912
[23564.375360] blk_update_request: I/O error, dev vda, sector 960227912
and things went south... The database server has corrupted and fails to start, crash or hung up immediately.
So recovering the mySQL's InnoDB (.ibd) files was very challenging, there was no luck with dd, ddrescue, so wrote a software that could help.
So the mistery,
- Physical HDDs are OK
- ZFS with checksuming: No Error
- qemu-img check: OK
- fsck on the Guest: OK
- mounting .qcow2 file on the VM Host, OK bad drive, sector error.
This is the first time that I saw a sector error on a virtual HDD with excellent underlaying storage.
My guess is the ext4 corrupted and the kernel showing a misleading error message after NULL read.