Published Date: 2021/11/26 by: DaVieS
I could say I saw a everthing but not, here is the case:
There is a VM Host with ZFS Storage, direct attached, and there is a Linux VPS with ext4 filesystem, running mySQL Database.
Without sign of error, suddenly the VPS kernel log started spamming:
[23564.244561] blk_update_request: I/O error, dev vda, sector 960227912
[23564.375360] blk_update_request: I/O error, dev vda, sector 960227912
and things went south... The database server has corrupted and fails to start, crash or hung up immediately.
So recovering the mySQL's InnoDB (.ibd) files was very challenging, there was no luck with dd, ddrescue, so wrote a software that could help.
So the mistery,
- Physical HDDs are OK
- ZFS with checksuming: No Error
- qemu-img check: OK
- fsck on the Guest: OK
- mounting .qcow2 file on the VM Host, OK bad drive, sector error.
This is the first time that I saw a sector error on a virtual HDD with excellent underlaying storage.
My guess is the ext4 corrupted and the kernel showing a misleading error message after NULL read.
This is how you can restore innoDB or any file from corrupted Disk.
That was an insane s@&k however a good thing in a bad story, we got richier by another software for data recovery.
If you like the article then don't forget to share!
MSSQL Fail to start on newer hardware
I recently faced with a strange issue with Microsoft Database Server.I used Bhyve as hypervisor and a Windows as Guest OS, tried to install MSSQL..
DELL PERC H710 turning into JBOD for ZFS
So at this very begining of the day I picked up an oldie machine Dell T620 and decided to want JBOD for ZFS, better IOPS and easier..
2021 Annual Report
Evolving. Trusted. Doing Better. I proudly can confirm that we still put many efforts to make our infrastucture more stable more green and..
nPulse.net going dark (again)
Back in time the most of the websites were DARK, specially the tech ones and forums. Then people started to use more slick, curved and light..
sysAdmin ToolBox v2.3.0
We just uploaded the very new release of this application.We added new function called: WiFi Discovery I'm sure everyone will love that, so..
New Software (PFR)
Our new Software released under Open Source license. PFR is a cross-platform easy-to-use powerfull tool to recover broken files that caused HDD..
BVCP for FreeBSD Bhyve Released today!
I'm happily announce that, one of our greatest product just released for the public! BVCP is a Webcontrol interface for FreeBSD Bhyve aka..
Upgrade into FreeBSD Bhyve was successfull
As I told before we were started to migrate our infrastructure from Linux/KVM (FreeBSD) into FreeBSD/Bhyve (FreeBSD). I announce that we..
Hello Bhyve, Im moving in ...
We are performing an update at this weekend, there will be some interrupts in our services. Please be patient! ..