Corrupted innoDB on linux ext4, data recovery

Published Date: 2021/11/26 by: DaVieS

Corrupted innoDB on linux ext4, data recovery
Corrupted innoDB on linux ext4, data recovery

I could say I saw a everthing but not, here is the case:

There is a VM Host with ZFS Storage, direct attached, and there is a Linux VPS with ext4 filesystem, running mySQL Database.

Without sign of error, suddenly the VPS kernel log started spamming:

[23564.244561] blk_update_request: I/O error, dev vda, sector 960227912
[23564.375360] blk_update_request: I/O error, dev vda, sector 960227912

and things went south... The database server has corrupted and fails to start, crash or hung up immediately.
So recovering the mySQL's InnoDB (.ibd) files was very challenging, there was no luck with dd, ddrescue, so wrote a software that could help.

So the mistery,
 - Physical HDDs are OK
 - ZFS with checksuming: No Error
 - qemu-img check: OK
 - fsck on the Guest: OK
 - mounting .qcow2 file on the VM Host, OK bad drive, sector error.

This is the first time that I saw a sector error on a virtual HDD with excellent underlaying storage.
My guess is the ext4 corrupted and the kernel showing a misleading error message after NULL read.

 

The InnoDB recovery procedure was the following:

  1. Attached a new disk with clean and fresh filesystem
  2. Moved all database files that could be accessed and did not throw I/O Error into the new partition
  3. Any other files which are thrown errors and could not been completely restored, rescued by the software I made
    The software reads the file regardless of I/O errors, if a bad block unavailable it can be replaced by any character, in case of innoDB I prefferred 0x00 and 0x0D (some case 0x00 caused errors).
  4. This method shown only 1-2 pages were lost on each broken file, one page is 16K, so the files were restored about 99.9% integrity.
  5. mySQL is a damn sensitive thing, after the restoration its the mySQL database server started and ran well, until reached a damaged page and crashed with buffer overflow.
  6. Here is the another magic thing, mySQL innoDB allows that you can DUMP the whole thing ANYWAY! To perform this action, innodb_recovery_mode = 6 should be set on configuration file.
  7. Dumped the whole database without a single missing row!
  8. Restored from dump, and running flawlessly.

 

This is how you can restore innoDB or any file from corrupted Disk.


That was an insane s@&k however a good thing in a bad story, we got richier by another software for data recovery.

https://github.com/DaVieS007/Partial-File-Recovery

 



If you like the article then don't forget to share!


nPulse.net going dark (again)

Back in time the most of the websites were DARK, specially the tech ones and forums. Then people started to use more slick, curved and light..


sysAdmin ToolBox v2.3.0

We just uploaded the very new release of this application.We added new function called: WiFi Discovery I'm sure everyone will love that, so..


New Software (PFR)

Our new Software released under Open Source license. PFR is a cross-platform easy-to-use powerfull tool to recover broken files that caused HDD..


BVCP for FreeBSD Bhyve Released today!

I'm happily announce that, one of our greatest product just released for the public! BVCP is a Webcontrol interface for FreeBSD Bhyve aka..


Upgrade into FreeBSD Bhyve was successfull

As I told before we were started to migrate our infrastructure from Linux/KVM (FreeBSD) into FreeBSD/Bhyve (FreeBSD). I announce that we..


Hello Bhyve, Im moving in ...

We are performing an update at this weekend, there will be some interrupts in our services. Please be patient! ..


Launched BVCP Today!

FreeBSD Bhyve Web Control Panel launched today as planned as pre-release.Project started at 2021.05 month and yet ready for production use..


bhyve webadmin, web control panel

FreeBSD uses bhyve as hypervisor! So.. no questions we are using FreeBSD for web, mailing, devel, for everything.Now we would like to drop KVM /..


Side Quest, BVCP WebUI for FreeBSD Bhyve

Okay, so we are in middle to upgrade and realign our infrastructure and happened days ago with a random facebook talk, someone hinted FreeBSD..


FreeBSD 13 Just Released

FreeBSD The best operating system for serve WEB and FTP has just announced yesterday, that the newest version is ready for production. One of the..

All rights reserved nPulse.net 2009 - 2021
Powered by: MVCP 2.0-RC / BVCP / ASPF-MILTER / PHP 7.4 / NGINX / FreeBSD