==== Simulated data recovery with Testdisk and Photorec ==== === Introduction === Testdisk is a data recovery tool. Its capabilities depend on the filesystem it's operating on. E.g., it can read backup ext3 partitions, or manually specify partition types. It's also good with deleted files or incomplete files. In this column, I'm going to test testdisk and photorec with a simulated corrupt filesystem. === Procedure === - Create a filesystem, - Populate it - Corrupt the filesystem - Attempt to recover the files === Create the filesystem === For testing I'm going to use a loopback filesystem on Linux. This means I can work within files and spare the pain of working with disks. blacktower:~# dd if=/dev/zero of=./baddisk.img bs=1024 count=50000 50000+0 records in 50000+0 records out 51200000 bytes (51 MB) copied, 0.26165 s, 196 MB/s blacktower:~# mkfs -t ext3 ./baddisk.img mke2fs 1.41.12 (17-May-2010) ./baddisk.img is not a block special device. Proceed anyway? (y,n) y Filesystem label= OS type: Linux Block size=1024 (log=0) Fragment size=1024 (log=0) Stride=0 blocks, Stripe width=0 blocks 12544 inodes, 50000 blocks 2500 blocks (5.00%) reserved for the super user First data block=1 Maximum filesystem blocks=51380224 7 block groups 8192 blocks per group, 8192 fragments per group 1792 inodes per group Superblock backups stored on blocks: 8193, 24577, 40961 Writing inode tables: done Creating journal (4096 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 24 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. blacktower:~# mkdir /mnt/baddisk; mount -o loop ./baddisk.img /mnt/baddisk Now to create some test files. i=0; while [ $i -lt 1000 ]; do /usr/games/fortune >> /mnt/baddisk/fortunes.txt; i=$((i+1)); done blacktower:~# cp /var/pub/testfiles/* /mnt/baddisk/ blacktower:~# ls -l /mnt/baddisk/ total 1216 -rw-r--r-- 1 root root 114616 Apr 6 11:03 fortunes.txt -rwxr--r-- 1 root root 17920 Apr 6 11:24 libreoffice writer.doc -rwxr--r-- 1 root root 24391 Apr 6 11:24 libreoffice writer.odt -rwxr--r-- 1 root root 42101 Apr 6 11:24 libreoffice writer.pdf -rwxr--r-- 1 root root 18541 Apr 6 11:24 libreoffice writer.rtf drwx------ 2 root root 12288 Apr 6 10:55 lost+found -rwxr--r-- 1 root root 962190 Apr 6 11:24 testimage.bmp -rwxr--r-- 1 root root 17444 Apr 6 11:24 testimage.jpg -rwxr--r-- 1 root root 18523 Apr 6 11:24 testimage.png This test is very synthetic because the filesystem has never had a delete. There's no fragmentation and all the files are sitting in an optimal location. === Making the filesystem more realistic === Let's fragment! (warning, dangerous script) #!/bin/bash # pick a file, copy it, delete the original targetdir=/mnt/baddisk mkdir "$targetdir"/tmp i=0 while [ $i -lt 10000 ] do # echo $i set -- "$targetdir"/* shift $(($RANDOM%$#)) if [[ -f $1 ]] then cp "$1" $targetdir/tmp/ rm "$1" mv "$targetdir/tmp/`basename \"$1\"`" "$targetdir" fi i=$(($i+1)) done MD5 hashes permit us to check the file integrity later. They're stored separately from the filesystem. blacktower:~# md5sum /mnt/baddisk/* 0b5080a051d53ba2432b666c90d7c0b4 /mnt/baddisk/fortunes.txt 8f301122918ab70d8cdccebdac46c8c5 /mnt/baddisk/libreoffice writer.doc 45f1fbff8dcd6a92eef33e5639837c81 /mnt/baddisk/libreoffice writer.odt 6dbeeb8ff8b82d35296dfc5265897f92 /mnt/baddisk/libreoffice writer.pdf 9d4cb7f894983e667a7c2173f9fa59c4 /mnt/baddisk/libreoffice writer.rtf md5sum: /mnt/baddisk/lost+found: Is a directory ed0eda5ba4f73f8d41eade656db27517 /mnt/baddisk/testimage.bmp 3c90cdc73c1b155b7cdd5e0a13584207 /mnt/baddisk/testimage.jpg 8b40c922b02ef43be7c986fa59536260 /mnt/baddisk/testimage.png md5sum: /mnt/baddisk/tmp: Is a directory === Corrupting the filesystem === First we'll unmount the filesystem and Check to see that it's okay. blacktower:~# umount /mnt/baddisk blacktower:~# fsck.ext3 ./baddisk.img e2fsck 1.41.12 (17-May-2010) ./baddisk.img: clean, 20/12544 files, 7703/50000 blocks To corrupt the disk, I'm throwing random nulls throughout the image with the code below. blacktower:~/testdisk# cat ./corrupt.bash #!/bin/bash i=0 while [ $i -lt 100 ] do target=$(( RANDOM % 5120 ))$(( RANDOM % 10000 )) echo $target #dd if=/dev/zero of=./baddisk.img bs=1 count=1 seek=$target 2> /dev/null ./corrupt ./baddisk.img $target i=$((i+1)) done blacktower:~/testdisk# cat ./corrupt.c #include /* program to write to specific location of a file */ int main(int argc, char* argv[]) { FILE* fh; if( argc != 3 ) { printf( "\nUsage: %s filename\n\n Where \"filename\" is the file to corrupt and \"location\" is the offset in bytes\n", argv[0] ); return(1); } fh = fopen( argv[1], "r+" ); if( fh == NULL ) { printf( "\nCould not open existing file for writing!\nUsage: %s filename\n\n Where \"filename\" is the file to corrupt and \"location\" is the offset in bytes\n", argv[0] ); return(1); } fseek( fh, atoi(argv[2])-2, SEEK_SET ); fputc( 0, fh ); fclose( fh ); return 0; } blacktower:~/testdisk# ./corrupt.bash blacktower:~/testdisk# mount -o loop ./baddisk.img /mnt/baddisk/ blacktower:~/testdisk# md5sum /mnt/baddisk/* 2> /dev/null 0b5080a051d53ba2432b666c90d7c0b4 /mnt/baddisk/fortunes.txt 8f301122918ab70d8cdccebdac46c8c5 /mnt/baddisk/libreoffice writer.doc 45f1fbff8dcd6a92eef33e5639837c81 /mnt/baddisk/libreoffice writer.odt 6dbeeb8ff8b82d35296dfc5265897f92 /mnt/baddisk/libreoffice writer.pdf 9d4cb7f894983e667a7c2173f9fa59c4 /mnt/baddisk/libreoffice writer.rtf 24896c06121360c4b3daa1deca7e854f /mnt/baddisk/testimage.bmp 3c90cdc73c1b155b7cdd5e0a13584207 /mnt/baddisk/testimage.jpg 8b40c922b02ef43be7c986fa59536260 /mnt/baddisk/testimage.png blacktower:~/testdisk# Looks like we hit mostly empty space. Only the bmp is corrupted. Change the 100 random bytes to 10000 and run again: blacktower:~/testdisk# fsck.ext3 ./baddisk.img e2fsck 1.41.12 (17-May-2010) ./baddisk.img: clean, 20/12544 files, 7703/50000 blocks blacktower:~/testdisk# md5sum /mnt/baddisk/* 2> /dev/null e04f2a97d51432d04a64f642e350b1f0 /mnt/baddisk/fortunes.txt 8f301122918ab70d8cdccebdac46c8c5 /mnt/baddisk/libreoffice writer.doc fc2475f7a1a70f2f31ae50ca59b96cb7 /mnt/baddisk/libreoffice writer.odt 6037209ee8c78156a1e6acea00b09dd1 /mnt/baddisk/libreoffice writer.pdf 16a93d89a8dedfd23ab629006104cbca /mnt/baddisk/libreoffice writer.rtf aaf2de72c3f2966bf414d598435dc432 /mnt/baddisk/testimage.bmp 3c8a111a8c9baaa09aea73e7d59e890e /mnt/baddisk/testimage.jpg e3912ad720969f46308cdccad73a66f8 /mnt/baddisk/testimage.png Only the .doc file is safe now. Again. blacktower:~/testdisk# fsck.ext3 ./baddisk.img e2fsck 1.41.12 (17-May-2010) ./baddisk.img: clean, 20/12544 files, 7703/50000 blocks blacktower:~/testdisk# fortunes.txt is visibly corrupted, and every file fails an MD5, but the filesystem hasn't reported any problems yet. {{:article-testdisk:corruption.png?nolink&300 |}} Different method: blacktower:~/testdisk# tail -n +10 ./baddisk-2.img > baddisk-2b.img blacktower:~/testdisk# ls -l total 299852 -rw-r--r-- 1 root root 50674559 Apr 10 18:10 baddisk-2b.img -rw-r--r-- 1 root root 51200000 Apr 6 21:45 baddisk-2.img blacktower:~/testdisk# mount -o loop ./baddisk-2b.img /mnt/baddisk/ mount: you must specify the filesystem type blacktower:~/testdisk# mount -t ext3 -o loop ./baddisk-2b.img /mnt/baddisk/ mount: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so blacktower:~/testdisk# fsck.ext3 ./baddisk-2b.img e2fsck 1.41.12 (17-May-2010) fsck.ext3: Superblock invalid, trying backup blocks... fsck.ext3: Bad magic number in super-block while trying to open ./baddisk-2b.img The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 blacktower:~/testdisk# fsck.ext3 -b 8193 ./baddisk-2b.img e2fsck 1.41.12 (17-May-2010) fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to open ./baddisk-2b.img Could this be a zero-length partition? blacktower:~/testdisk# ls -l ./baddisk-2b.img -rw-r--r-- 1 root root 50674559 Apr 10 18:10 ./baddisk-2b.img blacktower:~/testdisk# The partition table is missing and the offsets for the backups are all messed up. === Testdisk === non-partitioned === photorec == PhotoRec 6.11, Data Recovery Utility, April 2009 Christophe GRENIER http://www.cgsecurity.org Disk ./baddisk-2b.img - 50 MB / 48 MiB (RO) Partition Start End Size in sectors P Unknown 0 0 1 6 41 1 98974 14 files saved in /root/testdisk/recup_dir directory. Recovery completed. txt: 14 recovered blacktower:~/testdisk/recup_dir.3# head -1 *.txt ==> f0000000.txt <== Just to have it is enough. ==> f0000128.txt <== men proud. ==> f0003072.txt <== ==> f0004352.txt <== it classified? ==> f0004480.txt <== rail2\hyphmax0}\aspalpha\ltrpar\langfe255\lang255\cf0\kerning1\hich\af3\dbch\af4\afs24\lang255\loch\f0\fs24{\rtlch \ltrch\loch ==> f0006400.txt <== wiss\fprq2\fcharset0 Arial;}{\f3\fnil\fprq2\fcharset0 Andale Sans UI{\*\falt Arial Unicode MS};}{\f4\fnil\fprq2\fcharset0 Tahoma;}{\f5\fnil\fprq0\fcharset0 Tahoma;}} ==> f0007552.txt <== us. ==> f0014848.txt <== ker is a fellow who lends you his umbrella when the sun is shining ==> f0017024.txt <== ought, and the wisdom never to use either. ==> f0017280.txt <== owntown Newark is in your future. ==> f0017664.txt <== outh gets trashed. ==> f0078080.txt <== e ==> f0095488.txt <== amenco dancer's ==> f0095616.txt <== up to his neck in sand? blacktower:~/testdisk/recup_dir.3# === Try harder === Paranoid : Yes (Brute force enabled) 14 files === Expert Mode === 79 files saved in /root/testdisk/recup_dir directory. Recovery completed. txt: 79 recovered Unless we find the superblock, it looks like the fragmentation of the files is going to make it nearly impossible to find any files. We could create a signature to find the superblock... hmmm... http://www.virtualblueness.net/Ext2fs-overview/Ext2fs-overview-0.1-12.html === Additional Information === http://www.cgsecurity.org/wiki/TestDisk http://www.cgsecurity.org/wiki/PhotoRec