Backup of live sparse images is ok?

Started by AusTxNerd, May 02, 2010, 09:20:42 PM

Previous topic - Next topic

AusTxNerd

First, I'm totally happy with Espionage. It's exactly what I've been looking for.  Yea!  No more FileVault mess.

My question is about the built-in backup feature of Espionage.  I went through the forum post about correctly excluding things from Time Machine.  That's fine.  And I've setup the built-in Espionage backup to backup all of my encrypted folders to an external hard drive; upon unlocking and every hour while unlocked.

I'm surprised that it can backup the sparsebundle / sparseimage while that disk image is mounted.  I thought that once a disk image was mounted you really couldn't (shouldn't?) mess with the underlying image file.  I guess I'm mistaken since this seems to be how you're backing up the data.  Any caveats? Or does it really produce a clean backup every hour while in use?

Thanks!
  Mike

greg

#1
Hey Mike, that's an excellent question. I'm actually not sure. Operating systems do support the notion of file locking, which prevents files from being written to while something is reading from them, but in UNIX (which is what OS X is basically), this is not always enforced. That is the only scenario where I could see a possibility of corruption happening, other than that it should be fine.

I would guess that the likelyhood of a corrupted backup is slim even when something is editing files inside of that sparseimage, but I cannot give you a definitive answer on that yet. I'll have to get back to you with a definitive answer, but in the meantime if you're concerned about this you can simply disabled that checkbox and simply have Espionage backup the folders only when the folders are locked (or via manually initiated backups).
Follow @espionageapp@twitter.com or @espionage@mstdn.io for news and updates!

greg

#2
Hey Mike, sorry for the wait, we've finished our investigation of this question.

We created a program to simulate intensive file operations (7 processes simultaneously modifying and creating many files) on both sparseimages and sparsebundles while an rsync operation took place in the middle of it ('rsync' is the same program that Espionage uses to perform its backups), and I have mostly good news to share. :-p

Here's an example output that we got from the program:

$ ./test.lsp
Password:
Mounted: SI.sparsebundle
Deleting: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 As
Recreating: Backups/sparsebundle
Creating: /Volumes/SI/As
Creating: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Done!
worker [1-4] started!
worker [5-8] started!
worker [1-4] converted: 1
worker [9-12] started!
worker [5-8] converted: 5
worker [13-16] started!
worker [1-4] converted: 2
worker [17-20] started!
worker [9-12] converted: 9
worker [21-24] started!
Creating 100 A's in: /Volumes/SI/As
rsyncing: SI.sparsebundle -> Backups/sparsebundle
A's created!
worker [5-8] converted: 6
worker [13-16] converted: 13
worker [1-4] converted: 3
rsync done!
Killing children...
done.
Detached: /Volumes/SI
Checking rsync backup...
couldn't mount 'Backups/sparsebundle/SI.sparsebundle' (error 256): hdiutil: attach failed - no mountable file systems
Trying one last time...
Mounted: Backups/sparsebundle/SI.sparsebundle

Checking: /Volumes/SI
1 all B's
2 all B's
3 all A's
4 all A's
5 all B's
6 neither A nor B!
7 all A's
8 all A's
9 all B's
10 all A's
11 all A's
12 all A's
13 neither A nor B!
14 all A's
15 all A's
16 all A's
17 all A's
18 all A's
19 all A's
20 all A's
21 all A's
22 all A's
23 all A's
24 all A's

Checking: /Volumes/SI/As
1 is empty
Detached: /Volumes/SI

So the program first creates 24 1MB files with nothing but the letter 'A' in them. It then spawns off 6 child processes to convert the files, character-by-character to B's. In the middle of this it also creates 100 1MB files inside of a different folder on the encrypted volume (containing only the letter A), and begins the rsync backup of the entire disk image as well.

The results of this experiment showed that for both sparseimages and sparsebundles, the backup would fail to mount on the first try, but on subsequent attempts it would always succeed and Disk Utility reported no problems with the volume. Why it failed to mount on the first try I do not know, perhaps this is a bug in Apple's code somewhere.

Inspection of the files revealed that of the 24 files, some files were completely converted to B's, some remained as A's (these didn't finish the conversion before the backup finished), and some were partially converted, but only A's and B's were in the file (indicating no unexpected corruption).

It was with the creation of files that we noticed some possible corruption (of those files). So when the 100 A's were first created, we noticed that some of the files on the backup did not fall into either of these categories: A's, not copied at all, copied but empty. This was most pronounced with the sparsebundle, but the sparseimage did have a few of these as well. However, subsequent backups corrected these files, and they seemed to stay correct even when they were rewritten on the original disk. I don't think this is much of an issue though, because even though it appears file-level corruption is possible, it appears to be only with newly created files on the first backup, and the alternatives (empty, missing) are equally just as "bad" outcomes, but they are not at all unexpected. If you're making a backup in the middle of new files being created, obviously some files won't get copied unless you do the backup after they've all been created. The overall disk image remained healthy, and the "Backup on Lock" option pretty much makes this a non-issue.

So in summary: at least in our tests it looks like there's not much to be worried over, however you're always welcome (and in fact encouraged) to take extra precautions. :-)

I hope that answers you question!
Follow @espionageapp@twitter.com or @espionage@mstdn.io for news and updates!

AusTxNerd

#3
Greg,

Thank you so much for your research and your answer.  No worries about the wait.  Life is busy for me too!

Everything you've described makes sense.  And I have tried mounting my backup images and everything seems fine for me as well.

So, thank you!  It looks like the backup feature is doing exactly what it should be.

Mike

greg

#4
No problem! Thanks for the question Mike! :-)
Follow @espionageapp@twitter.com or @espionage@mstdn.io for news and updates!