Best way to compress/verify/checksum for Data storage.

knowhate · Nov 25, 2009

Hello everyone,

I am planning on doing a vast overhaul-backup of all my precious family videos and pics for long term storage- and the best thing I decided for long term storage is to rar/zip (some kind of compression) all this hours of data.

My question is how to do this in the best possible way while also having a system to check and verify that all bits of data remain the same throughout bit by bit (I believe this process is called checksuming)..

I just want to have a full proof way of ensuring my personal files stay the same if I choose to store them in multiple copies on DVD or Harddrive.

I'm looking for all solutions from intermediate to professional. I am willing to learn professional software; I just need the job to be done right.

I know there is a way to just zip files in OSX, and then doing checksums by means of terminal (I'm not quite sure how the process works); but would this be the simplest, best way for hours of data?

Thank you again.

ticedoff8 · Nov 25, 2009

knowhate said:
Hello everyone,

I am planning on doing a vast overhaul-backup of all my precious family videos and pics for long term storage- and the best thing I decided for long term storage is to rar/zip (some kind of compression) all this hours of data.

My question is how to do this in the best possible way while also having a system to check and verify that all bits of data remain the same throughout bit by bit (I believe this process is called checksuming)..

I just want to have a full proof way of ensuring my personal files stay the same if I choose to store them in multiple copies on DVD or Harddrive.

I'm looking for all solutions from intermediate to professional. I am willing to learn professional software; I just need the job to be done right.

I know there is a way to just zip files in OSX, and then doing checksums by means of terminal (I'm not quite sure how the process works); but would this be the simplest, best way for hours of data?

Thank you again.

The problem is long term storage and transfer. Once the file becomes corrupt (say, during a copy to a new HD), you are screwed.

These are pictures - Right? If they are the typical jpg or gif file, there really is no point trying to compress them - tif, jpg & gif are already compressed.

Just use "tar".

The "tar" program will maintain the folder structure, file names, permissions and ownerships. Plus - "tar" is kind of a universal format - you can untar on 99% of all OS out there (PC, Mac, UNIX, Linux) - and that won't change anytime soon.

I can still untar my archive files that I stored from my old SunOS 4 systems 15 years ago.

Or...
Atempo Time Navigator for Mac OS.
Atempo - Data Management, Simplified.
I used to be an SE for them - it's good SW - but expensive - $4,500 for the base license. But, it is "industrial strength backup / archive" for Mac, Windows Linux & Unix systems.

Backup / Archive to disk, tape or other forms of storage devices.

Apple Computer uses it in Cupertino in their iWorks, Pro Apps and OS Archive groups.

They even have an option to store the backup files in a "tar" format - so you don't need Time Navigator to recover the files - just plain vanilla "tar".

knowhate · Nov 25, 2009

ticedoff8 said:
The problem is long term storage and transfer. Once the file becomes corrupt (say, during a copy to a new HD), you are screwed.

These are pictures - Right? If they are the typical jpg or gif file, there really is no point trying to compress them - tif, jpg & gif are already compressed.

Just use "tar".

The "tar" program will maintain the folder structure, file names, permissions and ownerships. Plus - "tar" is kind of a universal format - you can untar on 99% of all OS out there (PC, Mac, UNIX, Linux) - and that won't change anytime soon.

I can still untar my archive files that I stored from my old SunOS 4 systems 15 years ago.

Or...
Atempo Time Navigator for Mac OS.
Atempo - Data Management, Simplified.
I used to be an SE for them - it's good SW - but expensive - $4,500 for the base license. But, it is "industrial strength backup / archive" for Mac, Windows Linux & Unix systems.

Backup / Archive to disk, tape or other forms of storage devices.

Apple Computer uses it in Cupertino in their iWorks, Pro Apps and OS Archive groups.

They even have an option to store the backup files in a "tar" format - so you don't need Time Navigator to recover the files - just plain vanilla "tar".

hello!

thank you for the feedback - much appreciated!

Acutally I do have .tif scans of the familiy photos, but it is mostly video that (old HI8, miniDV, digital) that has accumulated. At the moment I just want to find a way to compress and store - or maybe even just forget the compression part if that's possible. I'm just worried about redundancy and verifying the data is "pure" when I try to re-access it in the future.

Your .tar solution seems like a good bet. I dont know about buying a 4,000 dollar license though, though I do like that I know it's a possibility for the future - so thanks for the mention.

I don't know I guess the I can just try some methods and see what is best in time/ efficiency and gestation - as i have a LOT of video.

Would anyone else like to chime in?

Lifeisabeach · Nov 25, 2009

knowhate said:
hello!

thank you for the feedback - much appreciated!

Acutally I do have .tif scans of the familiy photos, but it is mostly video that (old HI8, miniDV, digital) that has accumulated. At the moment I just want to find a way to compress and store - or maybe even just forget the compression part if that's possible. I'm just worried about redundancy and verifying the data is "pure" when I try to re-access it in the future.

Your .tar solution seems like a good bet. I dont know about buying a 4,000 dollar license though, though I do like that I know it's a possibility for the future - so thanks for the mention.

I don't know I guess the I can just try some methods and see what is best in time/ efficiency and gestation - as i have a LOT of video.

Would anyone else like to chime in?

You could make a .par set of files. This works much like a RAID 4 array in which the data is spread out over multiple files, but there is some redundancy and built-in error checking so that, if one .par file gets corrupted, you can rebuild it using the other .par files from the set. This takes up more storage space than a straightforward archive set, but the key value to this is the redundancy and ability to recover from a missing or corrupted part. What you could do is make a .tar archive as suggested, then make a .par set of that .tar archive.

ticedoff8 · Nov 25, 2009

lifeisabeach said:
You could make a .par set of files. This works much like a RAID 4 array in which the data is spread out over multiple files, but there is some redundancy and built-in error checking so that, if one .par file gets corrupted, you can rebuild it using the other .par files from the set. This takes up more storage space than a straightforward archive set, but the key value to this is the redundancy and ability to recover from a missing or corrupted part. What you could do is make a .tar archive as suggested, then make a .par set of that .tar archive.

I found all kinds of ways to decode par files - how do you encode one?

Lifeisabeach · Nov 26, 2009

ticedoff8 said:
I found all kinds of ways to decode par files - how do you encode one?

With MacPAR Deluxe. Drag/drop a set of archive files into the MacPAR window, click Save.

Ideally you want to make a multi-part archive file first. RAR is very commonly used with PAR, but I believe you can do the same thing with TAR files. From what I understand, if you only add one archive file, then you lose the redundancy feature of the PAR format. You can't add anything with resource forks (like pictures), so you have to add archive files. So using multi-part RAR (or perhaps even TAR) with PAR is ideal.

I just found an app called SimplyRAR that makes RAR files and then can make a PAR set. It apparently is still lacking in some features.
Download SimplyRAR for Mac - GUI for RAR compression. MacUpdate Mac Compression Software Downloads

Lifeisabeach · Nov 26, 2009

After giving it some thought, doing a par/tar combo might be a bit overkill. PAR is used often for usenet, where there is a high chance of losing parts of an upload to a binary newsgroup. Quite frankly, for a personal archive, it may not be really necessary in light of the alternatives. Here's the wikipedia page that elaborates on the features of all the various archive types:
Comparison of archive formats - Wikipedia, the free encyclopedia

So it seems to me that your first priority would be an archive format that does error checking and can recover from such errors. An archive format that minimizes the amount of disc space required would be a bonus.

The key problem with the .tar format is lack of a recovery record. You'll have to read in more detail the ups and downs of the various archive formats, but it seems to me that RAR and ACE would be your two top choices. Or you could stick with .tar for the reasons ticed gave and add it to a .par file set for the error correction features it offers.