Topic: Getting the best compression with 7zip when compressing the bitcoin blockchain

[devnullius]

Ok, time to compress a backup of my bitcoin blockchain. It's a beast!

So I thought it was finally time to do a little light research to what options give the best compression results (lzma lzma2 ppmd bzip2)...

I first went here: http://sourceforge.net/p/sevenzip/discussion/45797/thread/9dd9e53c
And here: http://superuser.com/questions/281573/what-are-the-best-options-to-use-when-compressing-files-using-7-zip
And here: http://sourceforge.net/p/sevenzip/discussion/45798/thread/513dbd7d
And here: http://superuser.com/questions/432025/different-compression-methods-in-7zip-which-is-best-suited-for-what-task

Then, I followed the link to:  http://www.scribd.com/doc/95731105/Compression
(warning: first graph with green bars says "size"; should read: "time")

The graph below shows the best compression ratio, it is vertically adjusted with best time at the top:You can see from the results there is a slight improvement in compression speed between LZMAand LZMA2 when using a single thread. There is not much difference between them when usingtwo threads. However there is a consistant difference in archive size results with LZMA2 benifitingaround an extra 3MB. Maybe not much but better than nothing and does prove there has been animprovement with the compression engine.The big difference in LZMA2 is when taking advantage of the extra CPU threads. Whilst the archive size increased by around an extra 25MB when using 4 or more threads the compression speeds increased greatly.

Conclusion: if time is of no importance, use 1 thread for best compression, with LZMA2. Be warned though... My estimates went from < 90 minutes of compressing to > 16 hours of compressing (W2K12R2, 32 core, 128 GB RAM, swap file will be disabled to a fixed size of 800MB, for now it's still set on 20GB, 5 GB of free drive space on C: to grow).

I'll do both to double check and I'll let you know the results in a post-edit tomorrow For *my* first impression is that 32 threads actually DOES give smaller file... We'll see 

Devvie

POST-EDIT 1:
First single threaded compress round has finished. Took a good 20 hours... Max total RAM used was about 11GB. Needed RAM to decompress is 1 GB. Total size: 22.052.337 kB.
POST-EDIT 2:
32 threaded compress round has been aborted. Needed RAM to compress, 180GB. Needed RAM to decompress, still 1GB.
POST-EDIT 3:
Then, I went ahead with a 20 and 21 threaded compress round. Needed RAM to compress, 112GB. My current free RAM is 86 GB (let's see what Windows and the 20 GB swapfile can manage here! Only 5GB free space left to grow the swap, so 1GB short...).
Compressing was aborted prematurely: could not allocate enough RAM. Strange, not even half of available RAM memory was in use... I guess 7zip predicts the future too
POST-EDIT 4:
So, let's now try a 19 threaded compress round. Needed RAM to compress, 101,5 GB. My current free RAM is 86 GB (let's see what Windows and the 20 GB swap file can manage here! Only 5GB free space left to grow the swap, but with 4,5GB memory to spare... This should work!)
RESULT 4: swap file grew with another 5 GB to 25GB. My Max RAM usage for 7zip process was 100,4 GB. Time to compress, 83 minutes. Resulting file size: 22,492,750 kB - about 440 MB bigger when multiple threads are used. I can't explain it, but there it is

Decompressed size: 38,898,238,901 bytes (div 1024 = 37,986,561.4 kB)


Karma!

Devvie

[Samker]

"7zip" is not on your list on 7th place without reason - "Devnullius's Choices: a list of default programs to keep your PC running well": http://scforum.info/index.php/topic,8202.0.html

[devnullius]

 

: )