Members
  • Total Members: 12816
  • Latest: t114563
Stats
  • Total Posts: 28525
  • Total Topics: 8240
  • Online Today: 825
  • Online Ever: 51419
  • (01. January 2010., 10:27:49)












Author Topic: Getting the best compression with 7zip when compressing the bitcoin blockchain  (Read 1510 times)

0 Members and 1 Guest are viewing this topic.

devnullius

  • SCF VIP Member
  • *****
  • Posts: 3507
  • KARMA: 152
  • Gender: Female
    • SCForum.info
Ok, time to compress a backup of my bitcoin blockchain. It's a beast!

So I thought it was finally time to do a little light research to what options give the best compression results (lzma lzma2 ppmd bzip2)...

I first went here: http://sourceforge.net/p/sevenzip/discussion/45797/thread/9dd9e53c
And here: http://superuser.com/questions/281573/what-are-the-best-options-to-use-when-compressing-files-using-7-zip
And here: http://sourceforge.net/p/sevenzip/discussion/45798/thread/513dbd7d
And here: http://superuser.com/questions/432025/different-compression-methods-in-7zip-which-is-best-suited-for-what-task

Then, I followed the link to:  http://www.scribd.com/doc/95731105/Compression
(warning: first graph with green bars says "size"; should read: "time")

Quote
The graph below shows the best compression ratio, it is vertically adjusted with best time at the top:You can see from the results there is a slight improvement in compression speed between LZMAand LZMA2 when using a single thread. There is not much difference between them when usingtwo threads. However there is a consistant difference in archive size results with LZMA2 benifitingaround an extra 3MB. Maybe not much but better than nothing and does prove there has been animprovement with the compression engine.The big difference in LZMA2 is when taking advantage of the extra CPU threads. Whilst the archive size increased by around an extra 25MB when using 4 or more threads the compression speeds increased greatly.

Conclusion: if time is of no importance, use 1 thread for best compression, with LZMA2. Be warned though... My estimates went from < 90 minutes of compressing to > 16 hours of compressing (W2K12R2, 32 core, 128 GB RAM, swap file will be disabled to a fixed size of 800MB, for now it's still set on 20GB, 5 GB of free drive space on C: to grow).

I'll do both to double check and I'll let you know the results in a post-edit tomorrow :) For *my* first impression is that 32 threads actually DOES give smaller file... We'll see  :bih:

Devvie

POST-EDIT 1:
First single threaded compress round has finished. Took a good 20 hours... Max total RAM used was about 11GB. Needed RAM to decompress is 1 GB. Total size: 22.052.337 kB.
POST-EDIT 2:
32 threaded compress round has been aborted. Needed RAM to compress, 180GB. Needed RAM to decompress, still 1GB.
POST-EDIT 3:
Then, I went ahead with a 20 and 21 threaded compress round. Needed RAM to compress, 112GB. My current free RAM is 86 GB (let's see what Windows and the 20 GB swapfile can manage here! Only 5GB free space left to grow the swap, so 1GB short...).
Compressing was aborted prematurely: could not allocate enough RAM. Strange, not even half of available RAM memory was in use... I guess 7zip predicts the future too ;)
POST-EDIT 4:
So, let's now try a 19 threaded compress round. Needed RAM to compress, 101,5 GB. My current free RAM is 86 GB (let's see what Windows and the 20 GB swap file can manage here! Only 5GB free space left to grow the swap, but with 4,5GB memory to spare... This should work!)
RESULT 4: swap file grew with another 5 GB to 25GB. My Max RAM usage for 7zip process was 100,4 GB. Time to compress, 83 minutes. Resulting file size: 22,492,750 kB - about 440 MB bigger when multiple threads are used. I can't explain it, but there it is :)

Decompressed size: 38,898,238,901 bytes (div 1024 = 37,986,561.4 kB)


Karma!

Devvie
More information about bitcoin, altcoin & crypto in general? GO TO  j.gs/7385484/btc

Cuisvis hominis est errare, nullius nisi insipientis in errore persevare... So why not get the real SCForum employees to help YOUR troubled computer!!! SCF Remote PC Assist http://goo.gl/n1ONa9

Samker's Computer Forum - SCforum.info





Samker

  • SCF Administrator
  • *****
  • Posts: 7206
  • KARMA: 291
  • Gender: Male
  • Whatever doesn't kill us makes us stronger.
    • SCforum.info - Samker's Computer Forum

"7zip" :thumbsup: is not on your list on 7th place without reason - "Devnullius's Choices: a list of default programs to keep your PC running well": http://scforum.info/index.php/topic,8202.0.html


devnullius

  • SCF VIP Member
  • *****
  • Posts: 3507
  • KARMA: 152
  • Gender: Female
    • SCForum.info
More information about bitcoin, altcoin & crypto in general? GO TO  j.gs/7385484/btc

Cuisvis hominis est errare, nullius nisi insipientis in errore persevare... So why not get the real SCForum employees to help YOUR troubled computer!!! SCF Remote PC Assist http://goo.gl/n1ONa9

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
Second Anti-Bot trap, type or simply copy-paste below (only the red letters):www.scforum.info:

Enter your email address to receive daily email with 'SCforum.info - Samker's Computer Forum' newest content:

Terms of Use | Privacy Policy | Advertising