Why does Calibre Library take up so much space? 7000 books equals 50GB when exported

[droopy]

Originally Posted by haertig

Your awk script is bad. Actually, the awk commands are OK, but where you are failing is in trying to make things "neat" using numfmt. Delete that junk that attempts to beautify the output, just display the raw numbers, and your script works much better:

find -type f -name \*\.\* -printf '%f\0%s\n' | gawk '
  BEGIN {
    FS = "\0";
  }
  {
    split($1, a, ".");
    ext = tolower(a[length(a)]);
    files[ext] += 1;
    size[ext] += $2;
  }
  END {
    PROCINFO["sorted_in"] = "@ind_str_asc";
    for (ext in files) {
      print "*." ext " " files[ext] " files, " size[ext] " bytes";
    }
  }'

thanks haertig for correcting the code. It was originally created by someone else. Unfortunately, because I don't code, I didn't know how to read the code.
I now ran your corrected code and got this result:

*.azw 6 files, 34806823 bytes
*.azw3 178 files, 2569168678 bytes
*.azw4 3 files, 238715160 bytes
*.cbz 6 files, 1766121940 bytes
*.chm 1 files, 904553 bytes
*.db 1 files, 24219648 bytes
*.djvu 17 files, 70510310 bytes
*.epub 6839 files, 39640384895 bytes
*.htmlz 113 files, 491346536 bytes
*.jpg 7812 files, 2121182039 bytes
*.json 1 files, 30310 bytes
*.kepub 3 files, 2326374 bytes
*.kfx 1 files, 2052924 bytes
*.kfx-zip 7 files, 27629860 bytes
*.mbp 12 files, 1440 bytes
*.mobi 214 files, 365347885 bytes
*.odt 1 files, 31448 bytes
*.opf 3431 files, 40932854 bytes
*.original_epub 193 files, 1543362814 bytes
*.pdf 768 files, 7765519460 bytes
*.txt 1 files, 381732 bytes
*.zip 2 files, 1282446 bytes

I've attached a screenshot of filetypes in the Calibre program.

Update: The numbers (of files per type) is a match!

[haertig]

39Gb of epub, 7.7Gb of pdf, 2.5Gb of azw3, 2.1Gb of jpg, 1.5Gb of original_epub, ... plus more smaller numbers.

I can see where your files easily add up to the 50Gb you initially reported.

[droopy]

It has been good to confirm what was taking up so much space and to learn that there weren't some non-library files mistakenly added/pasted to the calibrelibrary directory. So each epub is about 5.7 MB. I guess that's understandable.

[haertig]

My epubs average 375Kb each. But they are all novels, so not many pictures (other than embedded cover art) and maybe a few simple diagrams or maps as you sometimes find in novels.

[droopy]

Attached is a screenshot of the biggest ePubs in my Calibre. All the titles in screenshot are non-fiction.

The smallest epubs are 100KB or less!

[haertig]

I looked at the epubs in my "Reference" library, I keep that separate from my"Novels" library in Calibre. My reference epubs average 1.6Mb each. Quite a bit larger than my average novel size. Those reference books are things like investment books, hobby books, programming books, cookbooks, travel books, survival books, etc. They contain a lot more pictures and diagrams.

[DNSB]

Originally Posted by droopy

thanks haertig for correcting the code. It was originally created by someone else. Unfortunately, because I don't code, I didn't know how to read the code.
I now ran your corrected code and got this result:

I've attached a screenshot of filetypes in the Calibre program.

Update: The numbers (of files per type) is a match!

Running an old add the numbers kludge (copy/paste a set of columns and it totals the numbers in the columns), divided the result by 1024^3 and out popped 51.22GiB. We have a winner!