Why does Calibre Library take up so much space? 7000 books equals 50GB when exported

[JSWolf]

Originally Posted by theducks

I suspect some form of Annotation archive

I've never had a need to download annotations as most of what I annotate are errors that I later correct.

[DNSB]

Originally Posted by theducks

You could recover 1.5G by off loading Original_EPUB. (click the format in the tag browser: select all:rightclick Delete: remove specific format:<select>

IMHO these can usually go after you have created you FINAL clean EPUB (or AZW3)
as always YMMV

Your EPUB must have lots of (big) embedded graphics or embedded fonts to hit 36G with 7K books

This seems odd. note the file count

*.json 7,748 files, 29.4KiB   <I only have ONE
*.kepub 7,748 files, 2.3MiB
*.kfx 7,748 files, 2.0MiB

Even more peculiar is that while json files may be small, I have trouble visualizing a library with 7748 kepub and 7748 KFX ebooks totalling 4.3Mib. That's what? ~290 bytes per file.

[BetterRed]

There should be an metadata.opf for every book (I assume that's 7748), as there are 7748 cover.jpg files, but there are only 3,352 .opf files. You could fix that via the calibredb backup_metadata --all command.

Apart from the .jpg and .opf counts, do the file counts from Linux match with the counts in the Tag Browser Format counts, e.g. EPUB 7,775 files, HTMLZ 113, KEPUB 7,748 MBP 12 etc. If not you need to get that sorted out:

1. Either by running the calibredb check_library --csv command and surgically culling the 'dross' formats and adding the 'lost' formats
2. Or by brute force by moving what the library database knows about into an empty library and dealing with what's left.

BR

[droopy]

Originally Posted by theducks

I suspect some form of Annotation archive

Indeed, I do use the Annotations plugin.

[droopy]

Originally Posted by DNSB

Even more peculiar is that while json files may be small, I have trouble visualizing a library with 7748 kepub and 7748 KFX ebooks totalling 4.3Mib. That's what? ~290 bytes per file.

I don't get the KFX number. I use a Kobo Forma and usually avoid Kindle formats like KFX, if I could choose epub.

As for kepub, when I "Send to Device", the format that is saved on the fly onto my Kobo is a kepub, whereas the originals (i.e., in Calibre) are 99.99% epub.

[droopy]

About CBZ
The terminal command says

*.cbz 3 files, 1.7GiB

But Calibre shows me 6 CBZ files which, interestingly, also adds up to 1.7 GB.

Terminal says
*.kepub 7,812 files, 2.3MiB
But Calibre says 3 Kepubs that add up, coincidentally to a very-close 2.2MB.

Terminal says
*.kfx 7,812 files, 2.0MiB
But Calibre says 1 KFX at 2.0MB

KFX-zip number in terminal and Calibre are both 7 files.
Number of MBP files in terminal and Calibre are the same.
Mobi number matches up.

Calibre says 214 ODT files but Calibre says just one ODT file.

Original_epub is a match.
PDF is a match.
ZIP is a match.

[droopy]

Originally Posted by BetterRed

There should be an metadata.opf for every book (I assume that's 7748), as there are 7748 cover.jpg files, but there are only 3,352 .opf files. You could fix that via the calibredb backup_metadata --all command.

Apart from the .jpg and .opf counts, do the file counts from Linux match with the counts in the Tag Browser Format counts, e.g. EPUB 7,775 files, HTMLZ 113, KEPUB 7,748 MBP 12 etc. If not you need to get that sorted out:

azw, azw3, azw4 are all identical in Calibre and in terminal.

The terminal command says
*.cbz 3 files, 1.7GiB
But Calibre shows me 6 CBZ files which, interestingly, also adds up to 1.7 GB.

Terminal says
*.kepub 7,812 files, 2.3MiB
But Calibre says 3 Kepubs that add up, coincidentally to a very-close 2.2MB.

Terminal says
*.kfx 7,812 files, 2.0MiB
But Calibre says 1 KFX at 2.0MB

KFX-zip number in terminal and Calibre are both 7 files.
Number of MBP files in terminal and Calibre are the same.
Mobi number matches up.

Calibre says 214 ODT files but Calibre says just one ODT file.

Original_epub is a match.
PDF is a match.
ZIP is a match.

[PeterT]

Have you tried doing both portions of the calibre library maintenance? One ensures that all content referenced by the database is present in the file system; the other verifies that all files in the file system are present in the database.

[jhowell]

Originally Posted by droopy

all righty folks,
I finally got the breakdown per file extension:

The file counts seem off. What command gave you those results?

[droopy]

Originally Posted by jhowell

The file counts seem off. What command gave you those results?

find -type f -name \*\.\* -printf '%f\0%s\n' | gawk '
BEGIN {
  FS = "\0";
}
{
  split($1, a, ".");
  ext = tolower(a[length(a)]);
  files[ext] += 1;
  size[ext] += $2;
}
END {
  PROCINFO["sorted_in"] = "@ind_str_asc";
  for (ext in files) {
    "numfmt --grouping " files[ext] | getline neatfiles
    "numfmt --to=iec-i --suffix=B --format='%.1f' " size[ext] | getline neatsize
    print "*." ext " " neatfiles " files, " neatsize;
  }
}'

[droopy]

Originally Posted by PeterT

Have you tried doing both portions of the calibre library maintenance? One ensures that all content referenced by the database is present in the file system; the other verifies that all files in the file system are present in the database.

Not yet. Are there any potential negative repercussions to doing so?

[haertig]

Originally Posted by droopy

azw, azw3, azw4 are all identical in Calibre and in terminal.

The terminal command says
*.cbz 3 files, 1.7GiB
But Calibre shows me 6 CBZ files which, interestingly, also adds up to 1.7 GB.

Terminal says
*.kepub 7,812 files, 2.3MiB
But Calibre says 3 Kepubs that add up, coincidentally to a very-close 2.2MB.

Terminal says
*.kfx 7,812 files, 2.0MiB
But Calibre says 1 KFX at 2.0MB

KFX-zip number in terminal and Calibre are both 7 files.
Number of MBP files in terminal and Calibre are the same.
Mobi number matches up.

Calibre says 214 ODT files but Calibre says just one ODT file.

Original_epub is a match.
PDF is a match.
ZIP is a match.

Taking the *.kepub example above, where your command line result said you have 7812 of those, can you tell us the exact command line that you used to come up with that number?

Earlier I suggested something like:

find . -iname "*.kepub" -print | wc -l

If you accidentally fat-fingered that and got something like this:

find . -iname "*" -print | wc -l

... then you would get a totally different (and massively incorrect) value returned.

The numbers from the find/wc command line should match exactly what Calibre tells you inside the app. Mine do. Make sure you are entering the command line correctly, and also that you are in the correct directory when executing the command. Although, I can't imagine you have a whole lot of *.kepub files on your computer outside of the Calibre directory, to give you such a largely incorrect number.

Just FYI, there are different parameters you can specify to the "wc" command that give different results (because they are counting different things). Make sure you are using the "-l" parameter. That is "dash lower-case-letter-ell".

Example (for my Calibre library):

$ find . -iname "*.azw3" -print | wc -l
3995
$ find . -iname "*.azw3" -print | wc -w
40580
$ find . -iname "*.azw3" -print | wc -c
312925
find . -iname "*.azw3" -print | wc
   3995   40580  312925

One more thing, try all of the following variations of the command (below). Before running the variations, change the "azw3" I used on my system to "kepub" for use on your system. In normal situations all these variations should return the same number. By "normal" I mean that your files end in "kepub", not "KEPUB" or "KepUB". -iname means case-insensitive match, -name means case-sensitive. We are grasping at straws here, but if you do indeed see different results returned, that might give us the clue we need to figure out what is going wrong for you.

$ find . -iname "*.azw3" -print | wc -l
3995
$ find . -iname "*azw3" -print | wc -l
3995
$ find . -iname "*\.azw3" -print | wc -l
3995
$ find . -name "*\.azw3" -print | wc -l
3995
$ find . -name "*.azw3" -print | wc -l
3995
$ find . -name "*azw3" -print | wc -l
3995

[droopy]

Originally Posted by haertig

Taking the *.kepub example above, where your command line result said you have 7812 of those, can you tell us the exact command line that you used to come up with that number?

Sure, I posted the code two posts up. See
https://www.mobileread.com/forums/sh...1&postcount=55
Update:
I'll put it in spoilers here too:

Spoiler:

find -type f -name \*\.\* -printf '%f\0%s\n' | gawk '
BEGIN {
FS = "\0";
}
{
split($1, a, ".");
ext = tolower(a[length(a)]);
files[ext] += 1;
size[ext] += $2;
}
END {
PROCINFO["sorted_in"] = "@ind_str_asc";
for (ext in files) {
"numfmt --grouping " files[ext] | getline neatfiles
"numfmt --to=iec-i --suffix=B --format='%.1f' " size[ext] | getline neatsize
print "*." ext " " neatfiles " files, " neatsize;
}
}'

[PeterT]

Originally Posted by droopy

Not yet. Are there any potential negative repercussions to doing so?

It will prompt you before correcting any issues. I can't help but think some of your experimentation has created extra files in the library.

Even if there are repercussions, I'm sure you do have backup(s) of your library....

[haertig]

Your awk script is bad. Actually, the awk commands are OK, but where you are failing is in trying to make things "neat" using numfmt. Delete that junk that attempts to beautify the output, just display the raw numbers, and your script works much better:

find -type f -name \*\.\* -printf '%f\0%s\n' | gawk '
  BEGIN {
    FS = "\0";
  }
  {
    split($1, a, ".");
    ext = tolower(a[length(a)]);
    files[ext] += 1;
    size[ext] += $2;
  }
  END {
    PROCINFO["sorted_in"] = "@ind_str_asc";
    for (ext in files) {
      print "*." ext " " files[ext] " files, " size[ext] " bytes";
    }
  }'