Why does Calibre Library take up so much space? 7000 books equals 50GB when exported

[gbm]

Originally Posted by droopy

GBM, I'm on Cinnamint 20 as well.
How do you get Nemo to calculate an entire folder's worth of subfolders and filter by file type, as it appears you've done?


I'm guessing there's somethingc more than just the ebook files (cbz/epub/htm/kfx) and the images. Maybe the "supporting structure" (let's call it the scaffolding) of Calibre is taking up a humongous amount of space.

Use nemo open the calibre library folder--select all--right click--properties; will give you the total number of items in the library and the total size of the library.

With the calibre library folder open use search for *.epub, etc, etc, will find all the epubs in the calibre library folder then use select all--right click--properties;will give you the total count of epub, etc, etc, plus the total size. Repeat until all ebook types have been checked.

bernie

[latepaul]

Originally Posted by droopy

Is there a command I can run on a folder (and all its subfolders) to tell me all the file types (file extensions) contained therein? Maybe we're missing some file types from our analysis.
Update: I changed your "100M" to "1M" and got this:

That's too verbose for me! Lol. so much lines about "jpeg image data".

Mostly jpg's, PDFs, Zips and the calibre sqlite database.

To be honest it's a bit of a mystery looking at what you've run. How big is metadata.db?

[droopy]

Originally Posted by latepaul

Mostly jpg's, PDFs, Zips and the calibre sqlite database.

To be honest it's a bit of a mystery looking at what you've run. How big is metadata.db?

In post #25 https://www.mobileread.com/forums/sh...3&postcount=25
I wrote

$ find . -iregex ".*\.\(db\|json\|opf\)$" -exec du -csh '{}' + | tail -1
6.9M total

Does that answer your question, or am I missing something?

I hope I can figure out what mysterious stuff are taking up a mammoth amount of space in my calibrelibrary.

[haertig]

You can look at your files, sorted largest to smallest, like this:

find . -type f -ls | cut -c48- | sort -rn | sed -e 's/\\//g' | pg

I am sending the output to "pg", so it pauses after ever screen of data. Hit 'return' for the next screen, "ctrl-c" to quit.

The output from the above command is not pretty, but it is functional. Make sure to enlarge your terminal window to full screen, otherwise the lines may wrap making it a nightmare to tell what is what.

[droopy]

find . -type f -ls | cut -c48- | sort -rn | sed -e 's/\\//g' | pg
bash: pg: command not found

Is there a way to do this while ignoring the file extensions I've checked already (epub, cbz, mobi, pdf, db, etc)?

[haertig]

Originally Posted by droopy

find . -type f -ls | cut -c48- | sort -rn | sed -e 's/\\//g' | pg
bash: pg: command not found

Is there a way to do this while ignoring the file extensions I've checked already (epub, cbz, mobi, pdf, db, etc)?

find . -type f -ls | egrep -v '\.(epub|cbz|mobi|pdf|db)' | cut -c48- | sort -rn | sed -e 's/\\//g' | more

Notice now I am using the "more" command rather than the "pg" command. They basically do the same thing. Note that with "more" you use the spacebar to advance a page, rather than enter key like you do with "pg". On Linux distros, you will usually find either the pg command or the more command installed, usually both. In your case, you don't have pg. So use more instead.

Nore: The -v option to the egrep command means "everything except". So the above command will list everything except epub's, cbz's, mobi's, pdf's and db's

[droopy]

Originally Posted by haertig

find . -type f -ls | egrep -v '\.(epub|cbz|mobi|pdf|db)' | cut -c48- | sort -rn | sed -e 's/\\//g' | more

Notice now I am using the "more" command rather than the "pg" command. They basically do the same thing. Note that with "more" you use the spacebar to advance a page, rather than enter key like you do with "pg". On Linux distros, you will usually find either the pg command or the more command installed, usually both. In your case, you don't have pg. So use more instead.

Nore: The -v option to the egrep command means "everything except". So the above command will list everything except epub's, cbz's, mobi's, pdf's and db's

Is there a way to print the output into some file? It may be easier to view/search/share if it's in some output file.
Update: I'm not with my Calibre computer right now, and plan to try this tomorrow.

[haertig]

Originally Posted by droopy

Is there a way to print the output into some file? It may be easier to view/search/share if it's in some output file.

find . -type f -ls | egrep -v '\.(epub|cbz|mobi|pdf|db)' | cut -c48- | sort -rn | sed -e 's/\\//g' > filename.txt

Commands like the above are just a bunch of simple, basic Linux commands strung together using "pipes". e.g., the output of one goes to the input of the next. The command above used to end with "| more", the output of the command before was "piped to the input of the more command". I replaced that with "> filename.txt". This takes the output of the previous command and "redirects it into a file".

You might want to do a Google search for "Linux command line free ebook". There are many available. Lot's on Kindle Unlimited too, sometimes you will find them for free purchase there as well. Using pipes and redirects are basic to understanding the command line.

[droopy]

all righty folks,
I finally got the breakdown per file extension:

*.azw 6 files, 33.2MiB
*.azw3 178 files, 2.4GiB
*.azw4 3 files, 227.7MiB
*.cbz 3 files, 1.7GiB
*.chm 1 files, 883.4KiB
*.db 1 files, 22.9MiB
*.djvu 17 files, 67.3MiB
*.epub 6,775 files, 36.5GiB
*.htmlz 113 files, 468.6MiB
*.jpg 7,748 files, 2.0GiB
*.json 7,748 files, 29.4KiB
*.kepub 7,748 files, 2.3MiB
*.kfx 7,748 files, 2.0MiB
*.kfx-zip 7 files, 26.4MiB
*.mbp 12 files, 1.5KiB
*.mobi 214 files, 348.5MiB
*.odt 214 files, 30.8KiB
*.opf 3,352 files, 38.1MiB
*.original_epub 193 files, 1.5GiB
*.pdf 768 files, 7.3GiB
*.txt 768 files, 372.8KiB
*.zip 2 files, 1.3MiB

[theducks]

You could recover 1.5G by off loading Original_EPUB. (click the format in the tag browser: select all:rightclick Delete: remove specific format:<select>

IMHO these can usually go after you have created you FINAL clean EPUB (or AZW3)
as always YMMV

Your EPUB must have lots of (big) embedded graphics or embedded fonts to hit 36G with 7K books

This seems odd. note the file count

*.json 7,748 files, 29.4KiB   <I only have ONE
*.kepub 7,748 files, 2.3MiB
*.kfx 7,748 files, 2.0MiB

[ownedbycats]

Polish ePubs has a "losslessly compress images" option that may also help reduce the filesize.

[droopy]

theducks,
ah, yeah, that's odd. that the file count for all 3 disparate extensions are identical.
what's going on, I wonder.

[theducks]

Originally Posted by droopy

theducks,
ah, yeah, that's odd. that the file count for all 3 disparate extensions are identical.
what's going on, I wonder.

They are not big, but it may be from some plugin you are using (I, obviously, am not, since the only json is the standard for recovery)

[JSWolf]

What's with the 7748 ,jason files?

[theducks]

Originally Posted by JSWolf

What's with the 7748 ,jason files?

I suspect some form of Annotation archive