content-server and CALIBRE_CACHE_DIRECTORY

[Geremia]

Running the content-server, I noticed my CALIBRE_CACHE_DIRECTORY gets very large (>2 GB) very quickly. Why is this? What are all the temp files for? It appears many of the actual PDFs etc. are being copied to the cache directory. Does this have something to do with zero-copy file transfers?

[kovidgoyal]

When the content server sends files it has to update metadata in them, and it does not do that to the copies in the library. Which means it has to make a copy of the file outside the library. Those files are kept around so that the operation does not need to be repeated each time a request for the file is made.

[nqk]

Are you talking about the srvb folder?

It has become a problem for me to sync the whole Calibre Portable to Cloud services (I currently choose to ignore that folder). Maybe Calibre should clean it up time by time based on the last access date, or set a size limit?

[kovidgoyal]

It's a cache folder, why are you syncing it? calibre portable has to use only the portable folder for everything, so if you are using calibre portable, simply exclude the cache folder from your syncs.

[Geremia]

Originally Posted by kovidgoyal

When the content server sends files it has to update metadata in them, and it does not do that to the copies in the library. Which means it has to make a copy of the file outside the library. Those files are kept around so that the operation does not need to be repeated each time a request for the file is made.

Is it safe to delete the /tmp/calibre-3.14.0-* folder if it gets too large, or does Calibre do its own garbage collection on it?

[kovidgoyal]

calibre cleans up temp files automatically, you should never delete temp files while it is running

[Geremia]

Originally Posted by kovidgoyal

When the content server sends files it has to update metadata in them

Why does it do that? What metadata is updated?

[Adoby]

Calibre implements lazy update of ebooks. If you update metadata for a book this is updated in the database but not inside the actual ebook file. When you send the ebook file the ebook file is updated in the sent copy of the ebook file. Possibly based on plugboards and/or when converted.

This means that ebook files don't have to be updated all the time. They can stay unchanged in their original pristine state. This is very good for safety reasons as well as for performance reasons. You may change metadata many times, for instance in batch jobs, but you might never actually send/save a copy of the ebook file. Or just a few times compared to how often you may update metadata. To update metadata in the database is very fast. To update metadata in the ebook file is very slow. In comparison, at least.

It is possible to to force calibre to update metadata inside the ebook files. In effect overwriting the original ebook file. Make sure you have good backups before you do that...

[Geremia]

Originally Posted by kovidgoyal

When the content server sends files it has to update metadata in them

Why? What metadata needs to be updated?

[davidfor]

Originally Posted by Geremia

Why? What metadata needs to be updated?

When you edit metadata in the library, calibre does not update the actual book. It only updates the database. The books will be updated when you do something that takes the book out of the library, such as doing a save-to-disk or send-to-device and when downloading the book using the server. If you want the book inside the library updated, it will be done during actions such as a conversion, editing the book, Polish books or using the Embed metadata function.

[DNSB]

Originally Posted by Geremia

Why? What metadata needs to be updated?

Short and simple answer is any metadata that was not read from the ebook during the import process.

To expand a bit, some people may be happy with the crap that passes for metadata in many downloaded ebooks. Others prefer to correct errors, add a decent cover image, etc. For many people, their workflow is download the book and import it into calibre. Then they update the metadata since the series is missing or incorrect, the book comment is blank, perhaps you want to update the cover, correct the author name since you are CERTAIN that Unknown did not write the book, they want to remove the mass of tags that many authors/publishers seem to use to "optimize" their books showing up in searches and replace them with more useful (to you) tags, etc.

This modified information is not added to the copy of the book in your calibre library unless you deliberately choose to do so (Polish, Embed Metadata, Modify Epub, whatever). When you export the book from your library by sending to your device, saving to a folder or download from the content server, prior to sending the book, a temporary copy will be created and updated with that modified metadata.

Hopefully, this helps you a bit.

[Geremia]

Originally Posted by davidfor

When you edit metadata in the library, calibre does not update the actual book. It only updates the database. The books will be updated when you do something that takes the book out of the library, such as doing a save-to-disk or send-to-device and when downloading the book using the server.

Originally Posted by DNSB

This modified information is not added to the copy of the book in your calibre library unless you deliberately choose to do so (Polish, Embed Metadata, Modify Epub, whatever).

Thank you. This helps.

[JSWolf]

Originally Posted by davidfor

When you edit metadata in the library, calibre does not update the actual book. It only updates the database. The books will be updated when you do something that takes the book out of the library, such as doing a save-to-disk or send-to-device and when downloading the book using the server. If you want the book inside the library updated, it will be done during actions such as a conversion, editing the book, Polish books or using the Embed metadata function.

You can configure Calibre to not update the metadata when you do actions such as save-to-disk. I don't want Calibre doing to the updating. If I want it done, I'll do it by hand in the editor (and I have the editor configured to not update the metadata when loading the ePub).

IMHO, the default should be to not update the metadata like is done now and let the user turn that on if wanted.

[davidfor]

Originally Posted by JSWolf

You can configure Calibre to not update the metadata when you do actions such as save-to-disk. I don't want Calibre doing to the updating. If I want it done, I'll do it by hand in the editor (and I have the editor configured to not update the metadata when loading the ePub).

IMHO, the default should be to not update the metadata like is done now and let the user turn that on if wanted.

Sorry, that's just wrong. And bad advice. Why on earth to you want to edit the metadata twice? Once in the calibre metadata editor and again manually when editing the book? And do you also turn of the metadata update when sending to the device? And turn off the update to your Kobo devices?

[BetterRed]

Originally Posted by davidfor

Sorry, that's just wrong. And bad advice. Why on earth to you want to edit the metadata twice? Once in the calibre metadata editor and again manually when editing the book?

In my case to remove what I regard as clutter from the book format file - e.g. description, subject, commercial identifiers etc. I only have the books I'm reading and a few reference texts on my devices - rarely more than a dozen.

BR