Temp Files when downloading from content server

mobirick · 06-26-2021, 11:47 PM

Whenever I download a file through the Content Server, it gets added to my TEMP (/tmp) directory.

Is there a way to not copy the file to the temp directory when downloading? I found that the directory was taking over 16GB!

mobirick · 06-27-2021, 12:44 AM

(Couldn't find the edit button, so adding some context to the post)

- Calibre 5.22.1 linuxserver.io docker
- Files were CBZ, CBR, so some files can be as large as 1GB

When restarting the docker, the /tmp/ files were not cleaned up, and Calibre created a new directory for itself in /tmp/. I suspect that might be a problem of the docker so created a startup script that purges /tmp/calibre* before starting.

How can I keep Calibre temp directory clean, and a reasonable size, without restarting the Docker and manually clearing the cache?

kovidgoyal · 06-27-2021, 01:43 AM

No, it's not possible as the files served by the content server have to be copied out of the library folder, otherwise the library would get locked for arbitrary periods of time while the file was being downloaded.

The content server thus maintains a cache of files outside the library folders. You can control its location using the CALIBRE_TEMP_DIR environment variable. And the cache is cleaned up automatically on server restart.

mobirick · 06-27-2021, 05:09 AM

First, thanks for the super quick response and the love you have for this community!!

If Calibre is running forever, does that mean that the cache will never get cleaned up?

I'm thinking of setting up a scheduler to delete large cached files (CBR, CBZ, PDF, etc.) a few hours after being created. Does that sound like the best way to handle it?

kovidgoyal · 06-27-2021, 05:14 AM

Yes cache files are never removed until exit, there isnt much point usually, since the cache will never be larger than the actual library and book libraries are typically small. Just think of it as the storage requirement for the library being twice the actual size.

You cant delete the files out from under the server, as the server has an in memory structure that it uses to manage the cache and the two cant get out of sync.

mobirick · 06-27-2021, 05:17 AM

> the server has an in memory structure that it uses to manage the cache and the two cant get out of sync.

Do you mean that calibre may crash if I remove files from the /tmp/ directory, even though they should never be accessed again?

kovidgoyal · 06-27-2021, 06:52 AM

If you remove temporary files from /tmp while a program that is using them is running any program will crash. You are free to delete /tmp after quitting a program not before.

06-26-2021, 11:47 PM	#1
mobirick Junior Member Posts: 4 Karma: 10 Join Date: Jun 2021 Device: Kindle Fire 10 HD Plus / Kindle Oasis 3rd Gen	Temp Files when downloading from content server Whenever I download a file through the Content Server, it gets added to my TEMP (/tmp) directory. Is there a way to not copy the file to the temp directory when downloading? I found that the directory was taking over 16GB!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Strange error downloading from content server	Metapath	Server	4	05-13-2021 09:46 PM
Calibre content server - issues downloading .cbz and .cbr files over Wi-Fi	Shiftedwrath	Devices	10	10-30-2020 01:38 AM
Downloading All From Content Server	ziz	Related Tools	7	05-10-2015 02:30 PM
Downloading with Content Server and Calibre Companion	K8HW	Calibre Companion	1	01-29-2015 01:21 PM
Downloading files from content server fails with sony prs t-2	sws	Devices	2	05-23-2013 03:36 PM

06-27-2021, 12:44 AM	#2
mobirick Junior Member Posts: 4 Karma: 10 Join Date: Jun 2021 Device: Kindle Fire 10 HD Plus / Kindle Oasis 3rd Gen	(Couldn't find the edit button, so adding some context to the post) - Calibre 5.22.1 linuxserver.io docker - Files were CBZ, CBR, so some files can be as large as 1GB When restarting the docker, the /tmp/ files were not cleaned up, and Calibre created a new directory for itself in /tmp/. I suspect that might be a problem of the docker so created a startup script that purges /tmp/calibre* before starting. How can I keep Calibre temp directory clean, and a reasonable size, without restarting the Docker and manually clearing the cache?

06-27-2021, 01:43 AM	#3
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No, it's not possible as the files served by the content server have to be copied out of the library folder, otherwise the library would get locked for arbitrary periods of time while the file was being downloaded. The content server thus maintains a cache of files outside the library folders. You can control its location using the CALIBRE_TEMP_DIR environment variable. And the cache is cleaned up automatically on server restart.

06-27-2021, 05:09 AM	#4
mobirick Junior Member Posts: 4 Karma: 10 Join Date: Jun 2021 Device: Kindle Fire 10 HD Plus / Kindle Oasis 3rd Gen	First, thanks for the super quick response and the love you have for this community!! If Calibre is running forever, does that mean that the cache will never get cleaned up? I'm thinking of setting up a scheduler to delete large cached files (CBR, CBZ, PDF, etc.) a few hours after being created. Does that sound like the best way to handle it?

06-27-2021, 05:14 AM	#5
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Yes cache files are never removed until exit, there isnt much point usually, since the cache will never be larger than the actual library and book libraries are typically small. Just think of it as the storage requirement for the library being twice the actual size. You cant delete the files out from under the server, as the server has an in memory structure that it uses to manage the cache and the two cant get out of sync.

06-27-2021, 05:17 AM	#6
mobirick Junior Member Posts: 4 Karma: 10 Join Date: Jun 2021 Device: Kindle Fire 10 HD Plus / Kindle Oasis 3rd Gen	> the server has an in memory structure that it uses to manage the cache and the two cant get out of sync. Do you mean that calibre may crash if I remove files from the /tmp/ directory, even though they should never be accessed again?

06-27-2021, 06:52 AM	#7
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	If you remove temporary files from /tmp while a program that is using them is running any program will crash. You are free to delete /tmp after quitting a program not before.