collect editings in a memcache before saving them alltogether

RotAnal · 02-10-2019, 01:12 PM

Dear Sirs,
I wonder if there is an option mechanism in the current release of Calibre so that one can collect somewhere in a memory cache several editings within one library and then eventually save all them in one click. In facts in my library it is a pain to wait for every saving of library (it lasts minutes each). I am aware that is not save in case of power outage, but it would be a nice thing if developer could let user to take his/her own risk.

Adoby · 02-10-2019, 01:53 PM

There is a similar system already in place.

When you update metadata then the updates are saved in the calibre library database file. The updates are usually not saved to the book formats. An exception to this is if you update the most basic metadata; Author(s) and Title. Then folders and files are renamed and moved around, as needed. That can take a while.

This means that you should strive to get Author(s) and Title right at once as you add books to calibre. And then not change those fields. All(?) other updates are only stored in the database, so they are really fast. Later, when you convert, send books to devices or save to disk, then the updated metadata in the database is applied to the actual books. The actual ebook files. .epub .mobi and so on.

The calibre library database file takes up relatively little space. That means that it can be stored separate from the rest of the library, for instance using shortcuts, on a fast disk. For instance on a system drive SSD. Or even, if you are very brave or very stupid, on a RAM-disk. This may improve performance when you change metadata.

When ebook files later are updated with the metadata, from the database, they are unpacked, updated and repacked. This is very "expensive" and it takes place in the TEMP folder. You can speed things up by making sure the TEMP folder is on the fastest drive or even on a RAM disk.

Some of these things can also be controlled with special environment variables that calibre recognizes:

CALIBRE_TEMP_DIR - sets the temporary directory used by calibre

CALIBRE_OVERRIDE_DATABASE_PATH - allows you to specify the full path to metadata.db. Using this variable you can have metadata.db be in a location other than the library folder.

https://manual.calibre-ebook.com/customize.html#id1

BetterRed · 02-10-2019, 04:25 PM

Quote:

Originally Posted by RotAnal

Dear Sirs,
I wonder if there is an option mechanism in the current release of Calibre so that one can collect somewhere in a memory cache several editings within one library and then eventually save all them in one click. In facts in my library it is a pain to wait for every saving of library (it lasts minutes each). I am aware that is not save in case of power outage, but it would be a nice thing if developer could let user to take his/her own risk.

Something is seriously wrong with your setup if you mean it takes 'minutes' to 'save' when you click OK in the single metadata edit dialogue, i.e.

Click image for larger version

Name: 1.jpg
Views: 91
Size: 144.6 KB
ID: 169619

I am unsure of what your comment about "power outages" is referring to, afaik calibre makes no special provisions in this regard. Perhaps you can be more explicit.

If your library is on a local drive that uses a file system such as NTFS, HFS+ or Ext3/4 a change to an author name or title will be fast because calibre will use hardlink clones to 'move' the files from the 'old' to the 'new' folders before deleting the 'old' folders.

I have a 'textual' library of almost 150,000 titles (about 1MB per title) and a 'media' library of almost 10,000 titles (average 500MB per title). I get no noticeable delays in editing metadata on either library. Even if I change the name of an author in my media library containing a 100 titles it is done more or less instantly because of the hardlink moves.

BR

theducks · 02-10-2019, 05:05 PM

IMHO If you are worried about Power outages, get a UPS to run your PC (and Monitor). That gives you ~5 min (typ small unit) to do a traditional exit.

OTOH Even having to shut down the Calibre Server, it takes me 3 -5 seconds to exit Calibre.

SOMETHING IS WRONG. a/v, NAS or a SHARE active on your Library
My metadata.db is 46M

02-10-2019, 01:12 PM	#1
RotAnal Connoisseur Posts: 87 Karma: 1234 Join Date: Sep 2012 Device: Onyx Boox M92	collect editings in a memcache before saving them alltogether Dear Sirs, I wonder if there is an option mechanism in the current release of Calibre so that one can collect somewhere in a memory cache several editings within one library and then eventually save all them in one click. In facts in my library it is a pain to wait for every saving of library (it lasts minutes each). I am aware that is not save in case of power outage, but it would be a nice thing if developer could let user to take his/her own risk.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre failed to collect news	rusky	Calibre	2	10-17-2011 03:42 AM
Losing files when saving saving to disk	theaccountant	Library Management	4	03-10-2011 02:38 PM
Journal Not Saving Writing When Moving to a New Page or Saving it	eberhardt333	enTourage Archive	5	11-24-2010 12:47 AM
Unutterably Silly Collect more 'no' than 'yes' votes	ProDigit	Lounge	52	05-02-2009 02:13 PM
iLiad How to collect system resource?	ericshliao	iRex Developer's Corner	5	10-27-2008 09:35 AM

02-10-2019, 01:53 PM	#2
Adoby Handy Elephant Posts: 1,737 Karma: 26785684 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Samsung Galaxy Tab S8 Ultra	There is a similar system already in place. When you update metadata then the updates are saved in the calibre library database file. The updates are usually not saved to the book formats. An exception to this is if you update the most basic metadata; Author(s) and Title. Then folders and files are renamed and moved around, as needed. That can take a while. This means that you should strive to get Author(s) and Title right at once as you add books to calibre. And then not change those fields. All(?) other updates are only stored in the database, so they are really fast. Later, when you convert, send books to devices or save to disk, then the updated metadata in the database is applied to the actual books. The actual ebook files. .epub .mobi and so on. The calibre library database file takes up relatively little space. That means that it can be stored separate from the rest of the library, for instance using shortcuts, on a fast disk. For instance on a system drive SSD. Or even, if you are very brave or very stupid, on a RAM-disk. This may improve performance when you change metadata. When ebook files later are updated with the metadata, from the database, they are unpacked, updated and repacked. This is very "expensive" and it takes place in the TEMP folder. You can speed things up by making sure the TEMP folder is on the fastest drive or even on a RAM disk. Some of these things can also be controlled with special environment variables that calibre recognizes: CALIBRE_TEMP_DIR - sets the temporary directory used by calibre CALIBRE_OVERRIDE_DATABASE_PATH - allows you to specify the full path to metadata.db. Using this variable you can have metadata.db be in a location other than the library folder. https://manual.calibre-ebook.com/customize.html#id1

02-10-2019, 05:05 PM	#4
theducks Well trained by Cats Posts: 31,249 Karma: 61360164 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	IMHO If you are worried about Power outages, get a UPS to run your PC (and Monitor). That gives you ~5 min (typ small unit) to do a traditional exit. OTOH Even having to shut down the Calibre Server, it takes me 3 -5 seconds to exit Calibre. SOMETHING IS WRONG. a/v, NAS or a SHARE active on your Library My metadata.db is 46M

Advert