sudden and remarkable slowdown after user stopping backup of metadata

[RotAnal]

Hi All,
despite the suggestion of the manual, I have been putting my bigger and bigger (40+k) ebook collection on a (single user) NAS and so far things have gone fine (some but acceptable delay in retrieving books and saving edited metadata). Last time, however, I had the unhappy idea to make a backup of the database via Library Metadata Backup Status. Once I realised that the backup process would have to be extremely long I stopped it.
Since then on, I found that any access to NAS library has been hugely delayed (every and each action to GUI, including list scrolling is painful and always [not reponding] remark appears beside library name in the window tool bar. Moreover, I notice that whenever I hold the library open, NAS drives are constantly in activity, which does not end overnight, even when I don't do anything. Therefore I suspect that anyway backup internal processes have been not stopped actually.
No trouble at all when I access library through a local USB external drive, but it is not practical. In the NAS arragnement, router has been rebooted many times and cables are fine.
Any clue to fix this newer trouble of mine?

[kovidgoyal]

Metadata backup writes individual OPF files to the library, so if your library is ona slow drive, it will be very slow. ANd stopping it wont help, as it will automatically restart (calibre tries hard to make sure your db is backed up).

Your best bet is to temporarily move your library to a fasst disk. DO the backup and then move it back.

[theducks]

NAS / cloud drives are not supported.
The Calibre Library use a Transactional model.
Files are written, data base updated, old files deleted (when renamed)
Network caching and delays can hamper or even lose a step.

A modern USB2 or 3 direct attached hard drive should be able to keep pace with most of Calibres common operations

[RotAnal]

Thanks for diagnosis (though there is no sign that it is backup'ing besides HDD activity, unfortunately) and suggestion provided, Kovid!
Then it has been as I suspected, Calibre WILL eventually do the backup. It was INDEED a serious, unforgivable mistake to request a backup for a NAS library with 43520 books!
Yes, sir, I will backup the library (due its value it is often synched) and work in the copy, then backup it again!
As your smart suggestion of setting the env CALIBRE_OVERRIDE_DATABASE_PATH variable, I wonder if it would be possible to change machinery so that it works on ONE certain library only.

[theducks]

Originally Posted by RotAnal

Thanks for diagnosis (though there is no sign that it is backup'ing besides HDD activity, unfortunately) and suggestion provided, Kovid!
Then it has been as I suspected, Calibre WILL eventually do the backup. It was INDEED a serious, unforgivable mistake to request a backup for a NAS library with 43520 books!
Yes, sir, I will backup the library (due its value it is often synched) and work in the copy, then backup it again!
As your smart suggestion of setting the env CALIBRE_OVERRIDE_DATABASE_PATH variable, I wonder if it would be possible to change machinery so that it works on ONE certain library only.

The OPF type backup status is the first item on the Library Maintenance menu

[BetterRed]

@RotAnal - if you can do the 'database backup' via calibredb backup_metadata command you should find it's a lot faster. By design, the option in the GUI does it as a very low priority 'background' task, it typically chugs along at about a book a second, whereas the command line option does it as a normal priority 'foreground' task and executes as fast as it can.

Originally Posted by RotAnal

<snip>

I wonder if it would be possible to change machinery so that it works on ONE certain library only.

FWIW - I now have the content of my libraries on a Windows server. I have symlinks at the server referencing workstation resident library metadata databases. 'Logically' they are something like:

At server <libraryName>\metadata.db files reference ...\Metadata\<libraryName>.metadata.sqlite files at workstation.

This may or may not be doable on your NAS and workstation setup.

I changed the extensions to .sqlite because Windows regards .db files as something 'special' - thumbs.db and all that jazz. Obviously the server side symlinks must conform to calibre's expectations, so they retain the metadata.db name. On Windows the name of a symlink and its target do not have to be the same, I have a vague recollection from aeons ago that on another OS they did have to be the same.

BR

[RotAnal]

Originally Posted by BetterRed

if you can do the 'database backup' via calibredb backup_metadata command you should find it's a lot faster. By design, the option in the GUI does it as a very low priority 'background' task, it typically chugs along at about a book a second, whereas the command line option does it as a normal priority 'foreground' task and executes as fast as it can.

That was a very useful hint, thanks. In fact I updated all the ebooks in less than an hour (in a local 3.0 USB HDD)! On the other hand, I noticed that many entries in the batch output have been labelled as unknown. Is that a normal thing?

Originally Posted by BetterRed

FWIW - I now have the content of my libraries on a Windows server. I have symlinks at the server referencing workstation resident library metadata databases. 'Logically' they are something like:

At server <libraryName>\metadata.db files reference ...\Metadata\<libraryName>.metadata.sqlite files at workstation.

This may or may not be doable on your NAS and workstation setup.

I changed the extensions to .sqlite because Windows regards .db files as something 'special' - thumbs.db and all that jazz. Obviously the server side symlinks must conform to calibre's expectations, so they retain the metadata.db name. On Windows the name of a symlink and its target do not have to be the same, I have a vague recollection from aeons ago that on another OS they did have to be the same.

This suggestion has concepts completely new to me and I am not quite sure to have understood the details. I will study better the stuff in a future, anyway.

Thanks again

[BetterRed]

Originally Posted by RotAnal

That was a very useful hint, thanks. In fact I updated all the ebooks in less than an hour (in a local 3.0 USB HDD)! On the other hand, I noticed that many entries in the batch output have been labelled as unknown. Is that a normal thing?

No I've never seen that, but I have only used calibredb backup-metadata a few times over the past 5-6 years

If you could capture the actual text and post it here someone could probably work out what's going on.

Originally Posted by RotAnal

This suggestion has concepts completely new to me and I am not quite sure to have understood the details. I will study better the stuff in a future, anyway.

See Symbolic link - Wikipedia. You may read elsewhere that Symbolic Links are the equivalent of Windows shortcuts, that is false. The WikiP page explains the differences.

BR

[theducks]

Originally Posted by RotAnal

That was a very useful hint, thanks. In fact I updated all the ebooks in less than an hour (in a local 3.0 USB HDD)! On the other hand, I noticed that many entries in the batch output have been labelled as unknown. Is that a normal thing?
Thanks again

I wonder if you now have a damaged Library (DB) from using it with 2 different OS filesystems. Case issues happen when making changes in Windows, but the File system is case sensitive. Windows thinks Windows and WINDOWS as the same, Linux, as 2 exclusive names.
(it is fun to cleanup after this happens )

[RotAnal]

Originally Posted by theducks

I wonder if you now have a damaged Library (DB) from using it with 2 different OS filesystems

Duck, after fast, batch command fix as suggested by BetterRed, library is working fine!
Still, the opportunity to keep the metadata.db file local to the notebok, while leaving actual library in the NAS would be desirable, and I do hope that Calibre developers will implement that eventually through Calibre adjustable, internal pointers. As I wrote before, using a global env variable is not very practical when one has many, different libraries.