Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 05-20-2018, 05:29 AM   #1
svda
Connoisseur
svda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipse
 
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
Conflicts when updating metadata

I used to manage my calibre database on my local computer and save the epubs to to Synology (Save to disk in one single directory). This went all smooth.
I recently moved my calibre database to my Synology NAS and furthermore arranged it to cloudsync this directory to ms onedrive.
I thought everything was going well, until I had some problems with newly bought epubs that had a python error and path conflict whenever I tried to change the metadata of the book (adding bookdescription or changing cover). I only got to see the error when I opened the epub. The multiple conflicting epubs from 1 book mention ADMIN and CONFLICT in the filename. Would I not have opened the epub to check, I would not have been aware of the conflict/error.

So, I decided to test. I moved my database back local to my computer. Added the new book, did all the changes (forgot to polish ). But hey book was accepted and opened up normally.

So, obviously my new working strategy is going wrong. I would like to know precisely where the culprit is. Is it already going wrong when I just keep my database on synology? My guess. Or is the Synology Cloudsync bothering the calibre process? Should I try to cloudsync a scheduled times and not immediately ?

Thanks for you help.

Last edited by svda; 05-20-2018 at 05:31 AM.
svda is offline   Reply With Quote
Old 05-20-2018, 05:44 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The most basic rule is: You have to make sure that only a single program is touching the calibre library at a time. That means do not run your sync tool while calibre is running and vice versa
kovidgoyal is offline   Reply With Quote
Old 05-20-2018, 05:54 AM   #3
svda
Connoisseur
svda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipse
 
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
Is keeping my database on Synology not the problem? I'm also not very successful at "fixing" the errors with the calibre database tools. It has missing files and duplicates for the same books. What is the correct way in correct order to fix missing bookformats while also having extra book formats for the same books ?

My calibre client is 64-bit on windows, the storage is on synology through samba.
The cloudsync is only allowed to read and sync the files to onedrive. The cloudsync process is not allowed to change the masterdirectory on my synology. I presumed this was safe.

Last edited by svda; 05-20-2018 at 05:58 AM.
svda is offline   Reply With Quote
Old 05-20-2018, 06:30 AM   #4
svda
Connoisseur
svda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipse
 
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
I tested it out and it's not the cloudsync that causes the errors.

I paused cloudsync. Made sure my database was errorfree by deleting the book that caused errors. Checked database with the maintenance tools.

Stopped and restarted Calibre. Downloaded the kindle books again in empty directory. Reimported them, changed metadata. Ran check, same errors.

As far as I'm aware there's no purging process or other RAID-5 maintenance/scrubbing process going on. I'm running latest calibre version 3.23

Also experienced this recently with google play books and kobo, this weekend ...


Anyone else with Synology having/had/solved a similar problem ?

Last edited by svda; 05-20-2018 at 08:06 AM.
svda is offline   Reply With Quote
Old 05-20-2018, 12:00 PM   #5
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
Windows and Linux filesystems has different case sensitivity. Your Synology NAS use a variant of Linux, DSM.

In Windows the files: "BookTitle.epub" and "booktitle.epub" are the same files.
In Linux the files: "BookTitle.epub" and "booktitle.epub" are two different files.
Same with folder names...

So when you copy or move your library and update metadata, fix spelling and title/author leading UPPER/lower spelling, and so on, you may slowly cause increasing corruption in your calibre library. Weird errors with missing formats/files or books and extra duplicate or orphan folders and books.

I stopped using Windows and Samba/CIFS to access my calibre library on my Synology NAS and instead used Linux and NFS. Now I have no problems. I only access my books on my NAS from either a laptop or a desktop PC, but never ever simultaneously. I assume other NAS brands have the exact same problem.

To fix your problems I suggest that you use the functions in calibre to MOVE your library back to your PC. Then check the folder for the old library for orphan books and add them to your libarary. That way you should not loose any books. You may have duplicates if you fixed "lost" books before...

Then use your NAS only to make backups of your calibre library. Or convert to Linux and NFS...

There may also be some settings to the Samba/filesystem configuration on the NAS that also might fix this, I never researched that.

Errors in calibre due to different case sensitivity in filesystems are often delayed. Meta data changes to files and filenames are not written to calibre library at once, just stored in the database. Later, perhaps after weeks, months or years, when filenames for books are updated for some other reason, some files and or folders may go missing and/or are duplicated seemingly at random. Very strange errors if you don't realize what is going on.
Adoby is offline   Reply With Quote
Old 05-20-2018, 05:26 PM   #6
pleiadesc
Connoisseur
pleiadesc began at the beginning.
 
Posts: 61
Karma: 10
Join Date: Nov 2016
Device: none
Hi Adoby, if the problem is case sensitivity, would it be technically possible to force calibre into using a windows or linux case? maybe an option to force everything in the library to use small letters and use up to a certain character count, so you wont have to worry where to use it.
pleiadesc is offline   Reply With Quote
Old 05-20-2018, 11:38 PM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
character case is just one possible problem there are others. Filesystems on linux an windows have fundamentally different semantics.
kovidgoyal is offline   Reply With Quote
Old 05-21-2018, 05:26 AM   #8
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
Quote:
Originally Posted by pleiadesc View Post
Hi Adoby, if the problem is case sensitivity, would it be technically possible to force calibre into using a windows or linux case? maybe an option to force everything in the library to use small letters and use up to a certain character count, so you wont have to worry where to use it.
Anything is technically possible. But is it practical?

It might be possible to add some form of file/folder name normalization to calibre. So only a common subset of characters are used for all file and folder names. For instance only lower case ASCII up to 127. But it might cause problems with existing libraries. I don't know. And there may be other issues as Kovid suggests. For instance allowed (combined) length of paths and file names differ a lot. Could perhaps also be normalized?

Is it technically possible for you to stop using calibre in Windows with the library on a NAS with a Linux filesystem? That is all you have to do.

Method 1.
Have the library on your Windows PC. Use the NAS only for backups.

Method 2.
Have the library on the NAS. Access it from a Linux install on your PC using NFS. Either a virtual machine or dual boot.

Method 3.
Stop using Windows. Use only Linux and NFS.

I used method 3 and like it! many other benefits as well...

Last edited by Adoby; 05-21-2018 at 05:43 AM.
Adoby is offline   Reply With Quote
Old 05-21-2018, 06:37 AM   #9
svda
Connoisseur
svda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipse
 
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
Thanks Adoby, much appreciated advice. I'm not so into Linux. I chose this setup because it was enormously practical. Access the database from any pc in the house (single user) and instantly backed-up in the cloud where I could access it anywhere from any smartphone/tablet.

So, first moving the db's back to pc and checking integrity. Then some rethinking to do.
svda is offline   Reply With Quote
Old 05-21-2018, 06:54 AM   #10
svda
Connoisseur
svda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipse
 
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
Found 383 books in my library, 2 formats : 303 azw3 and 77 mobi ... that doesn't add up, does it ?Integrity checks ok says calibre.
svda is offline   Reply With Quote
Old 05-21-2018, 07:08 AM   #11
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
There may be a few orphan books in the library folder. Move the library and see what is left behind.

Method 4.
Have your main calibre database on your main windows PC. And a full backup copy of the library on the NAS.

You can access the backup copy calibre library from other computers on your network. To avoid problems only update/add/delete/change books from the main PC. This setup means that you can access the main library from your main PC at the same time as someone else access the backup copy.

There are many sync-utilities that makes it easy to update the backup copy so it is identical to the main library. I use rsync.
Adoby is offline   Reply With Quote
Old 05-21-2018, 09:25 AM   #12
svda
Connoisseur
svda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipse
 
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
I located them in the search with formats:false, while the calibre maintenance tool did not report them.

Off topic, so if I keep my MP3 files on a linux/unix system like my Synology NAS AND perform some internal edits on them based from a windows PC and software, like with MP3tag, I could corrupt my MP3 files also ?

Last edited by svda; 05-21-2018 at 09:41 AM.
svda is offline   Reply With Quote
Old 05-21-2018, 10:35 AM   #13
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
Yes. But normally you fix errors as you go. Delete duplicates and so on. And edits are typically made directly in situ. But sometimes you may end up with corrupt or lost files.

calibre is extra sensitive because it stores metadata in the database and does often not update the files at once. It is a form of "lazy" update. Lazy updates makes it possible to do things like update metadata for thousands of ebooks in seconds. The actual ebooks are updated only when you save them or send them to a device. If you used the same type of direct updates as MP3tag does it could take hours to update a bunch of ebooks.

And if calibre is run on a Windows computer but store the library on a Linux filesystem the lazy updates may cause errors, as you noticed.

That is why the manual EXPLICITLY says that you should not have your library on a networked drive/NAS.

https://manual.calibre-ebook.com/faq...rked-drive-nas
Quote:
Do not put your calibre library on a networked drive.

A filesystem is a complex beast. Most network filesystems lack various filesystem features that calibre uses. Some don’t support file locking, some don’t support hardlinking, some are just flaky. Additionally, calibre is a single user application, if you accidentally run two copies of calibre on the same networked library, bad things will happen. Finally, different OSes impose different limitations on filesystems, so if you share your networked drive across OSes, once again, bad things will happen.
Despite this it is POSSIBLE to have the library on a networked drive/NAS, but you have to be extra extra careful and know what you are doing or you end up with a corrupt library.
Adoby is offline   Reply With Quote
Old 05-21-2018, 11:00 AM   #14
svda
Connoisseur
svda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipsesvda can illuminate an eclipse
 
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
Ok, Adoby. thank you for explaining it to me.
svda is offline   Reply With Quote
Old 05-22-2018, 01:07 AM   #15
pleiadesc
Connoisseur
pleiadesc began at the beginning.
 
Posts: 61
Karma: 10
Join Date: Nov 2016
Device: none
Thank you Kovid and Adoby, I will use the method 4
pleiadesc is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
how to compare new metadata with old before updating ahab1972 Library Management 7 09-24-2015 04:16 AM
Why is my metadata not updating? rhino79 Library Management 25 09-23-2014 01:34 AM
Metadata not updating? jstabe Conversion 0 03-02-2013 05:59 PM
batch updating metadata webwizard Library Management 5 04-15-2011 04:40 PM
Updating Metadata without and ISBN herbycanopy Calibre 7 05-22-2010 01:16 AM


All times are GMT -4. The time now is 04:53 PM.


MobileRead.com is a privately owned, operated and funded community.