05-20-2018, 05:29 AM | #1 |
Connoisseur
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
|
Conflicts when updating metadata
I used to manage my calibre database on my local computer and save the epubs to to Synology (Save to disk in one single directory). This went all smooth.
I recently moved my calibre database to my Synology NAS and furthermore arranged it to cloudsync this directory to ms onedrive. I thought everything was going well, until I had some problems with newly bought epubs that had a python error and path conflict whenever I tried to change the metadata of the book (adding bookdescription or changing cover). I only got to see the error when I opened the epub. The multiple conflicting epubs from 1 book mention ADMIN and CONFLICT in the filename. Would I not have opened the epub to check, I would not have been aware of the conflict/error. So, I decided to test. I moved my database back local to my computer. Added the new book, did all the changes (forgot to polish ). But hey book was accepted and opened up normally. So, obviously my new working strategy is going wrong. I would like to know precisely where the culprit is. Is it already going wrong when I just keep my database on synology? My guess. Or is the Synology Cloudsync bothering the calibre process? Should I try to cloudsync a scheduled times and not immediately ? Thanks for you help. Last edited by svda; 05-20-2018 at 05:31 AM. |
05-20-2018, 05:44 AM | #2 |
creator of calibre
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
The most basic rule is: You have to make sure that only a single program is touching the calibre library at a time. That means do not run your sync tool while calibre is running and vice versa
|
05-20-2018, 05:54 AM | #3 |
Connoisseur
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
|
Is keeping my database on Synology not the problem? I'm also not very successful at "fixing" the errors with the calibre database tools. It has missing files and duplicates for the same books. What is the correct way in correct order to fix missing bookformats while also having extra book formats for the same books ?
My calibre client is 64-bit on windows, the storage is on synology through samba. The cloudsync is only allowed to read and sync the files to onedrive. The cloudsync process is not allowed to change the masterdirectory on my synology. I presumed this was safe. Last edited by svda; 05-20-2018 at 05:58 AM. |
05-20-2018, 06:30 AM | #4 |
Connoisseur
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
|
I tested it out and it's not the cloudsync that causes the errors.
I paused cloudsync. Made sure my database was errorfree by deleting the book that caused errors. Checked database with the maintenance tools. Stopped and restarted Calibre. Downloaded the kindle books again in empty directory. Reimported them, changed metadata. Ran check, same errors. As far as I'm aware there's no purging process or other RAID-5 maintenance/scrubbing process going on. I'm running latest calibre version 3.23 Also experienced this recently with google play books and kobo, this weekend ... Anyone else with Synology having/had/solved a similar problem ? Last edited by svda; 05-20-2018 at 08:06 AM. |
05-20-2018, 12:00 PM | #5 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Windows and Linux filesystems has different case sensitivity. Your Synology NAS use a variant of Linux, DSM.
In Windows the files: "BookTitle.epub" and "booktitle.epub" are the same files. In Linux the files: "BookTitle.epub" and "booktitle.epub" are two different files. Same with folder names... So when you copy or move your library and update metadata, fix spelling and title/author leading UPPER/lower spelling, and so on, you may slowly cause increasing corruption in your calibre library. Weird errors with missing formats/files or books and extra duplicate or orphan folders and books. I stopped using Windows and Samba/CIFS to access my calibre library on my Synology NAS and instead used Linux and NFS. Now I have no problems. I only access my books on my NAS from either a laptop or a desktop PC, but never ever simultaneously. I assume other NAS brands have the exact same problem. To fix your problems I suggest that you use the functions in calibre to MOVE your library back to your PC. Then check the folder for the old library for orphan books and add them to your libarary. That way you should not loose any books. You may have duplicates if you fixed "lost" books before... Then use your NAS only to make backups of your calibre library. Or convert to Linux and NFS... There may also be some settings to the Samba/filesystem configuration on the NAS that also might fix this, I never researched that. Errors in calibre due to different case sensitivity in filesystems are often delayed. Meta data changes to files and filenames are not written to calibre library at once, just stored in the database. Later, perhaps after weeks, months or years, when filenames for books are updated for some other reason, some files and or folders may go missing and/or are duplicated seemingly at random. Very strange errors if you don't realize what is going on. |
05-20-2018, 05:26 PM | #6 |
Connoisseur
Posts: 61
Karma: 10
Join Date: Nov 2016
Device: none
|
Hi Adoby, if the problem is case sensitivity, would it be technically possible to force calibre into using a windows or linux case? maybe an option to force everything in the library to use small letters and use up to a certain character count, so you wont have to worry where to use it.
|
05-20-2018, 11:38 PM | #7 |
creator of calibre
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
character case is just one possible problem there are others. Filesystems on linux an windows have fundamentally different semantics.
|
05-21-2018, 05:26 AM | #8 | |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Quote:
It might be possible to add some form of file/folder name normalization to calibre. So only a common subset of characters are used for all file and folder names. For instance only lower case ASCII up to 127. But it might cause problems with existing libraries. I don't know. And there may be other issues as Kovid suggests. For instance allowed (combined) length of paths and file names differ a lot. Could perhaps also be normalized? Is it technically possible for you to stop using calibre in Windows with the library on a NAS with a Linux filesystem? That is all you have to do. Method 1. Have the library on your Windows PC. Use the NAS only for backups. Method 2. Have the library on the NAS. Access it from a Linux install on your PC using NFS. Either a virtual machine or dual boot. Method 3. Stop using Windows. Use only Linux and NFS. I used method 3 and like it! many other benefits as well... Last edited by Adoby; 05-21-2018 at 05:43 AM. |
|
05-21-2018, 06:37 AM | #9 |
Connoisseur
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
|
Thanks Adoby, much appreciated advice. I'm not so into Linux. I chose this setup because it was enormously practical. Access the database from any pc in the house (single user) and instantly backed-up in the cloud where I could access it anywhere from any smartphone/tablet.
So, first moving the db's back to pc and checking integrity. Then some rethinking to do. |
05-21-2018, 06:54 AM | #10 |
Connoisseur
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
|
Found 383 books in my library, 2 formats : 303 azw3 and 77 mobi ... that doesn't add up, does it ?Integrity checks ok says calibre.
|
05-21-2018, 07:08 AM | #11 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
There may be a few orphan books in the library folder. Move the library and see what is left behind.
Method 4. Have your main calibre database on your main windows PC. And a full backup copy of the library on the NAS. You can access the backup copy calibre library from other computers on your network. To avoid problems only update/add/delete/change books from the main PC. This setup means that you can access the main library from your main PC at the same time as someone else access the backup copy. There are many sync-utilities that makes it easy to update the backup copy so it is identical to the main library. I use rsync. |
05-21-2018, 09:25 AM | #12 |
Connoisseur
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
|
I located them in the search with formats:false, while the calibre maintenance tool did not report them.
Off topic, so if I keep my MP3 files on a linux/unix system like my Synology NAS AND perform some internal edits on them based from a windows PC and software, like with MP3tag, I could corrupt my MP3 files also ? Last edited by svda; 05-21-2018 at 09:41 AM. |
05-21-2018, 10:35 AM | #13 | |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Yes. But normally you fix errors as you go. Delete duplicates and so on. And edits are typically made directly in situ. But sometimes you may end up with corrupt or lost files.
calibre is extra sensitive because it stores metadata in the database and does often not update the files at once. It is a form of "lazy" update. Lazy updates makes it possible to do things like update metadata for thousands of ebooks in seconds. The actual ebooks are updated only when you save them or send them to a device. If you used the same type of direct updates as MP3tag does it could take hours to update a bunch of ebooks. And if calibre is run on a Windows computer but store the library on a Linux filesystem the lazy updates may cause errors, as you noticed. That is why the manual EXPLICITLY says that you should not have your library on a networked drive/NAS. https://manual.calibre-ebook.com/faq...rked-drive-nas Quote:
|
|
05-21-2018, 11:00 AM | #14 |
Connoisseur
Posts: 52
Karma: 8426
Join Date: Feb 2013
Location: Belgium
Device: none
|
Ok, Adoby. thank you for explaining it to me.
|
05-22-2018, 01:07 AM | #15 |
Connoisseur
Posts: 61
Karma: 10
Join Date: Nov 2016
Device: none
|
Thank you Kovid and Adoby, I will use the method 4
|
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
how to compare new metadata with old before updating | ahab1972 | Library Management | 7 | 09-24-2015 04:16 AM |
Why is my metadata not updating? | rhino79 | Library Management | 25 | 09-23-2014 01:34 AM |
Metadata not updating? | jstabe | Conversion | 0 | 03-02-2013 05:59 PM |
batch updating metadata | webwizard | Library Management | 5 | 04-15-2011 04:40 PM |
Updating Metadata without and ISBN | herbycanopy | Calibre | 7 | 05-22-2010 01:16 AM |