Problem updating database on Amazon Cloud

[kj_va]

Good afternoon:

I am having a problem with CC 5.1..2 showing the correct and updated number of books in my Calibre database. Earlier today I moved my Calibre database from my home PC to a folder on my Amazon Cloud account. Everything seems in order, and I have no problem adding books to my Calibre program and seeking the folder created on Amazon Cloud. I can see the new Calibre folders being created in the specific Amazon Cloud folder, and Calibre program on the PC correctly shows the newly added books. However, CC is not updating.

Presently CC shows "Book count: 2454" and my book count in Calibre is 2517. I have tried resetting the cloud connection, "change cloud libraries", toggling various settings, reinstalling the app...nothing seems to update the count or show the newest added books. Presently if I select "change cloud libraries" and then select my Kindle Library (database folder in Amazon Cloud), I see the message about it checking to see if the database is up to date, but then when I click on the round info button, it shows that it was last updated a half hour ago and the book count is outdated.

My primary objective is to simply be able to use CC to accurately depict the books in my Calibre database so that I can browse, do searches, view metadata, and sometimes download my phone to either read or email to kindle.

Any thoughts? Thank you.

[chaley]

It seems clear that CC is not seeing the same library that calibre is seeing. Why is a mystery.

First question: is there a reason you haven't updated CC? There have been two releases after 5.1.2.

Second question: are you sure you CC is using the same cloud account as the machine running calibre?

Third question: is there something on the computer that runs calibre that is updating the Amazon cloud account? Some background app? If not, your changes will be local but not show up in the cloud. You can check this by going to the cloud account and verifying the dates on calibre's metadata.db, and checking if the new book folders are there.

As far as I can tell, the official Amazon Cloud Drive app does not do background syncs. At least my copy doesn't. In fact, it doesn't seem to overwrite changed files when doing a manual upload, which makes it worse than useless.

If you send me a CC debug file I can verify what folder CC in Amazon's cloud that CC is checking and the last modification date of the calibre database. See our FAQ How do I get help? for instructions. Please erase the multipie.co.uk To: address.

[kj_va]

I should have written 5.1.4.2

I am using the same and only cloud account & Calibre database I have. Yesterday was the first time that I moved the Calibre database from my PC to Amazon Cloud, so the only location I am using is the folder I created on Amazon Cloud.

Seems like your third question gets to the root of the problem. I set things up so that when I upload a book to my Google Drive acct, the book will sync to a folder on my PC, and I have specified that folder in Calibre as my auto add folder. I thought that since I had moved my Calibre database to Amazon Cloud that when Calibre automatically added a book, everything would be updated in Amazon Cloud so that cc would see the updated db. In looking at my Amazon Cloud folder this morning, I see that the new book folders (for the 63 books not appearing in cc) were created 16 hours ago, but the metadata.db file was updated 21 hours ago.

I have emailed the log file. Thank you for your help.

[chaley]

The debug log says that the library is stored in "Amazon/.Kindle Library". I am not sure if the Amazon is in the path cloud path.

The calibre library is opened 17 times (in this log). The first one was at 2016-08-09 17:13:44.094. This was the first time the library was opened, and the metadata.db file was transferred. The metadata.db file is dated 2016-08-09T18:45:42.526Z

From then on Amazon reports the same modification time for metadata.db. Example: amazon currentRev=2016-08-09T18:45:42.526Z, lastRev=2016-08-09T18:45:42.526Z. I think we can safely say that the metadata.db file isn't being updated, which is consistent with what you saw.

I did a test using the Amazon Cloud client. It refused to modify my metadata.db file, saying that a different file was already there. I manually deleted metadata.db from the cloud then did a sync, and the file updated. That isn't a sustainable process, because (apparently) none of the files in the library will be updated. Of course, you can be sure that the cloud library is up to date if you delete it from the cloud and re-send it in its entirety. Also not a sustainable process.

There are some third-party apps that sync with amazon cloud, such as syncbackpro and goodsync (neither seems to be free). Something called "odrive" claims to sync, but it isn't obvious what it does.

[kj_va]

Thanks. Final question: I see that Calibre on my PC shows the updated number of books (bottom left corner, e.g., [2617 books, 1 selected], and I cannot find any metadata.db file on my PC. So what is Calibre accessing (presumably on Amazon Cloud?) to get the updated info?

If I understand correctly, it looks like everything is up to date in my Amazon Cloud folder except for the metadata.db file, but if I was to use a 3rd party sync program, what would it sync since there is no corresponding metadata.db file on the PC?

[chaley]

There must be a metadata.db file on your pc. If it isn't there, calibre won't start. The file is in your library folder at the top level, the same level where you see the author folders. If you are using Windows then depending on your settings the file might seem to be named "metadata".

As for updating, it appears that the Amazon app is happy to upload new files, but refuses to upload changed files. This means that changes you make to books such as updated covers will not make it to the cloud. In addition, the calibre auto-backup files won't update, rendering them less than useful.

[kj_va]

I am 99.9% certain that there is no metadata.db file on my PC as I have searched multiple times, incl hidden files, etc. Since I moved the library to Amazon Cloud yesterday, there is a single file remaining in my Calibre Library folder ("CalibreGoogleDriveFixer.log").

I think I previously should have mentioned exactly how I used Amazon Cloud: I use NetDrive2 to map my Amazon Could storage as a E:\ drive on my PC, so yesterday I simply moved the library from the local folder to a newly created folder I created on Amazon.

I just now enamed my "Kindle Library" folder on Amazon to "Kindle Library2" and then upon starting Calibre received a "Corrupted database" error message and it would not open (naively thinking that renaming the amazon folder back to the original name would permit Calibre to open normally, but instead the program is undertaking what looks to be a multiple-hour database restoration).

In any event, I think I will probably move the library back to my pc and set it up as a content server for remote connection.

Thanks again for your help/time.

[chaley]

Originally Posted by kj_va

I think I previously should have mentioned exactly how I used Amazon Cloud: I use NetDrive2 to map my Amazon Could storage as a E:\ drive on my PC, so yesterday I simply moved the library from the local folder to a newly created folder I created on Amazon.

Ahh, that means that almost everything I said isn't right. I assumed that you were doing the "normal" thing: that you had a local copy that is synced to the cloud.

First note: calibre specifically does not support network drives. See the calibre FAQ entry here.

Second note: CC depends on the cloud drive provider reporting a different "reference date" for a file so CC can know it changed. In your case I don't know what Amazon will report, or even if Amazon knows that the file changed. If it gives the creation date then CC will never notice the change. Resetting CC's cloud library should force a download but you said you tried that and it didn't happen, so it is possible that Amazon is not serving the modified file over the cloud access interface.

I just now enamed my "Kindle Library" folder on Amazon to "Kindle Library2" and then upon starting Calibre received a "Corrupted database" error message and it would not open (naively thinking that renaming the amazon folder back to the original name would permit Calibre to open normally, but instead the program is undertaking what looks to be a multiple-hour database restoration).

If the metadata.db file went away or was damaged on E:, calibre will indeed offer to rebuild it. It does this by scanning the entire library for "metadata.opf" files that contain the calibre metadata for the books, rebuilding the db from what it finds.

Why the metadata.db file was lost or damaged by the rename is an interesting question.

Things might go a lot faster if you copy the library to a local drive before you run the restore, but given that it has already started ...

In any event, I think I will probably move the library back to my pc and set it up as a content server for remote connection.

Thanks again for your help/time.

What you might consider doing is your own sync. You would indeed keep your library on a local drive. In addition, you would keep the E: access and manually sync your library to that copy. The sync would be done with some file sync program so calibre doesn't see the network drive. The page What’s the best free file synchronization software for Windows? lists several good, free programs. I have used freefilesync, but be aware that it tries to install crapware. I just installed the newest version and it asked me to install something called "registry reviver", but gave me the option to opt out.

It is possible that syncing the library to E: will result in the same problem, that metadata.db doesn't appear to change. You can easily work around that by using a script that deletes the file from E: then does the sync.

[kj_va]

excellent, thank you. I will continue to mess around a bit, looking into your suggestions. It helps now that I know exactly why cc was not showing the updated library.