Problems with library on OneDrive

Joy Phillip · 08-27-2015, 12:06 PM

Well, I'm glad I found this thread, I'm having nearly the same problem

My set though is with OneDrive, and only 3500 books. The books are all there, but I have had multiple copies of the same sets of books and sometimes folders in my library.

I know that there was a problem I was having with Onedrive downloading and syncing the same books over and over again, and Windows (in it's programming genius) added the (1) or (2) or (1) (1) to the book listings, metadata and covers. This had the effect of taking my 9 GB library and making it nearly 40 GB. The folders of the authors which contained copies of some of the book records had a "_" (underscore) added to the end of the author's name, but no similar entry in the library. I have found several recreations of the same book in different folders with different index numbers at the end of the title.

Some of these errors I've been able to fix with a library maintenance scan, but there's thousands of books that were just lost.

I'm replicating my Calibre library on OneDrive, not storing it there. The database runs from the local PC, and I use OneDrive to give me a backup. I also use COPS to replicate it on my website where I log in with COPS to pull down an OPDS version of the library as I sometimes want to pull a book down when I'm away from my home network.

Never used Calibre on two different systems at once.

I think I have the additional books with the replicated (1) etc handled. Got several programs and figured out ways to let Windows find the affected files, deleted them and I *hope* that it replicates up to the OneDrive. Then ran the library scan and found all the files that Calibre thinks doesn't belong, mostly MOBI copies of books already in the library.

So, right now I'm doing a database restore. If this doesn't work, I'm moving the library, readding the books that are "missing" and seeing where I stand from there.

theducks · 08-27-2015, 12:32 PM

If you allow your cloud to work bidirectional, you will cause problems with Calibre.
When you change an Author or Title Metadata, their folders also get changed.

Keystone Kops time with sync . Sync is trying to replace the Lost folder and store the Found folder

Pause Sync of the library folder

BetterRed · 08-27-2015, 05:57 PM

Moderator Notice
@Joy Phillip - Welcome to Mobileread

I put your post and theducks response into a separate thread

As a general guideline we prefer single issue threads

cheers BR

JSWolf · 08-27-2015, 06:45 PM

The solution is not to use cloud storage for your Calibre library. There are way too many issues that can cause problems and or loss of data. it's not recommended to do this and if you do and something goes wrong, it's your problem.

Joy Phillip · 08-27-2015, 07:27 PM

Sorry for the misplaced thread, I was hoping that it was the same basic issue.

That said, I restored the database, that screwed things up, moved the database entirely and re-added the files that were left in the original location, which wasn't as many books as there were folder/author entries.

So I wiped them both out, the library and the leftovers. Closed Calibre. So right now it thinks there is a library where there isn't, and it will create a blank empty library where I told it the actual library is.

I'm pulling the copy of the database and all the books from my website where I have the COPS install. This should be mostly free of errors and should work as a good backup. I"m going to check it for additional copies and leftover files, just to make sure before I add the whole shebang back to Calibre.

What I'm thinking about doing is a search in the place the FTP restore is at to get JUST the book formats, MOBI, EPUB, PDF, TXT and DOC(X) and then drag and drop those into the blank Calibre library to recreate the entries etc. However, I'm afraid I'm going to lose all my metadata, the sorts, the downloaded information from the internet on the various books, the series, the tags and so on. I'm wondering if it would be possible to direct Calibre to look at the metadata that the folder the files came from has, or do I just include that in the drop?

Also, I want to keep a backup of the library. My website is just that, a website and I have an emergency copy for access away from home. What can I do to let OneDrive have a copy of it up on the cloud without this cluster crud happening again?

eschwartz · 08-27-2015, 10:11 PM

Quote:

Originally Posted by JSWolf

The solution is not to use cloud storage for your Calibre library. There are way too many issues that can cause problems and or loss of data. it's not recommended to do this and if you do and something goes wrong, it's your problem.

Oh, you bundle of fun, you.

I have been keeping my libraries in Dropbox for years now, and the way to avoid problems is extraordinarily simple.

@Joy,

All I do is make sure I fully sync before running calibre, pause syncing while calibre is running, and fully sync after calibre is closed.

That avoids the two problems -- edit clashes with other computers, and edit clashes between the Cloud Sync client and calibre itself.

For added security, use a file syncing tool like rsync (unix) or MS SyncToy for Windows -- use it to mirror your main library into your Cloud Storage.

As for importing your current FTP backup, import as per the screenshot, it will derive metadata from the metadata.opf backups (it was designed for this purpose).

AnemicOak · 08-27-2015, 10:36 PM

Quote:

Originally Posted by eschwartz

I have been keeping my libraries in Dropbox for years now, and the way to avoid problems is extraordinarily simple.

Same. Been doing it since DropBox first launched with nary a problem.

Joy Phillip · 08-31-2015, 10:42 PM

Okay, after the FTP restore, I reimported the ENTIRE library, and have been wrestling with it ever since. In most cases it's a matter of it didn't import all the books for one reason or another. For instance, I have author Foo, Bill. In there there are 15 books, all different titles, each labeled with the various formats and metadata in there. I import, and get a folder with Foo, Bill that only has three titles in it, and now I have to go back to the original and get the other titles in the new library.

Other than that, it's little tweaks like adding new covers, making sure the various metadata is still there, and so on. I *think* it got all the authors, and I already talked about the titles.

I zipped up the FTP restore, and will mirror that to the OneDrive. I'm going to download the MS SyncToy and use that to keep a copy of the library in the OneDrive folder on my local PC, and have THAT copy synced to OneDrive. Have to re-upload the whole load to my website again, but that's minor.

Documentation should be developed for just this, if I had seen the problems inherent with having a live library in the folder used to sync to OneDrive, I would NEVER have done it this way.

Dropbox and GoogleDrive don't have the space I need, or I might have used them for this. Plus I have a whole virtual library subset (Prepper and SHTF) that is mirrored to a Facebook group I run, so that's an additional complication. Thankfully I CAN use GoogleDrive for that, so it's less of a hassle.

I'll find a thread for toolbars to get my next thought.

08-27-2015, 12:06 PM	#1
Joy Phillip Member Posts: 11 Karma: 10 Join Date: Aug 2015 Device: Nokia 1020, Kindle Touch	Problems with library on OneDrive Well, I'm glad I found this thread, I'm having nearly the same problem My set though is with OneDrive, and only 3500 books. The books are all there, but I have had multiple copies of the same sets of books and sometimes folders in my library. I know that there was a problem I was having with Onedrive downloading and syncing the same books over and over again, and Windows (in it's programming genius) added the (1) or (2) or (1) (1) to the book listings, metadata and covers. This had the effect of taking my 9 GB library and making it nearly 40 GB. The folders of the authors which contained copies of some of the book records had a "_" (underscore) added to the end of the author's name, but no similar entry in the library. I have found several recreations of the same book in different folders with different index numbers at the end of the title. Some of these errors I've been able to fix with a library maintenance scan, but there's thousands of books that were just lost. I'm replicating my Calibre library on OneDrive, not storing it there. The database runs from the local PC, and I use OneDrive to give me a backup. I also use COPS to replicate it on my website where I log in with COPS to pull down an OPDS version of the library as I sometimes want to pull a book down when I'm away from my home network. Never used Calibre on two different systems at once. I think I have the additional books with the replicated (1) etc handled. Got several programs and figured out ways to let Windows find the affected files, deleted them and I hope that it replicates up to the OneDrive. Then ran the library scan and found all the files that Calibre thinks doesn't belong, mostly MOBI copies of books already in the library. So, right now I'm doing a database restore. If this doesn't work, I'm moving the library, readding the books that are "missing" and seeing where I stand from there.

08-27-2015, 07:27 PM	#5
Joy Phillip Member Posts: 11 Karma: 10 Join Date: Aug 2015 Device: Nokia 1020, Kindle Touch	Sorry for the misplaced thread, I was hoping that it was the same basic issue. That said, I restored the database, that screwed things up, moved the database entirely and re-added the files that were left in the original location, which wasn't as many books as there were folder/author entries. So I wiped them both out, the library and the leftovers. Closed Calibre. So right now it thinks there is a library where there isn't, and it will create a blank empty library where I told it the actual library is. I'm pulling the copy of the database and all the books from my website where I have the COPS install. This should be mostly free of errors and should work as a good backup. I"m going to check it for additional copies and leftover files, just to make sure before I add the whole shebang back to Calibre. What I'm thinking about doing is a search in the place the FTP restore is at to get JUST the book formats, MOBI, EPUB, PDF, TXT and DOC(X) and then drag and drop those into the blank Calibre library to recreate the entries etc. However, I'm afraid I'm going to lose all my metadata, the sorts, the downloaded information from the internet on the various books, the series, the tags and so on. I'm wondering if it would be possible to direct Calibre to look at the metadata that the folder the files came from has, or do I just include that in the drop? Also, I want to keep a backup of the library. My website is just that, a website and I have an emergency copy for access away from home. What can I do to let OneDrive have a copy of it up on the cloud without this cluster crud happening again? Last edited by Joy Phillip; 08-27-2015 at 07:35 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre and OneDrive Windows 10	vaderag	Calibre	5	08-07-2015 11:40 AM
Using Calibre, able to use OneDrive as cloud service?	a_ahmed	Calibre	19	06-30-2015 04:28 AM
Export Caliber Library to Onedrive	Cybermidas	Calibre	23	04-03-2015 02:37 PM
synchronize library Calibre in OneDrive	beatriz	Devices	3	03-05-2015 12:20 PM
Calibre on Win 8.1 Desktop not working with Onedrive online-only files?	robcohen	Related Tools	1	12-23-2014 02:54 AM

08-27-2015, 12:32 PM	#2
theducks Well trained by Cats Posts: 29,820 Karma: 54830978 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	If you allow your cloud to work bidirectional, you will cause problems with Calibre. When you change an Author or Title Metadata, their folders also get changed. Keystone Kops time with sync . Sync is trying to replace the Lost folder and store the Found folder Pause Sync of the library folder

08-27-2015, 05:57 PM	#3
BetterRed null operator (he/him) Posts: 20,590 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	Moderator Notice @Joy Phillip - Welcome to Mobileread I put your post and theducks response into a separate thread As a general guideline we prefer single issue threads cheers BR

08-27-2015, 06:45 PM	#4
JSWolf Resident Curmudgeon Posts: 74,049 Karma: 129333562 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	The solution is not to use cloud storage for your Calibre library. There are way too many issues that can cause problems and or loss of data. it's not recommended to do this and if you do and something goes wrong, it's your problem.

08-31-2015, 10:42 PM	#8
Joy Phillip Member Posts: 11 Karma: 10 Join Date: Aug 2015 Device: Nokia 1020, Kindle Touch	Okay, after the FTP restore, I reimported the ENTIRE library, and have been wrestling with it ever since. In most cases it's a matter of it didn't import all the books for one reason or another. For instance, I have author Foo, Bill. In there there are 15 books, all different titles, each labeled with the various formats and metadata in there. I import, and get a folder with Foo, Bill that only has three titles in it, and now I have to go back to the original and get the other titles in the new library. Other than that, it's little tweaks like adding new covers, making sure the various metadata is still there, and so on. I think it got all the authors, and I already talked about the titles. I zipped up the FTP restore, and will mirror that to the OneDrive. I'm going to download the MS SyncToy and use that to keep a copy of the library in the OneDrive folder on my local PC, and have THAT copy synced to OneDrive. Have to re-upload the whole load to my website again, but that's minor. Documentation should be developed for just this, if I had seen the problems inherent with having a live library in the folder used to sync to OneDrive, I would NEVER have done it this way. Dropbox and GoogleDrive don't have the space I need, or I might have used them for this. Plus I have a whole virtual library subset (Prepper and SHTF) that is mirrored to a Facebook group I run, so that's an additional complication. Thankfully I CAN use GoogleDrive for that, so it's less of a hassle. I'll find a thread for toolbars to get my next thought.

Advert

Advert