Storing the Calibre library in "Documents Folder" on iCloud

HugoGernsback · 10-05-2018, 12:16 PM

I would like to move my Calibre library to my documents folder which is actually in Apple's iCloud. This is for automatic backup/protection, not for accessing the library from multiple devices.

In Switch/Create Library I see that the iCloud resident folders (i.e. the Desktop and Documents) are not visible. I assume that this is because iCloud is considered to be a network-attached storage device and there are a bunch of reasons in the FAQ why NAS is discouraged. My use case however is simple.

Calibre is my "library of record". I maintain all my books in Calibre and export them to iBooks as ePubs. This gives me access across all my devices. I don't maintain my books in iBooks because I don't trust it on a long-term basis. Apple over the years has made user-unfriendly changes to iBooks that make it an unreliable archive for books. But it's fine for reading. Having Calibre in my documents directory would provide me with continuous backup which is valuable.

Is there any way to move my Calibre library to my documents folder? I am running macOS Mojave (10.14) and Calibre 3.32.

Thanks.

gbm · 10-05-2018, 01:31 PM

Quote:

Originally Posted by HugoGernsback

I would like to move my Calibre library to my documents folder which is actually in Apple's iCloud. This is for automatic backup/protection, not for accessing the library from multiple devices.

In Switch/Create Library I see that the iCloud resident folders (i.e. the Desktop and Documents) are not visible. I assume that this is because iCloud is considered to be a network-attached storage device and there are a bunch of reasons in the FAQ why NAS is discouraged. My use case however is simple.

Calibre is my "library of record". I maintain all my books in Calibre and export them to iBooks as ePubs. This gives me access across all my devices. I don't maintain my books in iBooks because I don't trust it on a long-term basis. Apple over the years has made user-unfriendly changes to iBooks that make it an unreliable archive for books. But it's fine for reading. Having Calibre in my documents directory would provide me with continuous backup which is valuable.

Is there any way to move my Calibre library to my documents folder? I am running macOS Mojave (10.14) and Calibre 3.32.

Thanks.

Please read this:

I am getting errors with my calibre library on a networked drive/NAS?

iCloud in just a slower version of an NAS.

bernie

HugoGernsback · 10-05-2018, 11:40 PM

Quote:

Originally Posted by gbm

Please read this:

I am getting errors with my calibre library on a networked drive/NAS?

iCloud in just a slower version of an NAS.

bernie

I don't think it's exactly the same scenario as a NAS. The way iCloud is implemented on the Mac is that your documents are stored locally and synchronized in the background with the copies in iCloud. So applications aren't actually trying to read or write to remote files, they are reading and writing local files that a background process then copies up to the cloud. Think of it as similar to Carbonite or any other cloud-based backup solution.

Anyhow, my bad. I didn't notice that the "Choose your calibre library" actually does let me access the folders protected by iCloud. I just couldn't reach them from my user home directory.

I just pointed Calibre to my Documents directory (which is protected by iCloud) and everything is working fine.

DNSB · 10-06-2018, 12:50 AM

Quote:

Originally Posted by HugoGernsback

I don't think it's exactly the same scenario as a NAS. The way iCloud is implemented on the Mac is that your documents are stored locally and synchronized in the background with the copies in iCloud. So applications aren't actually trying to read or write to remote files, they are reading and writing local files that a background process then copies up to the cloud. Think of it as similar to Carbonite or any other cloud-based backup solution.

Perhaps think of it as a cloud backup solution that can and will take a backup copy of an in-use file. Unless your applications are designed for this type of use, it can end up with your files resembling scrambled eggs. Not a "feature" of iCloud alone -- Google Drive and Dropbox also have this miscapability.

HarryT · 10-06-2018, 03:19 AM

Quote:

Originally Posted by HugoGernsback

I don't think it's exactly the same scenario as a NAS. The way iCloud is implemented on the Mac is that your documents are stored locally and synchronized in the background with the copies in iCloud. So applications aren't actually trying to read or write to remote files, they are reading and writing local files that a background process then copies up to the cloud. Think of it as similar to Carbonite or any other cloud-based backup solution.

The point is, however, that this background backup process will be trying to backup open files and that will inevitably lead to data corruption. Don’t do it!

theducks · 10-06-2018, 11:49 AM

What Harry said.
(Background) Delays are bad.
What if Calibre does 2 consecutive near operations on a book file? And one of those takes longer to propagate to the cloud? What if something on your network also dumps a huge file to the 'cloud' at the same time?

Building in all the necessary features to make everything 'Transactionally safe' would severely impact speed (And I see howls over current HD limited speeds here all the time. It took 10 seconds

to transfer a 100 books to my X !)

AnemicOak · 10-06-2018, 12:23 PM

The only relatively safe way to do this would be if you could pause iCloud's syncing while Calibre is in use, but to the best of my knowledge it can't.

I have my library in DropBox and have for many years, but am always careful that the DropBox client is paused or shut down completely before Calibre is ever opened and in use.

phossler · 10-06-2018, 12:41 PM

My 2 cents …

1. I leave the Calibre DB on the PC HD alone -- no File Explorer, no NAS, No share, no nuthing

2. I do a normal RoboCopy backup to a local external HD - just in case -- each night, with a monthly to a disconnected external HD - just in case

3. I do a calibredb export (with options) to put a copy of just my epub and azw3 files in another flat folder. I use THAT folder to sideload my books onto my Kindle or other.

Now, if you keep the real Calibre DB on the local HD and just used iCloud (or whatever) only as a non-synced one-way (Mac Calibre DB --> Copy) repository of a backup copy of the real Calibre DB and refreshed the copy periodically, I'd think you'd be safe since you're not syncing.

I'm guessing that there's a way to control what iCloud folders sync. I know there is using Google Drive

You could maybe even run a copy of Calibre using the iCloud copy. It'll be out of date until you manually run the update.

Maybe some of the experts will jump in (heck -- I know they will)

stumped · 10-06-2018, 01:15 PM

As others have already said, keep Master library on local drive, store a library copy in cloud of your choice for backup and for 24\7 access.
Run a mirror operation a.k.a. a one way sync of local to cloud at whatever frequency suits you and with your preferred utility. I use free file sync and Dropbox.

HarryT · 10-06-2018, 01:18 PM

I use "SyncToy", which is also free. Make sure that you exit Calibre before running the sync operation!

stumped · 10-06-2018, 01:30 PM

Quote:

Originally Posted by HarryT

I use "SyncToy", which is also free. Make sure that you exit Calibre before running the sync operation!

TBH , I have been doing this for years and never bother to exit calibre before syncing
the incremental sync only takes a minute or so. I just avoid doing it while calibre has any active jobs.
it would definitely be best practice to exit calibre like you say, just saying you'll probably get away with not bothering to. Also, the cloud version is the backup, so any error should get fixed at the next sync.
I actually run a double sync, as I really don't want to ever lose my library, so it is mirrored to a 2nd local data drive as well as to dropbox
I have a couple of times retrieved a file after messing up an edit or convert.

I run my sync manually as needed; its probably not that hard to create a batch program that will close calibre, run sync, open calibre again but I don't know how you'd program step one of that - close the active calibre instance.

i do frequently access the dropbox copy to add a book to a tablet with calibre companion, when the library PC is off

stan999 · 03-02-2020, 05:36 AM

what if I sync one way from local calibre library to iCloud and then I use iCloud to open a book and make some highlights. I would obviously want that book synced back to the local calibre library to reflect the new highlights, bookmarks, table of content additions, etc. Will this corrupt calibre library? Let's say I would make these changes in the local calibre library, would there be a metadata change in calibre simply making some highlights and simple bookmarking using a third-party app?

stumped · 03-02-2020, 05:59 AM

Quote:

Originally Posted by stan999

what if I sync one way from local calibre library to iCloud and then I use iCloud to open a book and make some highlights. I would obviously want that book synced back to the local calibre library to reflect the new highlights, bookmarks, table of content additions, etc. Will this corrupt calibre library??

1. for that to work you have to turn on 2 way sync
2. and thus risk corrupting both the local and the cloud

your library, your choice, your funeral if /when it goes wrong.
only you can decide whether the convenience outweighs the risk
but expect an awful lot of no sympathy and we told you so, when you come back to ask for help fixing the damage

calibre is not designed to work in cloud storage

PS and your decision, in another thread , to get rid of all your .opf files ( just to save a handful of bytes? ) removes a damage repair option

stan999 · 03-02-2020, 07:29 AM

Well, that's why I'm asking. I'm aware of the issues. I'm making some technical judgments here. If a change inside the local library would not change metadata (as in open pdf with pdf expert, make highlights...close db) then I wouldn't see what the problem of doing a two sync be. And I now understand the main function of .opf files. ll keep them there even though I usually back up the library every day.

stumped · 03-02-2020, 07:37 AM

No one can assess the risk for you. I've lost count of the number of " help me troubleshoot this strange bug " threads where, about 20 posts in, with all contributors scratching their heads, the OP admits they were using a NAS or a cloud library.

There have also been cases reported here where " it worked perfectly for years" until one day it did not.
you are not in control of what background activites the cloud software gets up to, because it can be making its own silent backups and thus temp locking, unlocking critical files.

I would be guided by the program author's advice, but its up to you....

I often edit a book, using sigil, operating directly on the file stored in the calibre database, but that is all local, and I am the only user, and I have a backup. Been doing it for years, never any issues... [ there you go, that should bring down a curse any day now ]

10-05-2018, 12:16 PM	#1
HugoGernsback Junior Member Posts: 2 Karma: 10 Join Date: Oct 2018 Device: iBooks on iOS	Storing the Calibre library in "Documents Folder" on iCloud I would like to move my Calibre library to my documents folder which is actually in Apple's iCloud. This is for automatic backup/protection, not for accessing the library from multiple devices. In Switch/Create Library I see that the iCloud resident folders (i.e. the Desktop and Documents) are not visible. I assume that this is because iCloud is considered to be a network-attached storage device and there are a bunch of reasons in the FAQ why NAS is discouraged. My use case however is simple. Calibre is my "library of record". I maintain all my books in Calibre and export them to iBooks as ePubs. This gives me access across all my devices. I don't maintain my books in iBooks because I don't trust it on a long-term basis. Apple over the years has made user-unfriendly changes to iBooks that make it an unreliable archive for books. But it's fine for reading. Having Calibre in my documents directory would provide me with continuous backup which is valuable. Is there any way to move my Calibre library to my documents folder? I am running macOS Mojave (10.14) and Calibre 3.32. Thanks.

10-06-2018, 12:41 PM	#8
phossler Wizard Posts: 1,071 Karma: 412718 Join Date: Jan 2009 Location: Valley Forge, PA, USA Device: Kindle Paperwhite	My 2 cents … 1. I leave the Calibre DB on the PC HD alone -- no File Explorer, no NAS, No share, no nuthing 2. I do a normal RoboCopy backup to a local external HD - just in case -- each night, with a monthly to a disconnected external HD - just in case 3. I do a calibredb export (with options) to put a copy of just my epub and azw3 files in another flat folder. I use THAT folder to sideload my books onto my Kindle or other. Now, if you keep the real Calibre DB on the local HD and just used iCloud (or whatever) only as a non-synced one-way (Mac Calibre DB --> Copy) repository of a backup copy of the real Calibre DB and refreshed the copy periodically, I'd think you'd be safe since you're not syncing. I'm guessing that there's a way to control what iCloud folders sync. I know there is using Google Drive You could maybe even run a copy of Calibre using the iCloud copy. It'll be out of date until you manually run the update. Maybe some of the experts will jump in (heck -- I know they will) Last edited by phossler; 10-07-2018 at 09:15 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Why is my PC Calibre Library Folder a "device"?	KetchupKid	Devices	6	11-06-2015 12:13 PM
OK to delete "Calibre Library deleted" folder?	LaGringa	Library Management	3	10-26-2013 02:19 PM
Library keeps defaulting to "Calibre Library" folder	Coronach	Library Management	5	09-22-2012 01:36 PM
e-Books show in "My Computer" Kindle 3 documents folder	tcreek	Calibre	1	09-03-2010 10:34 AM
Sub folder in "Personal Documents"	CoolDragon	iRex	12	04-06-2010 09:26 AM

10-06-2018, 11:49 AM	#6
theducks Well trained by Cats Posts: 29,782 Karma: 54830978 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	What Harry said. (Background) Delays are bad. What if Calibre does 2 consecutive near operations on a book file? And one of those takes longer to propagate to the cloud? What if something on your network also dumps a huge file to the 'cloud' at the same time? Building in all the necessary features to make everything 'Transactionally safe' would severely impact speed (And I see howls over current HD limited speeds here all the time. It took 10 seconds to transfer a 100 books to my X !)

10-06-2018, 12:23 PM	#7
AnemicOak Bookaholic Posts: 14,391 Karma: 54969924 Join Date: Oct 2007 Location: Minnesota Device: iPad Mini 4, AuraHD, iPhone XR +	The only relatively safe way to do this would be if you could pause iCloud's syncing while Calibre is in use, but to the best of my knowledge it can't. I have my library in DropBox and have for many years, but am always careful that the DropBox client is paused or shut down completely before Calibre is ever opened and in use.

10-06-2018, 01:15 PM	#9
stumped Wizard Posts: 3,305 Karma: 10259306 Join Date: May 2016 Device: kobo forma, Kobo Libra, Huawei media Tab, fire HD10, PW3 HDX8.9,	As others have already said, keep Master library on local drive, store a library copy in cloud of your choice for backup and for 24\7 access. Run a mirror operation a.k.a. a one way sync of local to cloud at whatever frequency suits you and with your preferred utility. I use free file sync and Dropbox.

10-06-2018, 01:18 PM	#10
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	I use "SyncToy", which is also free. Make sure that you exit Calibre before running the sync operation!

03-02-2020, 05:36 AM	#12
stan999 Member Posts: 20 Karma: 24 Join Date: Jul 2019 Device: none	what if I sync one way from local calibre library to iCloud and then I use iCloud to open a book and make some highlights. I would obviously want that book synced back to the local calibre library to reflect the new highlights, bookmarks, table of content additions, etc. Will this corrupt calibre library? Let's say I would make these changes in the local calibre library, would there be a metadata change in calibre simply making some highlights and simple bookmarking using a third-party app?

03-02-2020, 07:29 AM	#14
stan999 Member Posts: 20 Karma: 24 Join Date: Jul 2019 Device: none	Well, that's why I'm asking. I'm aware of the issues. I'm making some technical judgments here. If a change inside the local library would not change metadata (as in open pdf with pdf expert, make highlights...close db) then I wouldn't see what the problem of doing a two sync be. And I now understand the main function of .opf files. ll keep them there even though I usually back up the library every day.

03-02-2020, 07:37 AM	#15
stumped Wizard Posts: 3,305 Karma: 10259306 Join Date: May 2016 Device: kobo forma, Kobo Libra, Huawei media Tab, fire HD10, PW3 HDX8.9,	No one can assess the risk for you. I've lost count of the number of " help me troubleshoot this strange bug " threads where, about 20 posts in, with all contributors scratching their heads, the OP admits they were using a NAS or a cloud library. There have also been cases reported here where " it worked perfectly for years" until one day it did not. you are not in control of what background activites the cloud software gets up to, because it can be making its own silent backups and thus temp locking, unlocking critical files. I would be guided by the program author's advice, but its up to you.... I often edit a book, using sigil, operating directly on the file stored in the calibre database, but that is all local, and I am the only user, and I have a backup. Been doing it for years, never any issues... [ there you go, that should bring down a curse any day now ]

Advert

Advert