Calibre Network Request

[CRussel]

For what it's worth, I run Calibre against a shared network drive all the time. But I'm VERY careful to always close Calibre on Machine A before opening it on Machine B. My Calibre library is on a network share from a domain joined Windows Storage Server Essentials box, with permissions set to full for everyone.

This would be a total disaster if anyone tried to access the share except me, and if I got careless and forgot to close Calibre. (I know, cause I've had to rebuild it when I forgot!) That being said, it is useable for my very carefully controlled environment.

[Adoby]

I also run calibre from a shared network filesystem, without any problems.

The library is on a Synology NAS and it is shared using NFS. Not SMB! Has been running fine for some years now.

I only access the library from one desktop computer, running Ubuntu.

I also access the NAS using SMB from other computers and devices, but NEVER to use the calibre library.

[cbits]

Well the last couple of Posts are re-assuring, because I installed Calibre a couple of months ago, and automatically put the library onto a QNap mirrored NAS Drive (all data goes onto NAS for security, backup and sharing), I've not used it much but it seems good, Yesterday my daughter needed a book for school work so I downloaded a copy, I had to do a calibre update as well so did little more. I pointed my Daughter to Kovids tutorial to watch with the intention this morning of loading Calibre (or hopefully perhaps a client version) onto her PC So she could add books, help maintain the Library and take stuff out to read on her Tablet or Phone
I was amazed therefore to find that this is seemingly not possible.., why?, I manage to keep a Video Library on the NAS & share it using Media Bowser, a music library (used to be Squeezeserver but recently abandoned that, now just a files and folder structure that works with all the Media Players I have tried), Documents, accessed variously by MS Office and Libre Office, surely a BOOKS LIBRARY is by definition a shared resource!

I would like to be able to have one data source that I can access and manipulate from networked devices, and load copies on and off those devices to read, I don't need an internet content server as my remote systems simply VPN in and can access the network effectively from the "Lan".

Is this what is being developed ? I would be pleased to help in any way but not being a coder do not have much of the skills you might need, do you have any likely time scale?

[itimpi]

The problem appears to be that not all NAS devices support all the semantics required to access the files across the network under all scenarios. For this reason it is not officially supported although you may find it works with a particular hardware/software combination. The issue seems to apply in particular to handling locking of database records across a LAN.

I have successfully run of a variety of NAS devices for some time by having the main library held on a mapped drive (I am using Calibre under Windows) and holding the database itself on a local drive. Holding the database on a local drive (SSD based) also helps with performance of Calibre. I then have a job that periodically syncs the local copy of the Calibre metadata.db file to the correct location in the Calibre library on the NAS so that the NAS copy is self-contained in the event of problems with my PC.

Whether this is likely to change when (if) Calibre finally gets its database layer able to support multi-user access I have no idea.

[cbits]

Hi
Since my Last Post here I have re-jigged the system,:
Now I have Calibre setup at work running on my Desktop PC, but the library itself is stored on a Netgear NAS (our business unit! - my QNap is awaiting larger Disks)

I am not happy about giving the access to my PC at work that this necessitates but until Calibre can be run safely on a NAS it has to be that way. (our Nas units are all set to simply "mirror", (2 Disks) the work data is copied to USB drive for Backup, and the Data, (Movies / Music) on my QNap is all duplicated onto a D-Link NAS at home)

The Calibre has the Apprentice Alf Plugins for the reasons explained by "Alf". (You can get the plugins here too)

This allows us to convert books to whatever format we like, download to the device of choice, my daughter likes to use google books on her Nexus, I use Pageturner on Android Tablet and have it on my phone (Not that I read on the phone), and I run it under Bluestacks Android emulator on my Laptop.

This allowed me, for instance to buy a book from Amazon on day of release, install it into Calibre, so my daughter could read it with google books (Google did not have the book available on the day!)

This may be breaking DRM rules, but hey, we are buying the book, putting on the Bookshelf (the physical ones at home are overflowing anyway!), taking it with us when we go out (without the risk of losing/damaging it), so what's different about that to hardcopy?

The advantage I find with e-readers is that you can use them on a tablet at night, without complaints from partners about lights being on. Of course reading in the bath / pool is much more damaging if you drop it!

This works for us, hope it's of help to you!

[BLinz]

I also use Calibre Portable - Calibre and the Calibre Library are stored at a shared Dropbox-Folder.

I wrote a tiny "Calibre Multiuser Starter" Tool.

I did not start the Calibre-Portabe.exe, i use my "CalibreMultiuserStarter.exe" wich is placed in the same folder.

At Startup the tools Create a little "Lock"-file.
The fle is small and quickly uploaded by Dropbox and other Cloud Services.

If the Tool start an find a Lock-File the use will get an Error Massage that "User xy on Computer zz" have already start Calibre an quit.

If the use close the Calibre exe my Tool will delete the Lockfile and quit also.

Here the Link and the Source-Code in AutoIt:

http://znil.net/index.php?title=Cali...ool_/_Funktion

I also wrote a Version for Terminal-Server (catch the the Computer Name of Client) if anyone want to have.

Bernhard.

[erwin1234]

Originally Posted by BLinz

I also use Calibre Portable - Calibre and the Calibre Library are stored at a shared Dropbox-Folder.

I wrote a tiny "Calibre Multiuser Starter" Tool.

That looks very usable for the share-via-dropbox scenario. However, you have to make sure that dropbox files are synced all the time. Really sure.. That can be tricky. The database can easility be corrupted if one computer "forgets" to sync. For example when you close your laptop and it goes into standby without dropbox completing the sync.

Also this is ofcourse not a solution for the issues that we encounter using a true networked database.

But still a usefull script. Thanks.

[Katsunami]

Originally Posted by erwin1234

Yes ofcourse I have seen this link. Multiple times, you are posting it all the time. That does not mean it makes sense. As a windows programmer I just do not understand why you write "Most network filesystems lack various filesystem features that calibre uses". Most corporate IT systems ONLY use networked filesystems! They are running their databases over network filesystems all the time. With redundancy and file locking, realtime version control, cloud backups, database transactions and everything what not!

Are they also running those databases, using 15 different versions of the same program, some on a Mac, some on Windows XP, Vista, 7, 8, some on Linux, with some databases on Windows servers, others on Linux servers running EXT4, others running Whatever file system... all at the same time? I think they aren't.

If Kovid would officially state that the Calibre library can be put onto a server or a NAS, he would have to support it. That means he would have to make 100% sure that it works, whatever OS the NAS or server runs, and whatever filesystem it uses, and it must work even if there are 3 different versions of Calibre accessing the same database at the same time, coming from 3 different operating systems.

In short:

Yes, the Calibre DB on a network could be done, probably.

No, ït won't be done, because it will be unsupportable. The permutations of different calibre versions, different operating systems (client and NAS/Server) and different file systems is just way too large.

It's just impossible to guarantee 100% that it will work in all possible scenario's, so the official statement is "It doesn't work".

[BLinz]

Originally Posted by erwin1234

... However, you have to make sure that dropbox files are synced all the time. Really sure.. That can be tricky. The database can easility be corrupted if one computer "forgets" to sync. For example when you close your laptop and it goes into standby without dropbox completing the sync...

You are right - but hey, it's Dropbox! and as Long Calibre ist running, the Lock-File is still there / not deleted.
All other User/Clients will get a "Sorry, Calibre is already running by User XY".

It's only a simple solutiuon for running a Calibre-Library on multiple Clients for one Person or for Sharing it over a cloud-service for 2 or 3 Users.

The bigger problem is if you add many ebooks and Change then the metadata - if the DropBox-Process is just sync the Folder with this ebook you get an error (but you can try again). So when i do big changes i stop the DropBox Sync (but the other can't start because the Lock file is there).

BLinz

[blotosmetek]

Originally Posted by erwin1234

calibre, version 0.9.40

Traceback (most recent call last):
File "site-packages\calibre\gui2\metadata\basic_widgets.py", line 102, in commit
File "site-packages\calibre\library\database2.py", line 2554, in set_title
File "site-packages\calibre\library\database2.py", line 653, in set_path
File "site-packages\calibre\library\database2.py", line 1529, in add_format
File "site-packages\calibre\library\database2.py", line 1375, in copy_format_to
File "site-packages\calibre\utils\filenames.py", line 348, in copy_path_to
File "site-packages\calibre\startup.py", line 156, in local_open
OSError: [Errno 13] Permission denied

Arggg this is driving me nuts. All persmissions are full control for everyone on the network!

Erwin, I have run into the same problem. Googling for the solution, I found this thread.

Our setup: Calibre is installed on my wife's Windows 7 PC, but data are held on our Linux (Debian) server, and accessible over Samba. This worked flawlessly for years, but yesterday I have upgraded the server, and Samba version has changed from 3.6.6 to 4.1.3. Since then, Calibre produces exactly the same error message as quoted above.

I believe all you people who seem to have this problem, like the person that filed this bug report: https://bugs.launchpad.net/calibre/+bug/1096737
just were running a newer version of Samba from the very beginning. I would also venture to guess that something has changed in Samba 4.x compared to 3.x to cause this, but I have no idea what that could be - with the exception of Calibre, everything seems to work as usual.

If someone with deeper understanding of network filesystems than me could shed some light on this, I would be extremely grateful.

[kenmac999]

a poor man's hack around this might be to install calibre on each client and ensure library location is consistance, ie c:\mylibrary then creat a script would test for a network temp.lck file and if not found create it, copy the network copy of your calibre library to local location then run calibre. have script delete the temp.lck file after calibre is shut down. I do not have home network to test this on but have used similiar scripts in linux to automate backups of my libraries.

[erwin1234]

Better a late reply then no reply. Ok I'll bite.

Originally Posted by Katsunami

Are they also running those databases, using 15 different versions of the same program, some on a Mac, some on Windows XP, Vista, 7, 8, some on Linux, with some databases on Windows servers, others on Linux servers running EXT4, others running Whatever file system... all at the same time? I think they aren't.

No ofcourse not and that is not what I am asking. You are trying to make my question sound silly. I am asking for running the same database with the same computer with the same windows os version and the same instance of the same program but with the files on a network drive in stead of a local drive. Thats all.

Originally Posted by Katsunami

If Kovid would officially state that the Calibre library can be put onto a server or a NAS, he would have to support it. That means he would have to make 100% sure that it works, whatever OS the NAS or server runs, and whatever filesystem it uses, and it must work even if there are 3 different versions of Calibre accessing the same database at the same time, coming from 3 different operating systems.

I was just asking why it would not work and was offering help.

Originally Posted by Katsunami

In short:
Yes, the Calibre DB on a network could be done, probably.
No, ït won't be done, because it will be unsupportable. The permutations of different calibre versions, different operating systems (client and NAS/Server) and different file systems is just way too large.

Sorry I did not check you out but are you actually involved in calibre programming yourself, or software development on Win-OS in general? if yes could you then answer my question why this would not work on just windows (remember to take everything the same except the filesystem)?

[erwin1234]

Originally Posted by kenmac999

a poor man's hack around this might be to install calibre on each client and ensure library location is consistance, ie c:\mylibrary then creat a script would test for a network temp.lck file and if not found create it, copy the network copy of your calibre library to local location then run calibre. have script delete the temp.lck file after calibre is shut down. I do not have home network to test this on but have used similiar scripts in linux to automate backups of my libraries.

I do not know if this may work in all scenario's. I succesfully run calibre over the network but are making sure the user has read-only permissions. This works.

If in read-write mode, the problem starts when changing metadata because not only the database file is updated, also files and folders are moved, changed, renamed etc. The funny thing is that this does not work correctly over the network because of the way calibre does its file access. You would think simple file moves/copy/deletes etc are fairly standard operations over the network but this is not the case for calibre. It only works correctly on the local filesystem not on the (windows) network. This cannot easily be solved in the current implementation.

So when I modify the database I run calibre on the server itself using remote desktop and for the rest the family has read-only access.

Your suggestion may work but you have to copy ALL files and folders to a local drive, do your changes and then move the complete database structure (all files/folders) back to the network drive. With a large library this may take some time. So in principle, yes this may work but maybe it is not that practical for a large library

[theducks]

erwin
Timing, caching and locking are the biggest banes to a shared network

(this assumes a caseless solution to OS filenames will be implemented)

Any change requires a full lock on (server side) for all affected records AND FILES/Folders

All local caches of data for those must be flushed and Locked, (then refreshed at the end of the change transaction)

BTW the locking system needs to be OS independent

[eschwartz]

Originally Posted by erwin1234

Your suggestion may work but you have to copy ALL files and folders to a local drive, do your changes and then move the complete database structure (all files/folders) back to the network drive. With a large library this may take some time. So in principle, yes this may work but maybe it is not that practical for a large library

Lots of people doing multiple-computer-access sync the files, so only the changes need to be copied over.