Calibre Network Request

[erwin1234]

Good to know you are working on a true multi user support. That is a big step! I would already be happy with a much smaller step.

My family really love calibre. Whith all those devices in the house, PC's, laptops, tablets, smartphones, ereaders and what more I decided to store all our ebook data on our main W7 server computer. And let all family members help with maintaining the database, store their books, sync their devices. And have an automated backup each night for safety.

But bummer... calibre does not work that well in this scenario. Even if you mount the library via a network drive letter between two Windows 7 computers running the NTFS filesystems it seems calibre has problems and will give you strange windows errors (no access, cannot write..). Note this is not a true multi user configuration - it is not working for a single user using a mapped network drive! I have never seen problems like this with other software. Should file operations like open/read/write/rename not be just transparant over the local network? Are there bugs or limitations in the code?

Users requested this scenario to work since 2010/2011. Before doing the full fledged multiuser implementation, would it be possible to first fix the one user/multiple computers/networked database scenario? I did not look at the code yet, but how much work do you think would this take?

I know file locking over network drives can be tedious. But you could use other mechanisms like creating temp .lock files to make sure there is only one instance running. Like e.g. the sync program freefilesyn does. Or forget about the lock at all and let the user decide how to run it. Just make sure the open/read/write/rename works on a network drive.

What do you think?

Oh and ofcourse thanks for this excellent software !

[kovidgoyal]

http://manual.calibre-ebook.com/faq....rked-drive-nas

[DoctorOhh]

Welocome to Mobileread.

Moderator Notice
Moved post to its own thread. Please read the "PLEASE READ BEFORE POSTING - The purpose of this subforum" sticky post for guidelines for posting in the development forum.

Here is the current FAQ entry on setting up calibre to run from a network.

[erwin1234]

Originally Posted by kovidgoyal

http://manual.calibre-ebook.com/faq....rked-drive-nas

Yes ofcourse I have seen this link. Multiple times, you are posting it all the time. That does not mean it makes sense. As a windows programmer I just do not understand why you write "Most network filesystems lack various filesystem features that calibre uses". Most corporate IT systems ONLY use networked filesystems! They are running their databases over network filesystems all the time. With redundancy and file locking, realtime version control, cloud backups, database transactions and everything what not!

So could you please be so kind to explain your statement "Most network filesystems lack various filesystem features that calibre uses" so we can fix the sourcecode accordingly.

Thank you, just trying to help.

[erwin1234]

Originally Posted by DoctorOhh

Welocome to Mobileread.

Moved post to its own thread. Please read the "PLEASE READ BEFORE POSTING - The purpose of this subforum" sticky post for guidelines for posting in the development forum.
Here is the current FAQ entry on setting up calibre to run from a network.

Thank you for making this a seperate thread.

The FAQ does not make sense because the answer to the question "on setting up calibre to run from a network" is smiply "Do not put your calibre library on a networked drive". That is not an answer !

The content server solution is not an option. The goal is to have ONE user at the time working with the full calibre software using the DB over the network. So multiple user but NO concurrency - only one user at the time. But from different locations.

Solution 1: All modern network filesystems have a file locking option - if you open a Word file over the network and another user opens the same file you get a dialog box with "file is already open". This could be a simpe check in calibre? http://en.wikipedia.org/wiki/File_locking

Solution 2: Alternatively you could just create a temp filename "metadata.db.locked" at startup and check for the presence of this file at startup and open it for read/write. If you succeed, you can continue. If not, it is already opened in another instance, so inform the user and quit. The normal program exit would remove this .lock file. But in case of a crash it still works because the file will be closed after some time by the OS.

[theducks]

Originally Posted by erwin1234

Thank you for making this a seperate thread.

The FAQ does not make sense because the answer to the question "on setting up calibre to run from a network" is smiply "Do not put your calibre library on a networked drive". That is not an answer !

The content server solution is not an option. The goal is to have ONE user at the time working with the full calibre software using the DB over the network. So multiple user but NO concurrency - only one user at the time. But from different locations.

Solution 1: All modern network filesystems have a file locking option - if you open a Word file over the network and another user opens the same file you get a dialog box with "file is already open". This could be a simpe check in calibre? http://en.wikipedia.org/wiki/File_locking

Solution 2: Alternatively you could just create a temp filename "metadata.db.locked" at startup and check for the presence of this file at startup and open it for read/write. If you succeed, you can continue. If not, it is already opened in another instance, so inform the user and quit. The normal program exit would remove this .lock file. But in case of a crash it still works because the file will be closed after some time by the OS.

Calibre works with more than just the metadata.DB.
Calibre also does a 'Transaction' with the file system (the contents of the Book Folders).

You change an Author or Title, The Author or Title folders get Renamed/Moved.
You change the cover, and the Cover file gets updated
You change any Metadata, and Metadata.OPF (the backup) gets updated

It is not as Simple as 'Let the OS locking do it'.
Caching created delays alone will wreak havoc if any file is delayed and another Calibre instance makes a request before the 'transaction' has reached stability.

Timing is everything.

Kovid is doing a re-write to accomplish Multi User.
Again, Timing.
It will only be released when it is working perfectly Right

[kovidgoyal]

Again, do not put your calibre library on a network drive. File locking is just one problem, others include data integrity across operating systems and most importantly, the fact that various filesystem APIs are BROKEN on network file systems, and when I say BROKEN I mean BROKEN, as in they LIE about what they do.

That FAQ entry gives you THREE options for how to make your library multi-user. Just in case you missed them:

1) Use the content server
2) Use a remote desktop
3) Use a file sync tool, being careful not to run it at the same time as calibre

[erwin1234]

calibre, version 0.9.40

Traceback (most recent call last):
File "site-packages\calibre\gui2\metadata\basic_widgets.py", line 102, in commit
File "site-packages\calibre\library\database2.py", line 2554, in set_title
File "site-packages\calibre\library\database2.py", line 653, in set_path
File "site-packages\calibre\library\database2.py", line 1529, in add_format
File "site-packages\calibre\library\database2.py", line 1375, in copy_format_to
File "site-packages\calibre\utils\filenames.py", line 348, in copy_path_to
File "site-packages\calibre\startup.py", line 156, in local_open
OSError: [Errno 13] Permission denied

Arggg this is driving me nuts. All persmissions are full control for everyone on the network!

Calibre works with more than just the metadata.DB.
Calibre also does a 'Transaction' with the file system (the contents of the Book Folders).

That is not relevant, it is even not working for ONE single user. Just make sure there is only one instance running! I do not understand these error messages at all. Is this specific to the database code used?

do not put your calibre library on a network drive

Sigh.....

[kovidgoyal]

Sigh, you just wont listen, will you. All the best in your endeavors, I'm done wasting my time.

[erwin1234]

Sorry - I do not want to offend you in any way. Its just - the three options you suggest do not solve my problem. That is why I asked the question in the first place.

You do not want my feedback or my help and you do not consider any of my suggestions seriously. Ok fair enough its your project. I won't bother you again.

To show you I am just a caring supporter of Calibre, which I love, I will donate USD 30 with paypal when Calibre works over a network drive (single or multi user).

Thanks again and good luck with the next version.

ps: you just could have directed me to github: https://github.com/kovidgoyal/calibre/issues/26

[PeterT]

Originally Posted by erwin1234

File "site-packages\calibre\startup.py", line 156, in local_open
OSError: [Errno 13] Permission denied

Arggg this is driving me nuts. All persmissions are full control for everyone on the network!

Just a thought; while sure, it's not supported (and even frowned on), why not try using SysInternals ProcMon to trace what is happening and see if you can determine what is causing the Permission denied issue?

[erwin1234]

Peter, that is a good idea, user skipcube used sysinternals to trace what was happening in the github reported issue. After that you can inspect the source code which modifies and copies the directories&filenames. But I expect no easy solution.

That issue 26 was closed by kovidgoyal without solving it. Reading that thread, it seems there is something mysterious going on in the way calibre (or the python IO libraries) work with the remote NTFS/SMB filesystem. I googled a bit and this issue does not seem that simple to solve - probably that is why kovidgoyal closed it. He is working on a totally new multi-user version anyway, I suspect this will use a true transactions based database model for all operations (SQL?, maybe client/server?).

Even Microsoft has problems with their own transactions based NTFS file operations. They recommend not to use their API but use SQL filestreams instead... go figure..

I cloned the source code to see what I can track down. When I have time.. But I think kovidgoyal will come up with a multiuser version before I can find a cure for this issue

[FencepostErr1r]

Perhaps a bit missing the topic, but I don't have any problems with this combo:
OS X 10.8 Client, Openindiana (Solaris) ZFS storage server using netatalk 3.

I make sure that the share is mounted before starting Calibre, nothing else.

With 10.9 I'll make the move over to SMB 2.0, crossing my fingers...

-FencepostErr1r

[erwin1234]

Originally Posted by FencepostErr1r

Perhaps a bit missing the topic, but I don't have any problems with this combo

No it is not offtopic, good to know you do not have problems with ZFS + natatalk. It proves that the calibre code actually can work over the network. would it be possible to test a Windows 7/8 client on that server?

The test case would be to change metadata for book titles and authors, so operations that change filenames and directory names, and see if you get consistent results. So no error messages during the changes. And, after playing with the metadata, if you check the library from whithin calibre no errors should be listed (SQL should be ok, no missing or extra titels, authors etc).

The ZFS/SMB2.0 server + OSX client results would be interesting too.

Anyone running other combo's that work? Like linux EFS/ZFS + SMB on the server and windows on the client? If that works I want to try running a stripped down linux instance on my windows7 server using the free VMWARE player.

[FencepostErr1r]

I'll provide my findings when I get around to upgrade my server in time to test it with OS X 10.9 and SMB2.0, but I can't help you with the Windows side.

But if you want to test it yourself, you may want to set up a Win 8 or Server2012 VM, because those have the latest SMB implementation available.

-FencepostErr1r