Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 07-20-2013, 08:10 PM   #1
erwin1234
Member
erwin1234 doesn't littererwin1234 doesn't littererwin1234 doesn't litter
 
Posts: 22
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
Question Calibre Network Request

Good to know you are working on a true multi user support. That is a big step! I would already be happy with a much smaller step.

My family really love calibre. Whith all those devices in the house, PC's, laptops, tablets, smartphones, ereaders and what more I decided to store all our ebook data on our main W7 server computer. And let all family members help with maintaining the database, store their books, sync their devices. And have an automated backup each night for safety.

But bummer... calibre does not work that well in this scenario. Even if you mount the library via a network drive letter between two Windows 7 computers running the NTFS filesystems it seems calibre has problems and will give you strange windows errors (no access, cannot write..). Note this is not a true multi user configuration - it is not working for a single user using a mapped network drive! I have never seen problems like this with other software. Should file operations like open/read/write/rename not be just transparant over the local network? Are there bugs or limitations in the code?

Users requested this scenario to work since 2010/2011. Before doing the full fledged multiuser implementation, would it be possible to first fix the one user/multiple computers/networked database scenario? I did not look at the code yet, but how much work do you think would this take?

I know file locking over network drives can be tedious. But you could use other mechanisms like creating temp .lock files to make sure there is only one instance running. Like e.g. the sync program freefilesyn does. Or forget about the lock at all and let the user decide how to run it. Just make sure the open/read/write/rename works on a network drive.

What do you think?

Oh and ofcourse thanks for this excellent software !
erwin1234 is offline   Reply With Quote
Old 07-21-2013, 12:07 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,338
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
http://manual.calibre-ebook.com/faq....rked-drive-nas
kovidgoyal is offline   Reply With Quote
Old 07-21-2013, 12:10 AM   #3
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,888
Karma: 12755553
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Welocome to Mobileread.

Moderator Notice
Moved post to its own thread. Please read the "PLEASE READ BEFORE POSTING - The purpose of this subforum" sticky post for guidelines for posting in the development forum.

Here is the current FAQ entry on setting up calibre to run from a network.
DoctorOhh is offline   Reply With Quote
Old 07-24-2013, 08:21 AM   #4
erwin1234
Member
erwin1234 doesn't littererwin1234 doesn't littererwin1234 doesn't litter
 
Posts: 22
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
Quote:
Originally Posted by kovidgoyal View Post
Yes ofcourse I have seen this link. Multiple times, you are posting it all the time. That does not mean it makes sense. As a windows programmer I just do not understand why you write "Most network filesystems lack various filesystem features that calibre uses". Most corporate IT systems ONLY use networked filesystems! They are running their databases over network filesystems all the time. With redundancy and file locking, realtime version control, cloud backups, database transactions and everything what not!

So could you please be so kind to explain your statement "Most network filesystems lack various filesystem features that calibre uses" so we can fix the sourcecode accordingly.

Thank you, just trying to help.
erwin1234 is offline   Reply With Quote
Old 07-24-2013, 08:35 AM   #5
erwin1234
Member
erwin1234 doesn't littererwin1234 doesn't littererwin1234 doesn't litter
 
Posts: 22
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
Quote:
Originally Posted by DoctorOhh View Post
Welocome to Mobileread.

Moved post to its own thread. Please read the "PLEASE READ BEFORE POSTING - The purpose of this subforum" sticky post for guidelines for posting in the development forum.
Here is the current FAQ entry on setting up calibre to run from a network.
Thank you for making this a seperate thread.

The FAQ does not make sense because the answer to the question "on setting up calibre to run from a network" is smiply "Do not put your calibre library on a networked drive". That is not an answer !

The content server solution is not an option. The goal is to have ONE user at the time working with the full calibre software using the DB over the network. So multiple user but NO concurrency - only one user at the time. But from different locations.

Solution 1: All modern network filesystems have a file locking option - if you open a Word file over the network and another user opens the same file you get a dialog box with "file is already open". This could be a simpe check in calibre? http://en.wikipedia.org/wiki/File_locking

Solution 2: Alternatively you could just create a temp filename "metadata.db.locked" at startup and check for the presence of this file at startup and open it for read/write. If you succeed, you can continue. If not, it is already opened in another instance, so inform the user and quit. The normal program exit would remove this .lock file. But in case of a crash it still works because the file will be closed after some time by the OS.
erwin1234 is offline   Reply With Quote
Old 07-24-2013, 09:11 AM   #6
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,098
Karma: 5939999
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by erwin1234 View Post
Thank you for making this a seperate thread.

The FAQ does not make sense because the answer to the question "on setting up calibre to run from a network" is smiply "Do not put your calibre library on a networked drive". That is not an answer !

The content server solution is not an option. The goal is to have ONE user at the time working with the full calibre software using the DB over the network. So multiple user but NO concurrency - only one user at the time. But from different locations.

Solution 1: All modern network filesystems have a file locking option - if you open a Word file over the network and another user opens the same file you get a dialog box with "file is already open". This could be a simpe check in calibre? http://en.wikipedia.org/wiki/File_locking

Solution 2: Alternatively you could just create a temp filename "metadata.db.locked" at startup and check for the presence of this file at startup and open it for read/write. If you succeed, you can continue. If not, it is already opened in another instance, so inform the user and quit. The normal program exit would remove this .lock file. But in case of a crash it still works because the file will be closed after some time by the OS.
Calibre works with more than just the metadata.DB.
Calibre also does a 'Transaction' with the file system (the contents of the Book Folders).

You change an Author or Title, The Author or Title folders get Renamed/Moved.
You change the cover, and the Cover file gets updated
You change any Metadata, and Metadata.OPF (the backup) gets updated

It is not as Simple as 'Let the OS locking do it'.
Caching created delays alone will wreak havoc if any file is delayed and another Calibre instance makes a request before the 'transaction' has reached stability.

Timing is everything.

Kovid is doing a re-write to accomplish Multi User.
Again, Timing.
It will only be released when it is working perfectly Right
theducks is offline   Reply With Quote
Old 07-24-2013, 09:35 AM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,338
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Again, do not put your calibre library on a network drive. File locking is just one problem, others include data integrity across operating systems and most importantly, the fact that various filesystem APIs are BROKEN on network file systems, and when I say BROKEN I mean BROKEN, as in they LIE about what they do.

That FAQ entry gives you THREE options for how to make your library multi-user. Just in case you missed them:

1) Use the content server
2) Use a remote desktop
3) Use a file sync tool, being careful not to run it at the same time as calibre
kovidgoyal is offline   Reply With Quote
Old 07-24-2013, 12:28 PM   #8
erwin1234
Member
erwin1234 doesn't littererwin1234 doesn't littererwin1234 doesn't litter
 
Posts: 22
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
calibre, version 0.9.40

Traceback (most recent call last):
File "site-packages\calibre\gui2\metadata\basic_widgets.py", line 102, in commit
File "site-packages\calibre\library\database2.py", line 2554, in set_title
File "site-packages\calibre\library\database2.py", line 653, in set_path
File "site-packages\calibre\library\database2.py", line 1529, in add_format
File "site-packages\calibre\library\database2.py", line 1375, in copy_format_to
File "site-packages\calibre\utils\filenames.py", line 348, in copy_path_to
File "site-packages\calibre\startup.py", line 156, in local_open
OSError: [Errno 13] Permission denied

Arggg this is driving me nuts. All persmissions are full control for everyone on the network!

Quote:
Calibre works with more than just the metadata.DB.
Calibre also does a 'Transaction' with the file system (the contents of the Book Folders).
That is not relevant, it is even not working for ONE single user. Just make sure there is only one instance running! I do not understand these error messages at all. Is this specific to the database code used?

Quote:
do not put your calibre library on a network drive
Sigh.....
erwin1234 is offline   Reply With Quote
Old 07-24-2013, 12:30 PM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,338
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Sigh, you just wont listen, will you. All the best in your endeavors, I'm done wasting my time.
kovidgoyal is offline   Reply With Quote
Old 07-24-2013, 12:48 PM   #10
erwin1234
Member
erwin1234 doesn't littererwin1234 doesn't littererwin1234 doesn't litter
 
Posts: 22
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
Sorry - I do not want to offend you in any way. Its just - the three options you suggest do not solve my problem. That is why I asked the question in the first place.

You do not want my feedback or my help and you do not consider any of my suggestions seriously. Ok fair enough its your project. I won't bother you again.

To show you I am just a caring supporter of Calibre, which I love, I will donate USD 30 with paypal when Calibre works over a network drive (single or multi user).

Thanks again and good luck with the next version.

ps: you just could have directed me to github: https://github.com/kovidgoyal/calibre/issues/26

Last edited by erwin1234; 07-24-2013 at 01:12 PM.
erwin1234 is offline   Reply With Quote
Old 07-24-2013, 05:10 PM   #11
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 7,197
Karma: 45329895
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
Quote:
Originally Posted by erwin1234 View Post
File "site-packages\calibre\startup.py", line 156, in local_open
OSError: [Errno 13] Permission denied

Arggg this is driving me nuts. All persmissions are full control for everyone on the network!
Just a thought; while sure, it's not supported (and even frowned on), why not try using SysInternals ProcMon to trace what is happening and see if you can determine what is causing the Permission denied issue?
PeterT is offline   Reply With Quote
Old 07-24-2013, 07:44 PM   #12
erwin1234
Member
erwin1234 doesn't littererwin1234 doesn't littererwin1234 doesn't litter
 
Posts: 22
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
Peter, that is a good idea, user skipcube used sysinternals to trace what was happening in the github reported issue. After that you can inspect the source code which modifies and copies the directories&filenames. But I expect no easy solution.

That issue 26 was closed by kovidgoyal without solving it. Reading that thread, it seems there is something mysterious going on in the way calibre (or the python IO libraries) work with the remote NTFS/SMB filesystem. I googled a bit and this issue does not seem that simple to solve - probably that is why kovidgoyal closed it. He is working on a totally new multi-user version anyway, I suspect this will use a true transactions based database model for all operations (SQL?, maybe client/server?).

Even Microsoft has problems with their own transactions based NTFS file operations. They recommend not to use their API but use SQL filestreams instead... go figure..

I cloned the source code to see what I can track down. When I have time.. But I think kovidgoyal will come up with a multiuser version before I can find a cure for this issue
erwin1234 is offline   Reply With Quote
Old 07-25-2013, 11:53 AM   #13
FencepostErr1r
Enthusiast
FencepostErr1r began at the beginning.
 
Posts: 28
Karma: 10
Join Date: Jul 2013
Device: Kobo Glo
Perhaps a bit missing the topic, but I don't have any problems with this combo:
OS X 10.8 Client, Openindiana (Solaris) ZFS storage server using netatalk 3.

I make sure that the share is mounted before starting Calibre, nothing else.

With 10.9 I'll make the move over to SMB 2.0, crossing my fingers...

-FencepostErr1r
FencepostErr1r is offline   Reply With Quote
Old 07-26-2013, 08:23 AM   #14
erwin1234
Member
erwin1234 doesn't littererwin1234 doesn't littererwin1234 doesn't litter
 
Posts: 22
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
Quote:
Originally Posted by FencepostErr1r View Post
Perhaps a bit missing the topic, but I don't have any problems with this combo
No it is not offtopic, good to know you do not have problems with ZFS + natatalk. It proves that the calibre code actually can work over the network. would it be possible to test a Windows 7/8 client on that server?

The test case would be to change metadata for book titles and authors, so operations that change filenames and directory names, and see if you get consistent results. So no error messages during the changes. And, after playing with the metadata, if you check the library from whithin calibre no errors should be listed (SQL should be ok, no missing or extra titels, authors etc).

The ZFS/SMB2.0 server + OSX client results would be interesting too.

Anyone running other combo's that work? Like linux EFS/ZFS + SMB on the server and windows on the client? If that works I want to try running a stripped down linux instance on my windows7 server using the free VMWARE player.

Last edited by erwin1234; 07-26-2013 at 08:32 AM.
erwin1234 is offline   Reply With Quote
Old 07-26-2013, 08:59 AM   #15
FencepostErr1r
Enthusiast
FencepostErr1r began at the beginning.
 
Posts: 28
Karma: 10
Join Date: Jul 2013
Device: Kobo Glo
I'll provide my findings when I get around to upgrade my server in time to test it with OS X 10.9 and SMB2.0, but I can't help you with the Windows side.

But if you want to test it yourself, you may want to set up a Win 8 or Server2012 VM, because those have the latest SMB implementation available.

-FencepostErr1r
FencepostErr1r is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Which port will Calibre connect to a network share? awj Devices 3 03-02-2013 11:14 AM
Calibre with library on mapped network drive litiki777 Related Tools 1 10-30-2012 06:31 AM
Kobo & Calibre library over network Issachar Which one should I buy? 7 12-21-2011 12:19 AM
Calibre with OS X 10.6 Network User fiveandfriends Devices 0 03-03-2011 01:15 PM
Calibre performance over the network mrwindex Calibre 11 07-30-2009 01:45 AM


All times are GMT -4. The time now is 09:37 PM.


MobileRead.com is a privately owned, operated and funded community.