Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 09-20-2012, 09:15 PM   #1
murkyl
Junior Member
murkyl began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Sep 2012
Device: Nexus 7
Calibre server not updating over NAS

I think I've run into an interesting issue and wanted to see if anyone else can confirm that it is an issue or if I am configuring something improperly.

I am using the folder watch option along with the Calibre server in hopes of having an automated way of adding new books to my collection. My normal setup is to run Calibre in it's own virtual machine with the library located on a NAS mounted over SMB (Windows file sharing). The NAS box is another virtual machine running Linux and Samba.

Adding books via the watch folder works fine. Calibre automatically picks up the change and pulls in the book into the library properly. The part that does not work is that the Calibre server does not display the new books unless I restart the server or Calibre itself.

What I have discovered is this. If I have both the watch directory and the library on a "local" drive, local in this sense being the same virtual machine, so no NAS in the picture, everything updates properly. When I house the watch folder on the NAS and the library on the local drive, again everything works properly. It's when I put the library on the NAS that there is an issue.

The issue with the library sitting on the NAS is that the time stamp on the metadata.db file does NOT get updated properly after a book is added to the library. The timestamp is incorrect from any machine that I mount that same file share. So for example, if I started Calibre at 13:00 the metadata.db file will have the timestamp of 13:00 no matter how many books I add or modify over the course of an entire day. The interesting thing here is that the timestamp on the NAS itself IS correct. So if I login to the Linux NAS box directly and look at the timestamp it reports correctly. However, from any SMB client (Windows for example) the timestamp still says 13:00.

I thought let's try to isolate the NAS and see if it's just an SMB issue. My thinking is that perhaps I've configured something incorrectly on the Samba server. So the test was to use a virtual host mount form my virtual machine. This basically presents a local directory to the guest OS as a network share. No real NAS is in play here and everything is running on a Windows 7 box. When I do the same tests over the virtual share, I get the same results.

Calibre is running on a Windows XP box (to save on resources) and everything else seems to be running properly.

I am suspecting, with no evidence whatsoever, that Calibre is doing something with a file lock such that the SMB protocol is not updating the timestamp properly, thus the Calibre server is not updating properly.

Has anyone else encountered this or have any workaround? If not, can anyone help confirm on their machines that this is a reproducible issue? Is there anything I can do to assist in any further debugging?

Running Calibre v0.8.69
OS: Windows XP
Samba version: 3.6.7
SMBv2 is disabled
Library stored on a mapped drive

Thanks!

Last edited by murkyl; 09-20-2012 at 09:18 PM. Reason: Adding system info
murkyl is offline   Reply With Quote
Old 09-20-2012, 11:38 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,775
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
calibre uses the sqlite library to access metadata.db. sqlite requires networked filesystems to implement file locking correctly. Presumably whatever filesystem driver you are using does not. There's no way to fix this in calibre. You either need to change the sqlite locking mechanism or fix the filesystem driver you are using.
kovidgoyal is online now   Reply With Quote
Old 09-23-2012, 02:53 PM   #3
murkyl
Junior Member
murkyl began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Sep 2012
Device: Nexus 7
So I did some additional testing. Everything works fine if I use a Windows 7 machine to host the DB over SMB. It just is not working with my Samba server. I've run a wire shark trace and the SMB protocol is returning back the old time stamp even though the local Linux file system sees the change. I'm still not sure what is causing the issue but I will reply back if I figure this out.

The strange thing is that the same problem occurs when using the VMware host shares.

Is anyone else putting the DB on a Linux server shared over SMB?
murkyl is offline   Reply With Quote
Old 09-24-2012, 01:22 AM   #4
murkyl
Junior Member
murkyl began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Sep 2012
Device: Nexus 7
So there is a workaround fortunately! Calibre has the capability to have the metadata database in a different directory from the library itself. You need to set it with the full path including the name of the database file in an environment variable named:
e.g. C:\Some\Path\metadata.db
This is documented here: http://manual.calibre-ebook.com/customize.html

This link here in the forums also helped: https://www.mobileread.com/forums/sho...d.php?t=190436

It turns out there is some incompatibility between SQLite and Samba. You can find a ton of posts about this if you search on 'sqlite cifs'. It looks like it is a form of file locking which is causing the problem. A full Microsoft client and server do no seem to have this problem. It seems to only occur if either the client or server is running Samba.

I've used the environment variable to move the DB and now the Calibre server updates properly when new books are added. The last step is to backup the metadata DB files now that they are no longer on the NAS.

Hope this helps someone else in the future.
murkyl is offline   Reply With Quote
Old 02-09-2013, 01:53 PM   #5
mwaterbu
Junior Member
mwaterbu began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Jan 2012
Device: Kindle Keyboard 3G
Quote:
Originally Posted by murkyl View Post
So there is a workaround fortunately! Calibre has the capability to have the metadata database in a different directory from the library itself. You need to set it with the full path including the name of the database file in an environment variable named:
e.g. C:\Some\Path\metadata.db
This is documented here: http://manual.calibre-ebook.com/customize.html

This link here in the forums also helped: https://www.mobileread.com/forums/sho...d.php?t=190436

It turns out there is some incompatibility between SQLite and Samba. You can find a ton of posts about this if you search on 'sqlite cifs'. It looks like it is a form of file locking which is causing the problem. A full Microsoft client and server do no seem to have this problem. It seems to only occur if either the client or server is running Samba.

I've used the environment variable to move the DB and now the Calibre server updates properly when new books are added. The last step is to backup the metadata DB files now that they are no longer on the NAS.

Hope this helps someone else in the future.
Hi, I am having trouble making this work (having my library hosted on a NAS), and am hoping you can assist me. Calibre is installed to my machine (C:\Program Files\Calibre2), and my eBook files (organized by Calibre in directories with metadata.opf files and cover.jpg files) are stored on a NAS mapped to the N:\ drive.
I edited the calibre-portable.bat file to set the ebook location as follows:
Code:
IF EXIST N:\eBooks (
	SET CALIBRE_LIBRARY_DIRECTORY=N:\eBooks
	ECHO LIBRARY FILES:      N:\eBooks
)
And I set my metadata database location as follows:
Code:
IF EXIST "C:\Documents and Settings\Administrator\Calibre Library\metadata.db" (
	IF NOT "%CALIBRE_LIBRARY_DIRECTORY%" == "C:\Documents and Settings\Administrator\Calibre Library" (
		SET CALIBRE_OVERRIDE_DATABASE_PATH="C:\Documents and Settings\Administrator\Calibre Library\metadata.db"
		ECHO DATABASE:           C:\Documents and Settings\Administrator\Calibre Library\metadata.db
		ECHO '
		ECHO ***CAUTION*** Library Switching will be disabled 
		ECHO '
	)
)
so I have copied my metadata.db file from N:\eBooks (where it normally was) to C:\Documents and Settings\Administrator\Calibre Library.
However, when I launch the calibre-portable.bat file, I get the following error:
Code:
calibre, version 0.9.17
Corrupted database: The library database at N:\eBooks appears to be corrupted. Do you want calibre to try and rebuild it automatically? The rebuild may not be completely successful. If you say No, a new empty calibre library will be created.

Traceback (most recent call last):
  File "site-packages\calibre\gui2\main.py", line 228, in initialize_db
  File "site-packages\calibre\library\database2.py", line 166, in __init__
  File "site-packages\calibre\library\database2.py", line 114, in connect
  File "site-packages\calibre\library\sqlite.py", line 366, in connect
DatabaseException: unable to open database file
	RemoteTraceback (most recent call last):
	  File "site-packages\calibre\library\sqlite.py", line 248, in run
	  File "site-packages\calibre\library\sqlite.py", line 225, in connect
	OperationalError: unable to open database file
Trying to rebuild the library fails, so that will not work, unfortunately. I can confirm that my library is ok though, since I can launch Calibre normally with the metadata.db file that is on the NAS at N:\eBooks (but this takes a very long time, which is why I want to use the local copy of metadata.db).
Am I missing a step?
Thanks for the help!
mwaterbu is offline   Reply With Quote
Old 02-09-2013, 06:40 PM   #6
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
It is definitely possible to run with that sort of setup - I do it all the time.

To help yourself identify what is wrong, it is probably worth removing the 'echo off' at the start of the batch file. If you then start it from within a command window you can see the various lies of the batch file executing which may help?

If you still do not get anywhere by tomorrow leave another post, and I can post my batch file as a comparison (I am not at home until tomorrow evening).
itimpi is offline   Reply With Quote
Old 02-10-2013, 12:23 AM   #7
mwaterbu
Junior Member
mwaterbu began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Jan 2012
Device: Kindle Keyboard 3G
Thanks for your reply. I ended up getting everything working. I made an actual environment variable on my system instead of just using the calibre-portable.bat. Doing this worked right away.
It is taking several minutes to load my library though. It seems faster than before, but still takes a while. Not complaining, but I am wondering if this is fairly normal. Anything I can do to speed it up? I have disabled several columns in the main display window which helped loading times.
mwaterbu is offline   Reply With Quote
Reply

Tags
server update nas smb

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre Server Not Updating Fuller Calibre 1 11-21-2011 01:36 PM
Calibre Content server and Iomega NAS subivan Devices 9 08-12-2011 05:26 PM
Calibre-server not updating skells Calibre 16 07-22-2011 04:38 PM
Headless server on NAS readonly problem artifact Related Tools 1 06-28-2011 09:34 PM
Library on NAS server? bous Calibre 5 11-26-2009 05:03 AM


All times are GMT -4. The time now is 11:27 AM.


MobileRead.com is a privately owned, operated and funded community.