Calibre slowness on Webdav

madeira · 12-19-2012, 05:05 AM

Hi all, I set up a cloud storage space with Webdav access and put on it the Calibre library. In this way I can share it with on different devices. It works but I noticed the things are a little slow. I know that Webdav is slow by itself but there is a huge difference while accessing the folders in the cloud via the Finder if compared to Calibre behavior. So I did some tests and collected the I/o information. Following the results of a test run.

Reads Count Bytes Read Delta reads Delta Bytes
1 Start 11279 36.624.145,00
2 Switch to My library 11792 37.865.863,00 513 1.241.718,00
3 Click on a line 11822 38.568.167,00 30 702.304,00
4 Close Panel 11861 39.272.338,00 39 704.171,00
5 Open Panel 11900 39.976.509,00 39 704.171,00
6 Click on line 11933 41.216.208,00 33 1.239.699,00
7 Right Click 11951 41.322.889,00 18 106.681,00
8 Close Panel 11990 42.623.485,00 39 1.300.596,00
9 Click on a line 12032 43.135.599,00 42 512.114,00
10 Right Click 12032 43.135.599,00 0 -
11 Right Click on a different line 12056 43.708.691,00 24 573.092,00
12 Swirch to Calibre Library 12179 44.127.570,00 123 418.879,00

What I did at 4,5,8 was to Open/close the panel where is showed the Book cover and summary.

In my opinion there is some unnecessary I/O in several points.
Why perform 42 Reads for 512.114 bytes while just clicking on a line with the preview panel closed?
In any case In my opinion there are several unnecessary reads from the disk that could be optimized.
I'll perform other tests and submit more results.

Thanks. Ciao.

DoctorOhh · 12-19-2012, 05:20 AM

Moderator Notice
Moved the thread to general calibre forum. Please read the "PLEASE READ BEFORE POSTING - The purpose of this subforum" sticky post from the top of the Development forum.

madeira · 12-19-2012, 05:27 AM

I'm so sorry.... I misunderstood the sticky. Ciao.

kovidgoyal · 12-19-2012, 05:30 AM

The way to run calibre networked is not by networking the filesystem. Frankly, the whole idea of networked filesystems is a god-awful kludge. Filesystems have the assumption of reliable, fast, synchronous I/O baked into their semantics in too many places.

Your options are:
1) Use a remote desktop solution
2) Use the content server for read only access
3) use some kind of file sync like unison or dropbox and ensure you only run one copy of calibre at a time.
4) Help with implementing the calibre networked db backend

No matter how much optimization you do, you will never get acceptable performance going through a networked filesystem. But hey, if you feel differently, calibre is open source, patches are welcome

madeira · 12-19-2012, 06:05 AM

Thanks for the prompt reply. I agree with you that networked filesystems are not the best solution when you seek performance.
I already use a dropbox based solution that is ok unless the need to replicate the same library on some machines. Not efficient.
What I reported is not that should be implemented a solution to manage remote calibre DB with user access control but that in my opinion some details of the processing related to the GUI use maybe could be optimized. I mean, If a panel is hidden, why access data that never will be shown. If I right click on a item to get a context menu, why about 0,5 Mb should be read.
Today I don't have the right skills to help in the development but in testing I have my words to say. Ciao.

12-19-2012, 05:05 AM	#1
madeira Junior Member Posts: 4 Karma: 10 Join Date: Jul 2010 Device: BeBook Mini	Calibre slowness on Webdav Hi all, I set up a cloud storage space with Webdav access and put on it the Calibre library. In this way I can share it with on different devices. It works but I noticed the things are a little slow. I know that Webdav is slow by itself but there is a huge difference while accessing the folders in the cloud via the Finder if compared to Calibre behavior. So I did some tests and collected the I/o information. Following the results of a test run. Reads Count Bytes Read Delta reads Delta Bytes 1 Start 11279 36.624.145,00 2 Switch to My library 11792 37.865.863,00 513 1.241.718,00 3 Click on a line 11822 38.568.167,00 30 702.304,00 4 Close Panel 11861 39.272.338,00 39 704.171,00 5 Open Panel 11900 39.976.509,00 39 704.171,00 6 Click on line 11933 41.216.208,00 33 1.239.699,00 7 Right Click 11951 41.322.889,00 18 106.681,00 8 Close Panel 11990 42.623.485,00 39 1.300.596,00 9 Click on a line 12032 43.135.599,00 42 512.114,00 10 Right Click 12032 43.135.599,00 0 - 11 Right Click on a different line 12056 43.708.691,00 24 573.092,00 12 Swirch to Calibre Library 12179 44.127.570,00 123 418.879,00 What I did at 4,5,8 was to Open/close the panel where is showed the Book cover and summary. In my opinion there is some unnecessary I/O in several points. Why perform 42 Reads for 512.114 bytes while just clicking on a line with the preview panel closed? In any case In my opinion there are several unnecessary reads from the disk that could be optimized. I'll perform other tests and submit more results. Thanks. Ciao.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Mini WebDAV HowTo	knc1	Kindle Developer's Corner	10	10-04-2013 10:02 AM
Calibre slowness as library expands	microcity	Library Management	10	06-14-2012 03:59 PM
Calibre Content Server in Cloud via WebDAV	cyana	Library Management	3	12-23-2011 11:52 PM
Overall slowness	iharley	Calibre	2	07-06-2010 12:29 AM
Database locked using WebDav folder on Mac OS X	astrodad	Calibre	4	08-04-2008 01:01 PM

12-19-2012, 05:20 AM	#2
DoctorOhh US Navy, Retired Posts: 9,864 Karma: 13806776 Join Date: Feb 2009 Location: North Carolina Device: Icarus Illumina XL HD, Nexus 7	Moderator Notice Moved the thread to general calibre forum. Please read the "PLEASE READ BEFORE POSTING - The purpose of this subforum" sticky post from the top of the Development forum.

12-19-2012, 05:27 AM	#3
madeira Junior Member Posts: 4 Karma: 10 Join Date: Jul 2010 Device: BeBook Mini	I'm so sorry.... I misunderstood the sticky. Ciao.

12-19-2012, 05:30 AM	#4
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The way to run calibre networked is not by networking the filesystem. Frankly, the whole idea of networked filesystems is a god-awful kludge. Filesystems have the assumption of reliable, fast, synchronous I/O baked into their semantics in too many places. Your options are: 1) Use a remote desktop solution 2) Use the content server for read only access 3) use some kind of file sync like unison or dropbox and ensure you only run one copy of calibre at a time. 4) Help with implementing the calibre networked db backend No matter how much optimization you do, you will never get acceptable performance going through a networked filesystem. But hey, if you feel differently, calibre is open source, patches are welcome

12-19-2012, 06:05 AM	#5
madeira Junior Member Posts: 4 Karma: 10 Join Date: Jul 2010 Device: BeBook Mini	Thanks for the prompt reply. I agree with you that networked filesystems are not the best solution when you seek performance. I already use a dropbox based solution that is ok unless the need to replicate the same library on some machines. Not efficient. What I reported is not that should be implemented a solution to manage remote calibre DB with user access control but that in my opinion some details of the processing related to the GUI use maybe could be optimized. I mean, If a panel is hidden, why access data that never will be shown. If I right click on a item to get a context menu, why about 0,5 Mb should be read. Today I don't have the right skills to help in the development but in testing I have my words to say. Ciao.

Advert

Advert