Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 06-28-2013, 10:14 PM   #1
epubfan
Member
epubfan began at the beginning.
 
Posts: 13
Karma: 10
Join Date: Jun 2013
Device: kobo touch
Please implement faster index-search

I have a library with 21.000 books and the index-search (title, author and so one) is very very slow (3 minutes and longer).
Any chance to implement a faster index-search-algorithm to calibre?

epubfan is offline   Reply With Quote
Old 06-28-2013, 11:40 PM   #2
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,244
Karma: 6020307
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by epubfan View Post
I have a library with 21.000 books and the index-search (title, author and so one) is very very slow (3 minutes and longer).
Any chance to implement a faster index-search-algorithm to calibre?

Your Anti-Virus is probably killing performance
OR
You are doing an unqualified search (Not specifying the search field) or adjusted the tweak to limit fields searched when not qualified.
theducks is offline   Reply With Quote
Old 06-29-2013, 02:03 AM   #3
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,965
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by epubfan View Post
I have a library with 21.000 books and the index-search (title, author and so one) is very very slow (3 minutes and longer).
Any chance to implement a faster index-search-algorithm to calibre?

I got faster results after removing comments from Search preferences - see attachment

BR
Attached Thumbnails
Click image for larger version

Name:	Capture.JPG
Views:	53
Size:	45.6 KB
ID:	107438  
BetterRed is offline   Reply With Quote
Old 06-29-2013, 03:39 PM   #4
epubfan
Member
epubfan began at the beginning.
 
Posts: 13
Karma: 10
Join Date: Jun 2013
Device: kobo touch
OK, thanks.
Qualified search in 1 column is faster.
My default search before was "title, authors, tags, series, publisher".
But i am nevertheless a little confused that searches in 5 colums need so long time.
epubfan is offline   Reply With Quote
Old 06-29-2013, 05:42 PM   #5
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,244
Karma: 6020307
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by epubfan View Post
OK, thanks.
Qualified search in 1 column is faster.
My default search before was "title, authors, tags, series, publisher".
But i am nevertheless a little confused that searches in 5 colums need so long time.
IMHO Tags is the un-qualified search performance killer.
Each tag is a record below the skirts. How many different tags do you have?
theducks is offline   Reply With Quote
Old 06-30-2013, 12:06 PM   #6
epubfan
Member
epubfan began at the beginning.
 
Posts: 13
Karma: 10
Join Date: Jun 2013
Device: kobo touch
Quote:
Originally Posted by theducks View Post
IMHO Tags is the un-qualified search performance killer.
Each tag is a record below the skirts. How many different tags do you have?
active are title, author, date, size, words, language, format.
epubfan is offline   Reply With Quote
Old 06-30-2013, 08:18 PM   #7
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,965
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by epubfan View Post
active are title, author, date, size, words, language, format.
I suspect format may slow things down because I think it refers to the extensions of files in the book folders

I thought there was a corresponding database column with a different name that was faster - but I've forgotten the name and I can't see it anywhere so maybe I dreamt it

BR
BetterRed is offline   Reply With Quote
Old 06-30-2013, 08:33 PM   #8
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 7,355
Karma: 46281507
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
Quote:
Originally Posted by BetterRed View Post
I suspect format may slow things down because I think it refers to the extensions of files in the book folders

I thought there was a corresponding database column with a different name that was faster - but I've forgotten the name and I can't see it anywhere so maybe I dreamt it

BR
Does this refresh your memory --> http://www.mobileread.com/forums/sho...d.php?t=188312
PeterT is online now   Reply With Quote
Old 06-30-2013, 09:01 PM   #9
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,965
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by PeterT View Post
- approximate_formats() is what I was thinking of, good to know I've better things than Calibre. perchance to dream upon

epubfan - suggest you do a search on approximate_formats to see what its all about - its a function rather than a column - you'll need to create a custom column - see attachment

BR
Attached Thumbnails
Click image for larger version

Name:	Capture.JPG
Views:	36
Size:	53.0 KB
ID:	107516  

Last edited by BetterRed; 07-01-2013 at 12:14 AM. Reason: add screen shot
BetterRed is offline   Reply With Quote
Old 06-30-2013, 09:08 PM   #10
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 7,355
Karma: 46281507
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
Quote:
Originally Posted by BetterRed View Post
- approximate_formats() is what I was thinking of, good to know I've better things than Calibre of which to dream on

epubfan - suggest you do a search on approximate_formats to see what its all about - its a function rather than a column - I don't know for certain that functions can be used in searches but I'd be surprised if they can't.

BR
Just create a custom column based on other columns and then use that fiction call in the definition of the column. When I'm back at my PC will post how to do it.
PeterT is online now   Reply With Quote
Old 07-01-2013, 10:35 AM   #11
epubfan
Member
epubfan began at the beginning.
 
Posts: 13
Karma: 10
Join Date: Jun 2013
Device: kobo touch
Quote:
Originally Posted by PeterT View Post
Just create a custom column based on other columns and then use that fiction call in the definition of the column. When I'm back at my PC will post how to do it.
Hm, then all informations in 1 db-column and searches will be faster or what is the trick?
epubfan is offline   Reply With Quote
Old 07-01-2013, 11:23 AM   #12
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 7,355
Karma: 46281507
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
No idea... All I was trying to do was to aid with the possible format column issue. Try searching the calibre forums for other people with performance issues...

The column I refer can be added via Preferences | Interface | Add your own column
Code:
Lookup name:    approxformats
Column Heading: Rough List of Formats
Column Type:     Column built from other columns
Template:          {:'approximate_formats()'}
PeterT is online now   Reply With Quote
Old 07-01-2013, 06:43 PM   #13
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,965
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by epubfan View Post
Hm, then all informations in 1 db-column and searches will be faster or what is the trick?
You wrote you search in title, author, date, size, words, language, format.

With the exception of format, these are database columns, the database is 'heavily indexed' in order to make searching and sorting fast.

However, 'formats' refers to the extensions of files in book folders, hence a search (or sort) must crawl through the library folder tree looking for files whose extensions match the search term, which is slow.

But, 'approximate_formats()' refers to a derived list of formats that you need to put in a custom database column which will be fast for the same reasons other database column based searches are.

See the link that Peter T posted for a discussion of why & how the function came to implemented. And see http://manual.calibre-ebook.com/temp...ximate-formats for definition and caveats

BR
BetterRed is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Index: Making a linked index in epub virtual_ink ePub 21 10-20-2011 12:23 AM
search/api index of ebooks? kbs Upload Help 0 08-25-2011 09:31 PM
Setting series index in bulk metadata search&replace bubak Calibre 4 12-19-2010 05:04 PM
Please implement proper quoting JSWolf Feedback 12 12-02-2010 05:39 AM
The Public Index: Google book search settlement website Elfwreck News 0 07-28-2009 12:23 PM


All times are GMT -4. The time now is 11:21 AM.


MobileRead.com is a privately owned, operated and funded community.