Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 07-12-2012, 06:42 PM   #1
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,671
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
db.find_identical_books() and large numbers of authors

In my FFDL plugin I've been successfully using LibraryDatabase2.find_identical_books(mi) to find existing, matching books for some time, like so:

Code:
            mi = MetaInformation(story.getMetadata("title"),
                                 story.getList("author"))
            identicalbooks = db.find_identical_books(mi)
This works until somewhere between 50 and 120 authors. 50 works, but when there's 120 authors the find fails due to recursion depth exceeded:

Spoiler:
Traceback (most recent call last):
File "site-packages\calibre\library\database2.py", line 1096, in find_identical_books
File "site-packages\calibre\utils\search_query_parser.py", line 207, in parse
File "site-packages\calibre\utils\search_query_parser.py", line 215, in _parse
File "site-packages\calibre\utils\pyparsing.py", line 1021, in parseString
File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache
File "site-packages\calibre\utils\pyparsing.py", line 2623, in parseImpl
File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache
File "site-packages\calibre\utils\pyparsing.py", line 2478, in parseImpl
File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache
File "site-packages\calibre\utils\pyparsing.py", line 2623, in parseImpl
...
File "site-packages\calibre\utils\pyparsing.py", line 2478, in parseImpl
File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache
File "site-packages\calibre\utils\pyparsing.py", line 2623, in parseImpl
File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache
File "site-packages\calibre\utils\pyparsing.py", line 2351, in parseImpl
File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache
File "site-packages\calibre\utils\pyparsing.py", line 2819, in parseImpl
File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache
File "site-packages\calibre\utils\pyparsing.py", line 2478, in parseImpl
File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache
File "site-packages\calibre\utils\pyparsing.py", line 2351, in parseImpl
File "site-packages\calibre\utils\pyparsing.py", line 898, in _parseNoCache
RuntimeError: maximum recursion depth exceeded


Is there a better way to find matching books?

I hesitate to call it a bug because 120 authors is kind of crazy. But calibre is okay with having books with that many authors, it's just the find_identical_books call that fails.
JimmXinu is offline   Reply With Quote
Old 07-12-2012, 11:27 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,906
Karma: 5035037
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
find_identical_books constructs a search query, which in turn is parsed by a recursive descent parser, hence then recursion depth error on too large a query. You have two options, you can run the search manually, by iterating over book records, or you can break up the list of authors into smaller chunks and combine the results of multiple calls to find_identical_books. The latter will be much slower, but easier to implement.
kovidgoyal is offline   Reply With Quote
 
Enthusiast
Old 07-14-2012, 10:55 AM   #3
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,671
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
FYI, I bumped into the same error while trying to copy(with delete after copy) a book with 120 authors from one library to another. 0.8.60. So it's not just a problem for plugins.
JimmXinu is offline   Reply With Quote
Old 07-14-2012, 11:45 AM   #4
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,399
Karma: 821638
Join Date: Jan 2010
Location: France
Device: Many android devices
@kovid: what about parenthesizing the query expression every 50 authors or so, such as
"(a and b and c and d) and (e and f and g and h)"? The parens should halt the recursion on the list of authors, replacing it with recursion on the outside-of-parens "and"s.
chaley is offline   Reply With Quote
Old 07-14-2012, 12:36 PM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,906
Karma: 5035037
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
That would just mean we'd have to wait until someone comes up with a book with 6000 authors

Better would be to just use the first author and then iterate over the returned book ids checking for the remaining authors manually.
kovidgoyal is offline   Reply With Quote
Old 07-14-2012, 01:05 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,906
Karma: 5035037
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I've committed a fix along those lines.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Suggestions for adding large numbers of titles. chas0039 Amazon Kindle 4 07-18-2011 01:37 PM
Tip from Amazon for large numbers of titles. chas0039 Amazon Kindle 4 06-16-2011 12:18 AM
Is there a hack for displaying page numbers rather than location numbers? nesler Kindle Developer's Corner 16 02-15-2011 12:00 AM
Best way to do a TOC for large numbers of entries? HarryT Workshop 5 01-18-2009 07:02 AM
Short Fiction Authors, Various: Stories by Foreign Authors: Polish, Greek, Belgian. v1, 20 Feb 2008 Madam Broshkina BBeB/LRF Books 0 02-20-2008 08:40 PM


All times are GMT -4. The time now is 11:10 PM.


MobileRead.com is a privately owned, operated and funded community.