db.find_identical_books() and large numbers of authors

JimmXinu · 07-12-2012, 06:42 PM

In my FFDL plugin I've been successfully using LibraryDatabase2.find_identical_books(mi) to find existing, matching books for some time, like so:

Code:

            mi = MetaInformation(story.getMetadata("title"),
                                 story.getList("author"))
            identicalbooks = db.find_identical_books(mi)

This works until somewhere between 50 and 120 authors. 50 works, but when there's 120 authors the find fails due to recursion depth exceeded:

Spoiler:

Is there a better way to find matching books?

I hesitate to call it a bug because 120 authors is kind of crazy. But calibre is okay with having books with that many authors, it's just the find_identical_books call that fails.

kovidgoyal · 07-12-2012, 11:27 PM

find_identical_books constructs a search query, which in turn is parsed by a recursive descent parser, hence then recursion depth error on too large a query. You have two options, you can run the search manually, by iterating over book records, or you can break up the list of authors into smaller chunks and combine the results of multiple calls to find_identical_books. The latter will be much slower, but easier to implement.

JimmXinu · 07-14-2012, 10:55 AM

FYI, I bumped into the same error while trying to copy(with delete after copy) a book with 120 authors from one library to another. 0.8.60. So it's not just a problem for plugins.

chaley · 07-14-2012, 11:45 AM

@kovid: what about parenthesizing the query expression every 50 authors or so, such as
"(a and b and c and d) and (e and f and g and h)"? The parens should halt the recursion on the list of authors, replacing it with recursion on the outside-of-parens "and"s.

kovidgoyal · 07-14-2012, 12:36 PM

That would just mean we'd have to wait until someone comes up with a book with 6000 authors

Better would be to just use the first author and then iterate over the returned book ids checking for the remaining authors manually.

kovidgoyal · 07-14-2012, 01:05 PM

I've committed a fix along those lines.

07-12-2012, 06:42 PM	#1
JimmXinu Plugin Developer Posts: 6,320 Karma: 3966249 Join Date: Dec 2011 Location: Midwest USA Device: Kindle Paperwhite(10th)	db.find_identical_books() and large numbers of authors In my FFDL plugin I've been successfully using LibraryDatabase2.find_identical_books(mi) to find existing, matching books for some time, like so: Code: mi = MetaInformation(story.getMetadata("title"), story.getList("author")) identicalbooks = db.find_identical_books(mi) This works until somewhere between 50 and 120 authors. 50 works, but when there's 120 authors the find fails due to recursion depth exceeded: Spoiler: Traceback (most recent call last): File "site-packages\calibre\library\database2.py", line 1096, in find_identical_books File "site-packages\calibre\utils\search_query_parser.py", line 207, in parse File "site-packages\calibre\utils\search_query_parser.py", line 215, in _parse File "site-packages\calibre\utils\pyparsing.py", line 1021, in parseString File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache File "site-packages\calibre\utils\pyparsing.py", line 2623, in parseImpl File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache File "site-packages\calibre\utils\pyparsing.py", line 2478, in parseImpl File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache File "site-packages\calibre\utils\pyparsing.py", line 2623, in parseImpl ... File "site-packages\calibre\utils\pyparsing.py", line 2478, in parseImpl File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache File "site-packages\calibre\utils\pyparsing.py", line 2623, in parseImpl File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache File "site-packages\calibre\utils\pyparsing.py", line 2351, in parseImpl File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache File "site-packages\calibre\utils\pyparsing.py", line 2819, in parseImpl File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache File "site-packages\calibre\utils\pyparsing.py", line 2478, in parseImpl File "site-packages\calibre\utils\pyparsing.py", line 894, in _parseNoCache File "site-packages\calibre\utils\pyparsing.py", line 2351, in parseImpl File "site-packages\calibre\utils\pyparsing.py", line 898, in _parseNoCache RuntimeError: maximum recursion depth exceeded Is there a better way to find matching books? I hesitate to call it a bug because 120 authors is kind of crazy. But calibre is okay with having books with that many authors, it's just the find_identical_books call that fails.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Suggestions for adding large numbers of titles.	chas0039	Amazon Kindle	4	07-18-2011 01:37 PM
Tip from Amazon for large numbers of titles.	chas0039	Amazon Kindle	4	06-16-2011 12:18 AM
Is there a hack for displaying page numbers rather than location numbers?	nesler	Kindle Developer's Corner	16	02-15-2011 12:00 AM
Best way to do a TOC for large numbers of entries?	HarryT	Workshop	5	01-18-2009 07:02 AM
Short Fiction Authors, Various: Stories by Foreign Authors: Polish, Greek, Belgian. v1, 20 Feb 2008	Madam Broshkina	BBeB/LRF Books (offline)	0	02-20-2008 08:40 PM

07-12-2012, 11:27 PM	#2
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	find_identical_books constructs a search query, which in turn is parsed by a recursive descent parser, hence then recursion depth error on too large a query. You have two options, you can run the search manually, by iterating over book records, or you can break up the list of authors into smaller chunks and combine the results of multiple calls to find_identical_books. The latter will be much slower, but easier to implement.

07-14-2012, 10:55 AM	#3
JimmXinu Plugin Developer Posts: 6,320 Karma: 3966249 Join Date: Dec 2011 Location: Midwest USA Device: Kindle Paperwhite(10th)	FYI, I bumped into the same error while trying to copy(with delete after copy) a book with 120 authors from one library to another. 0.8.60. So it's not just a problem for plugins.

07-14-2012, 11:45 AM	#4
chaley Grand Sorcerer Posts: 11,742 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	@kovid: what about parenthesizing the query expression every 50 authors or so, such as "(a and b and c and d) and (e and f and g and h)"? The parens should halt the recursion on the list of authors, replacing it with recursion on the outside-of-parens "and"s.

07-14-2012, 12:36 PM	#5
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	That would just mean we'd have to wait until someone comes up with a book with 6000 authors Better would be to just use the first author and then iterate over the returned book ids checking for the remaining authors manually.

07-14-2012, 01:05 PM	#6
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I've committed a fix along those lines.

Advert

Advert