[Metadata Source Plugin] FictionDB - Page 8

davidfor · 03-10-2022, 10:28 PM

Quote:

Originally Posted by Rellwood

No, I thought I could do it, because it seemed possible, I wrote too quickly...lol I am still trying to figure out how it would be possible.

I did look at this after the previous posts, and it does appear that the site may be usable now. Either they no longer use CloudFlare, or how it is protecting the site has changed. But, the site has also changed in other ways and the plugin doesn't work for that reason. When I get a chance, I'll look at it, but, I don't know when that will be.

iamagloworm · 05-13-2022, 06:34 AM

Quote:

Originally Posted by davidfor

I did look at this after the previous posts, and it does appear that the site may be usable now. Either they no longer use CloudFlare, or how it is protecting the site has changed. But, the site has also changed in other ways and the plugin doesn't work for that reason. When I get a chance, I'll look at it, but, I don't know when that will be.

This is fantastic news. Looking forward to following the development and hopefully being able to use this plugin again!

Thank you!

davidfor · 05-15-2022, 05:35 AM

As mentioned above, it appears that at some point FictionDB have changed how the site is hosted. The site stopped working as it was using CloudFlare for caching and protection. They have stopped using this, and the plugin can now work.

The attached plugin is updated to work with the site as it now is. Please test and report here how it goes.

iamagloworm · 05-15-2022, 06:19 AM

Quote:

Originally Posted by davidfor

As mentioned above, it appears that at some point FictionDB have changed how the site is hosted. The site stopped working as it was using CloudFlare for caching and protection. They have stopped using this, and the plugin can now work.

The attached plugin is updated to work with the site as it now is. Please test and report here how it goes.

Thank you!

I have just tested it, and it doesn't seem to work for me? I disabled all other metadata sources and this is the error I get when I run metadata download for a book that I know has all the details on fictiondb:

Quote:

calibre, version 5.42.0
ERROR: Download failed: Failed to download metadata or cover for the selected book.

The Five Misfits (Failed metadata)

The IP it is connecting to appears to be 188.114.96.16 / www.fictiondb.com which when I look it up is a Cloudflare service...

https://www.fictiondb.com/title/the-...na~1209356.htm

Please let me know if there is something I am doing wrong.

davidfor · 05-15-2022, 06:37 AM

Quote:

Originally Posted by iamagloworm

Thank you!

I have just tested it, and it doesn't seem to work for me? I disabled all other metadata sources and this is the error I get when I run metadata download for a book that I know has all the details on fictiondb:

The IP it is connecting to appears to be 188.114.96.16 / www.fictiondb.com which when I look it up is a Cloudflare service...

https://www.fictiondb.com/title/the-...na~1209356.htm

Please let me know if there is something I am doing wrong.

I created a book with author "Beatrice Alemagna" and title "The Misfits". I was able to get the metadata from FictionDB for that. Can you post the contents of the log? That might show some the issue.

iamagloworm · 05-15-2022, 06:40 AM

Quote:

Originally Posted by davidfor

I created a book with author "Beatrice Alemagna" and title "The Misfits". I was able to get the metadata from FictionDB for that. Can you post the contents of the log? That might show some the issue.

Sorry, I may be ignorant in this. Is there a way to get a verbose log? I have tried restarting Calibre in debug mode and still it only reports "The Five Misfits (Failed metadata)"

Is there a fictiondb log saved somewhere?

Edit: I added back default sources, and this appears in the log:

Spoiler:

Code:

#################### Log for The Five Misfits ####################
Running identify query with parameters: 
{'title': 'The Five Misfits', 'authors': ['Beatrice Alemagna'], 'identifiers': {}, 'timeout': 30} 
Using plugins: FictionDB (1, 2, 1), Google (1, 0, 3), Amazon.com (1, 2, 27) 
The log from individual plugins is below 

****************************** FictionDB (1, 2, 1) ****************************** 
Found 0 results 
Downloading from FictionDB took 0.5285921096801758 
create_query - isbn: "None"
Querying: https://www.fictiondb.com/search/searchresults.php - author=Beatrice+Alemagna&title=The+Five+Misfits&isbn=&datepublished=&synopsis=&rating=-&anthology=&imprint=&pubgroup=&srchtxt=multi&styp=6
_parse_search_results - No first row
No matches found with query: 'https://www.fictiondb.com/search/searchresults.php' - 'author=Beatrice+Alemagna&title=The+Five+Misfits&isbn=&datepublished=&synopsis=&rating=-&anthology=&imprint=&pubgroup=&srchtxt=multi&styp=6' 

******************************************************************************** 

****************************** Google (1, 0, 3) ****************************** 
Found 1 results 
Downloading from Google took 0.7474691867828369 


--- 
Title               : The Five Misfits
Author(s)           : Beatrice Alemagna
Publisher           : Frances Lincoln Children's Bks
Tags                : Juvenile Fiction, Social Themes, Friendship, Emotions & Feelings, Bullying
Languages           : eng
Published           : 2017-05-04T10:51:15.567004+00:00
Identifiers         : google:AXZBDQEACAAJ, isbn:9781786030092
Comments            : There were five of them altogether. Five misfits. The five had never amounted to much in life, nor did they feel like doing anything. And then one day, from who knows where, came an extraordinary fellow...When Mr Perfect comes to stay, the five misfits find themselves questioning what their purpose in life is. Can they teach him that not fitting in is more fun than trying to be perfect? A quirky, allegorical picture book from the multi award-winning creator of A Lion in Paris. 
Making query: https://books.google.com/books/feeds/volumes?q=intitle%3AFive+intitle%3AMisfits%2Binauthor%3ABeatrice+inauthor%3AAlemagna&max-results=20&start-index=1&min-viewability=none 

******************************************************************************** 

****************************** Amazon.com (1, 2, 27) ****************************** 
Found 0 results 
Downloading from Amazon.com took 1.206207036972046 
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=Five+Misfits+Beatrice+Alemagna+site%3Awww.amazon.com
Ignoring 'The Five Misfits Hardcover - Alemagna, Beatricehttps://www.amazon.com › Five-Mis...' as it has no cached page
Ignoring 'The Five Misfits (Chinese Edition): Beatrice Alemagnahttps://www.amazon.com › Five-Mis...' as it has no cached page
Failed to find any results on results page, with title: Five Misfits Beatrice Alemagna site:www.amazon.com - Recherche Google
No search engine results for terms: Five Misfits Beatrice Alemagna
Trying the bing search engine instead
Making bing query: https://www.bing.com/search?q=Five+Misfits+Beatrice+Alemagna+site%3Awww.amazon.com
Failed to find any results on results page, with title: Five Misfits Beatrice Alemagna site:www.amazon.com - Rechercher
No search engine results for terms: Five Misfits Beatrice Alemagna
No matches found with query: ['Five', 'Misfits', 'Beatrice', 'Alemagna'] 

******************************************************************************** 
The identify phase took 1.21 seconds 
The longest time (1.206207) was taken by: Amazon.com 
Merging results from different sources 
We have 1 merged results, merging took: 0.00 seconds

For the sake of completeness, here is my configuration:

Spoiler:

iamagloworm · 05-15-2022, 07:04 AM

Ok, I think I chose a poor example to test with. The title of the book is The Five Misfits but fictiondb is missing the "five".

I tried another example, and it works!
https://www.fictiondb.com/title/a-pe...ell~880434.htm

The missing bit of metadata I am looking for is still not coming through, it is the Age-Level.

Is it possible to add Age-Level? Ideally, there is a way I can put it into a column other than Tags. Thanks!

iamagloworm · 05-17-2022, 03:29 AM

I have been trying to figure out how to do this myself, but failing.

worker.py has...

Code:

        if cfg.plugin_prefs[cfg.STORE_NAME][cfg.KEY_GET_GENRE_AS_TAGS]:
            self._append_tags(root, 'Genre', calibre_tags, '//div[@class="row"]/div/div[@class="row"]/div/h4/../ul[@class="tagcloud-list"]')
        if cfg.plugin_prefs[cfg.STORE_NAME][cfg.KEY_GET_CLASSIFICATION_AS_TAGS]:
            self._append_tags(root, 'Classification', calibre_tags, '//div[@class="row"]/div/div[@class="row"]/div/h4/i[@class="fa fa-table"]/../../ul/li')
        if cfg.plugin_prefs[cfg.STORE_NAME][cfg.KEY_GET_TIME_PERIOD_AS_TAGS]:
            self._append_tags(root, 'Time Period', calibre_tags, '//div[@class="row"]/div/div[@class="row"]/div/h4/i[@class="fa fa-clock-o"]/../../ul/li')

And the relevant part of the page it is scraping looks like this:

Code:

<div class="col-lg-8 col-sm-8 mb-20">

                                <ul class="project-details-list">
                                    <li>
                                        <h6>Published:</h6>
                                        <div class="project-terms">
                                            Oct-2014 <small>(Hardcover)</small>                                        </div>
                                    </li>
                                    <li>
                                        <h6>Formats:</h6>
                                        <div class="project-terms">
                                            Print / eBook                                        </div>
                                    </li>																	
                                    <li>
                                        <h6>Rating:</h6>
                                        <div class="project-terms">
 	                                        <svg class="bi text-secondary" width="12" height="12" fill="currentColor"><use xlink:href="../images/fdb-icons.svg#star-fill"></use></svg>
				                    		<svg class="bi text-secondary" width="12" height="12" fill="currentColor"><use xlink:href="../images/fdb-icons.svg#star-fill"></use></svg>
          							        <svg class="bi text-secondary" width="12" height="12" fill="currentColor"><use xlink:href="../images/fdb-icons.svg#star-fill"></use></svg>
       							            <svg class="bi text-secondary" width="12" height="12" fill="currentColor"><use xlink:href="../images/fdb-icons.svg#star-fill"></use></svg>
         						           <svg class="bi text-secondary" width="12" height="12" fill="currentColor"><use xlink:href="../images/fdb-icons.svg#star-fill"></use></svg>                                            	
                                        </div>
                                    </li>	
                                    									
                                                                                                            
                                        <li>
                                            <h6>Pages:</h6>
                                            <div class="project-terms">
                                                40                                            </div>
                                        </li>
                                                                                                                <li>
                                            <h6>Age Level:</h6>
                                            <div class="project-terms">
                                                02-05                                            </div>
                                        </li>

iamagloworm · 05-17-2022, 04:00 AM

I tried this, but did not get any result. No tags show at all...

Code:

        if cfg.plugin_prefs[cfg.STORE_NAME][cfg.KEY_GET_AGE_LEVEL_AS_TAGS]:
            self._append_tags(root, 'Age Level', calibre_tags, '//li/h6[text()[contains(.,"Age Level")]]/../div[@class="project-terms"]/a')

Log:

Spoiler:

Code:

#################### Log for The Apple Doll ####################
Running identify query with parameters: 
{'title': 'The Apple Doll', 'authors': ['Elisa Kleven'], 'identifiers': {'mobi-asin': 'B006G31EB4'}, 'timeout': 30} 
Using plugins: FictionDB (1, 2, 1) 
The log from individual plugins is below 

****************************** FictionDB (1, 2, 1) ****************************** 
Found 1 results 
Downloading from FictionDB took 1.6465802192687988 


--- 
Title               : The Apple Doll
Author(s)           : Elisa Kleven
Publisher           : Farrar, Straus and Giroux
Published           : 2008-08-01T00:00:00+00:00
Identifiers         : fictiondb:the-apple-doll~elisa-kleven~328704
Comments            : Lizzy loves the big apple tree in her yard more than anything. So when the first day of school comes, she picks a beautiful apple, turns it into a makeshift doll she names Susanna, and takes it along to keep her company. But her teacher tells her that dolls aren't allowed at school. Even worse, her sister says that Susanna won't last forever. Then Lizzy's mom shows her a way to turn Susanna into a real apple doll. And with the help of Susanna the Apple Doll, Lizzy overcomes her shyness at school and makes plenty of new friends to bring home to play in her beloved apple tree. Detailed, delightful collage illustrations accompany this sweet story about one girl's success in bringing together her home world and her school world. Instructions for making an apple doll just like Susanna are included! 
create_query - isbn: "None"
Querying: https://www.fictiondb.com/search/searchresults.php - author=Elisa+Kleven&title=The+Apple+Doll&isbn=&datepublished=&synopsis=&rating=-&anthology=&imprint=&pubgroup=&srchtxt=multi&styp=6
Initial search including ISBN found something: 'https://www.fictiondb.com/title/the-apple-doll~elisa-kleven~328704.htm'
_parse_search_results - No first row
FictionDB book url: 'https://www.fictiondb.com/title/the-apple-doll~elisa-kleven~328704.htm'
parse_title: title_node= The Apple Doll
parse_authors: authors_node= Elisa Kleven
parse_publisher - publisher_node:  ['\xa0', 'Farrar, Straus and Giroux', 'Hardcover', '0374303800', '9780374303808', '\xa0', 'Farrar, Straus and Giroux', 'eBook', '1466808233', '9781466808232', '\xa0', 'Farrar, Straus and Giroux BYR', 'eBook', '146680341X', '9781466803411', '\xa0', 'Farrar, Straus, and Giroux', 'eBook (Kindle)'] 

******************************************************************************** 
The identify phase took 1.82 seconds 
The longest time (1.646580) was taken by: FictionDB 
Merging results from different sources 
We have 1 merged results, merging took: 0.00 seconds

Actually, upon further testing, it seems that no genres/tags are being scraped, even after reverting to your beta.

The same for classification and time period.

Does this work for you?

Log:

Spoiler:

Code:

#################### Log for Great expectations ####################
Running identify query with parameters: 
{'title': 'Great expectations', 'authors': ['Charles Dickens'], 'identifiers': {'isbn': '9781434000651'}, 'timeout': 30} 
Using plugins: FictionDB (1, 2, 1) 
The log from individual plugins is below 

****************************** FictionDB (1, 2, 1) ****************************** 
Found 1 results 
Downloading from FictionDB took 2.079063892364502 


--- 
Title               : Great Expectations
Author(s)           : Charles Dickens
Publisher           : Createspace
Series              : Classic Starts (Sterling) #1
Published           : 1862-01-01T00:00:00+00:00
Identifiers         : fictiondb:great-expectations~charles-dickens~167194
Comments            : Great Expectations is at once a superbly constructed novel of spellbinding mystery and a profound examination of moral values. Written at a time when Dickens' relationship with Victorian society had reached a crisis, this novel is peopled by characters unmistakably bearing Dickens' familiar stamp -- but here they appear in a new and questioning light. The orphan, Pip, and the convict, Magwitch ... the beautiful Estella, and her guardian, the embittered and vengeful Miss Havisham ... the strangely ambiguous figure of the master lawyer, Mr. Jaggers all play their part in a story whose title itself reflects the deep irony that shapes Dickens' searching reappraisal of the Victorian middle class. From the agony of his disenchantment comes a work that gives an added dimension to his matchless genius. 
create_query - isbn: "9781434000651"
Querying: https://www.fictiondb.com/search/searchresults.php - author=Charles+Dickens&title=Great+expectations&isbn=9781434000651&datepublished=&synopsis=&rating=-&anthology=&imprint=&pubgroup=&srchtxt=multi&styp=6
_parse_search_results - No first row
No matches found with identifiers, retrying using only title and authors
create_query - isbn: "None"
Querying: https://www.fictiondb.com/search/searchresults.php - author=Charles+Dickens&title=Great+expectations&isbn=&datepublished=&synopsis=&rating=-&anthology=&imprint=&pubgroup=&srchtxt=multi&styp=6
Initial search including ISBN found something: 'https://www.fictiondb.com/title/great-expectations~charles-dickens~167194.htm'
_parse_search_results - No first row
FictionDB book url: 'https://www.fictiondb.com/title/great-expectations~charles-dickens~167194.htm'
parse_title: title_node= Great Expectations
parse_authors: authors_node= Charles Dickens
series_node: series_node found - Classic Starts (Sterling)
parse_publisher - publisher_node:  ['\xa0', 'Createspace', 'Trade Paperback', '150861038X', '9781508610380', '\xa0', 'Globe Fearon', 'Paperback', '0822492601', '9780822492603', '\xa0', 'Longman Publishing Group', 'Paperback', '0582530032', '9780582530034', '\xa0', 'Signet', 'Mass Market Paperback', '0451523598', '9780451523594', '\xa0', 'Signet', 'Mass Market Paperback', '0451525248', '9780451525246', '\xa0', 'Signet', 'Mass Market Paperback', '0451520769', '9780451520760', '\xa0', 'MacMillan (UK)', 'Hardcover', '0333046374', '9780333046371', '\xa0', 'Dutton', 'Hardcover', '0460012347', '9780460012348', '\xa0', 'Hart Associates', 'Hardcover', '0805511989', '9780805511987', '\xa0', 'Pendulum Press', 'Hardcover', '0883014009', '9780883014004', '\xa0', 'Pendulum Press', 'Hardcover', '0883014122', '9780883014127', '\xa0', 'Dutton', 'Hardcover', '0130435031', '9780130435033', '\xa0', "St. Martin's", 'eBook', '1466805315', '9781466805316', '\xa0', 'Bantam', 'eBook', '0553897624', '9780553897623', '\xa0', 'Pocket', 'eBook', '1416503005', '9781416503002', '\xa0', 'Pan Macmillan', 'eBook', '1509831304', '9781509831302', '\xa0', 'Sheba Blake Publishing', 'eBook', '1329372603', '9781329372603', '\xa0', 'Diversion Books', 'eBook', '1682300277', '9781682300275', '\xa0', 'DH Audio', 'Audio Cassette', '0886460565', '9780886460563', '\xa0', 'DH Audio', 'Audio Cassette', '0886468035', '9780886468033', '\xa0', 'Audio Literature', 'Audio Cassette', '1558001387', '9781558001381', '\xa0', 'Soundelux Audio Pub', 'Audio Cassette', '0881423688', '9780881423686', '\xa0', 'Books in Motion', 'Audio Cassette', '1556863306', '9781556863301', '\xa0', 'DH Audio', 'Audio Cassette', '0886462908', '9780886462901', '\xa0', 'Ulverscroft Large Print', 'Hardcover', '0708980627', '9780708980620', 'Large Print', '\xa0', 'New Promise Inc', 'Trade Paperback', '1557012121', '9781557012128', 'Large Print', '\xa0', 'Cyber Classics', 'Trade Paperback', '1588550257', '9781588550255', 'Large Print', '\xa0', 'Thorndike Press', 'Hardcover', '078625405X', '9780786254057', 'Large Print', '\xa0', 'Echo Library', 'Paperback', '1847022499', '9781847022493', 'Large Print', '\xa0', 'Createspace', 'Trade Paperback', '149032870X', '9781490328706', 'Large Print'] 

******************************************************************************** 
The identify phase took 2.23 seconds 
The longest time (2.079064) was taken by: FictionDB 
Merging results from different sources 
We have 1 merged results, merging took: 0.00 seconds

Edit: for reference, this is the full tags code from worker.py - I wonder if it is using an old method that needs to be updated?

Spoiler:

davidfor · 05-17-2022, 04:25 AM

Quote:

Originally Posted by iamagloworm

Ok, I think I chose a poor example to test with. The title of the book is The Five Misfits but fictiondb is missing the "five".

Yes, that didn't work because the title you had did not match the title FictionDB had.

Quote:

I tried another example, and it works!
https://www.fictiondb.com/title/a-pe...ell~880434.htm

The missing bit of metadata I am looking for is still not coming through, it is the Age-Level.

Is it possible to add Age-Level? Ideally, there is a way I can put it into a column other than Tags. Thanks!

I'll have a look at what you are doing and see what is wrong.

davidfor · 05-17-2022, 06:15 AM

@iamagloworm: Attached is a beta of the plugin that should do what you want.

The issue with adding the "Age Level" in that way, was that it did not match the structure that "_append_tags" expected. I had to handle it separately. Also, the code for the Genre, Classification and Time period had changed, so they were not working either.

The beta has the following changes:

Fix: Updates for site changes
New: Add option for getting "Age Level" and putting it in the tags.

iamagloworm · 05-17-2022, 09:18 AM

Seems to work great, thank you!

Amazing!

kiwidude · 10-16-2022, 12:33 PM

Release Notes:
https://github.com/kiwidude68/calibr...ctiondb-v1.3.0

MegAnne · 11-01-2022, 12:13 PM

With the newest version, even when I check only "Include 'Genres' in the Tags column", I still get Age Level. Is that expected?

kiwidude · 11-01-2022, 12:47 PM

Quote:

Originally Posted by MegAnne

With the newest version, even when I check only "Include 'Genres' in the Tags column", I still get Age Level. Is that expected?

Hi MegAnne - do you have the details of the book on their website? Also the log output would be especially helpful. It could be something about that web page which has the plugin scraping data incorrectly - or it is possible that FictionDB didn't classify things properly themselves...

05-17-2022, 06:15 AM	#116
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	@iamagloworm: Attached is a beta of the plugin that should do what you want. The issue with adding the "Age Level" in that way, was that it did not match the structure that "_append_tags" expected. I had to handle it separately. Also, the code for the Genre, Classification and Time period had changed, so they were not working either. The beta has the following changes: Fix: Updates for site changes New: Add option for getting "Age Level" and putting it in the tags. Last edited by kiwidude; 10-16-2022 at 12:32 PM. Reason: Removed old version

10-16-2022, 12:33 PM	#118
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	FictionDB v1.3.0 Released Release Notes: https://github.com/kiwidude68/calibr...ctiondb-v1.3.0

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Metadata Source Plugin] BOL.com (NL)	Pr.BarnArt	Plugins	505	12-14-2023 03:23 PM
[Metadata Source Plugin] Skoob	rodrigoccurvo	Plugins	11	06-13-2019 06:44 PM
[Metadata Source Plugin] DIZZIE_NL	Pr.BarnArt	Plugins	7	08-11-2014 01:48 PM
[Metadata Source Plugin] catawiki	drMerry	Plugins	0	06-28-2011 08:42 PM
[Metadata Source Plugin] Amazon.it	nandocuci	Plugins	2	05-18-2011 02:36 AM

05-15-2022, 07:04 AM	#112
iamagloworm Connoisseur Posts: 59 Karma: 10 Join Date: Apr 2013 Device: Kindle Paperwhite 3 (Manga)	Ok, I think I chose a poor example to test with. The title of the book is The Five Misfits but fictiondb is missing the "five". I tried another example, and it works! https://www.fictiondb.com/title/a-pe...ell~880434.htm The missing bit of metadata I am looking for is still not coming through, it is the Age-Level. Is it possible to add Age-Level? Ideally, there is a way I can put it into a column other than Tags. Thanks!

05-17-2022, 09:18 AM	#117
iamagloworm Connoisseur Posts: 59 Karma: 10 Join Date: Apr 2013 Device: Kindle Paperwhite 3 (Manga)	Seems to work great, thank you! Amazing!

11-01-2022, 12:13 PM	#119
MegAnne Enthusiast Posts: 39 Karma: 10 Join Date: Aug 2011 Location: Atlantic Beach, FL Device: Amazon Fire HD 10	With the newest version, even when I check only "Include 'Genres' in the Tags column", I still get Age Level. Is that expected?

Advert

Advert