[Metadata Source Plugin] Fantastic Fiction

[kiwidude]

Hi feuille, thanks for replying. Perhaps I didn't phrase my question well. I know FF only has the english version of books, no issue there. However my question was is it likely that lets say a German or Spanish user of calibre, who has only their native language translation copy of the book (i.e. non english) would be using the Fantastic Fiction metadata plugin and able to get a match? They might have a mix of english and non-english books in their library and likely not diligent about disabling the FF metadata plugin when getting data for the non-english ones.

Because the problem is that if I force the metadata for the book to be english, that will impact what calibre thinks that copy of the book is. Which to me sounds like a really bad thing to be doing.

Maybe this situation cannot happen in reality. Will a foreign language translation of a book always have a different (non english) name, so FF could not match the book. Certainly I would expect it to have different identifiers like ISBN, assuming it is not incorrectly recorded for that book.

I just don't want to make the change and then have x amount of users complaining about how their libraries are now corrupted...

[kiwidude]

Release Notes
https://github.com/kiwidude68/calibr...fiction-v1.7.2

[MegAnne]

Just wondering... Did Fantastic Fiction stopped working for everyone or is it just me?

[kiwidude]

Originally Posted by MegAnne

Just wondering... Did Fantastic Fiction stopped working for everyone or is it just me?

I just took a quick look now, in the calibre download log it is showing a 403 Forbidden error. So they have changed something to break things. I will try to take a look this weekend.

[kiwidude]

I have taken a look at this now. It appears FF have wound up their anti-scraping measures, and I don't have the time to figure out a workaround at this point. If someone else wants to take a look at it I will happily except a fix.

There are two changes I have made in the attached version to bring the previous plugin in line with what I can see that differs:

1. I added the searchstr: prefix in the query and some extra values to the end of it. Other than that the url looks identical outside of encoding (which should not matter).

2. I added a Referer header in the json request. Without it we were getting redirected to a 403 Forbidden page where it says your IP has been blocked and you must contact support. I don't believe they are "actually" blocking the IP though on the basis that querying through the website page still works fine.

With those two changes it is no longer redirecting to the 403 forbidden page - but equally it returns a blank result rather than the expected json. So at this point my available time to tinker has run out.

This is what I can tell someone willing to give this a look to help you get started:
- FF has always been a difficult site to scrape. The reason being that it doesn't return search results directly as an html page. Instead it returns a placeholder html /search page that executes javascript to retrieve a json document. It then parses that json and decides whether to transform that into html as a list of matches for the user to choose from, or if just a single match then redirects straight to that page.
- I found it useful in Notepad++ to use the JSTools plugin which can reformat minified javascript to make it readable.
- pop-search.014.min.js at the bottom of that script contains the json query that gets constructed where it starts with:

ff.loadJson("/dbs/books2?q.parser=structured&q=(and db:'FF' searchstr:'" + escape(keywords) ...

- ff.011.min.js contains the ff.loadJson function, which is obfuscated to another function:

function h(e, t, n)

which uses XMLHttpRequest to retrieve the JSON and parse it.

So putting it together, a search for "61 hours" results in hitting this URL:

https://www.fantasticfiction.com/dbs/books2?q.parser=structured&q=(and%20db:%27FF%27%20searchstr:%2761%20hours%27)&start=0&size=20&sort=visits_uk%20desc&return=booktype,title,atitle,vtitle,year,pfn,hasimage,authorsinfo,seriesinfo,db,imageloc,imageurl_amazon,imageurl_amazonuk,imageurl_amazonca,genrepage,series_links,vtitlecountry,hidevtitle

which should return this response:

{"status":{"rid":"l4uHja8zm+OBAQrUR7k=","time-ms":0},"hits":{"found":1,"start":0,"hit":[{"id":"w253217","fields":{"booktype":"1","title":"61 Hours","pfn":"c/lee-child/61-hours.htm","year":"2010","authorsinfo":"c/lee-child|Lee Child|15807|FF","genrepage":["T"],"series_links":["/c/lee-child/jack-reacher/"],"seriesinfo":"Jack Reacher|14","imageurl_amazon":"https://m.media-amazon.com/images/I/51PdZTNGZ5L._SL500_.jpg","imageurl_amazonuk":"https://m.media-amazon.com/images/I/41UR4mMa8CS._SL500_.jpg","imageurl_amazonca":"https://m.media-amazon.com/images/I/51PdZTNGZ5L._SL500_.jpg","db":["FF"]}}]}}

Ive tried replicating the query in Postman, trying to line up what I thought were the most important headers and get the same behaviour I see when run from calibre - 403 forbidden without the Referer header, and blank response with it.

So there must be something else needing to be passed or some other difference I haven't spotted. Anyone up for the challenge of figuring it out?