[Metadata Source Plugin] Fantastic Fiction

[kiwidude]

Hi feuille, thanks for replying. Perhaps I didn't phrase my question well. I know FF only has the english version of books, no issue there. However my question was is it likely that lets say a German or Spanish user of calibre, who has only their native language translation copy of the book (i.e. non english) would be using the Fantastic Fiction metadata plugin and able to get a match? They might have a mix of english and non-english books in their library and likely not diligent about disabling the FF metadata plugin when getting data for the non-english ones.

Because the problem is that if I force the metadata for the book to be english, that will impact what calibre thinks that copy of the book is. Which to me sounds like a really bad thing to be doing.

Maybe this situation cannot happen in reality. Will a foreign language translation of a book always have a different (non english) name, so FF could not match the book. Certainly I would expect it to have different identifiers like ISBN, assuming it is not incorrectly recorded for that book.

I just don't want to make the change and then have x amount of users complaining about how their libraries are now corrupted...

[kiwidude]

Release Notes
https://github.com/kiwidude68/calibr...fiction-v1.7.2

[MegAnne]

Just wondering... Did Fantastic Fiction stopped working for everyone or is it just me?

[kiwidude]

Originally Posted by MegAnne

Just wondering... Did Fantastic Fiction stopped working for everyone or is it just me?

I just took a quick look now, in the calibre download log it is showing a 403 Forbidden error. So they have changed something to break things. I will try to take a look this weekend.

[kiwidude]

I have taken a look at this now. It appears FF have wound up their anti-scraping measures, and I don't have the time to figure out a workaround at this point. If someone else wants to take a look at it I will happily except a fix.

There are two changes I have made in the attached version to bring the previous plugin in line with what I can see that differs:

1. I added the searchstr: prefix in the query and some extra values to the end of it. Other than that the url looks identical outside of encoding (which should not matter).

2. I added a Referer header in the json request. Without it we were getting redirected to a 403 Forbidden page where it says your IP has been blocked and you must contact support. I don't believe they are "actually" blocking the IP though on the basis that querying through the website page still works fine.

With those two changes it is no longer redirecting to the 403 forbidden page - but equally it returns a blank result rather than the expected json. So at this point my available time to tinker has run out.

This is what I can tell someone willing to give this a look to help you get started:
- FF has always been a difficult site to scrape. The reason being that it doesn't return search results directly as an html page. Instead it returns a placeholder html /search page that executes javascript to retrieve a json document. It then parses that json and decides whether to transform that into html as a list of matches for the user to choose from, or if just a single match then redirects straight to that page.
- I found it useful in Notepad++ to use the JSTools plugin which can reformat minified javascript to make it readable.
- pop-search.014.min.js at the bottom of that script contains the json query that gets constructed where it starts with:

ff.loadJson("/dbs/books2?q.parser=structured&q=(and db:'FF' searchstr:'" + escape(keywords) ...

- ff.011.min.js contains the ff.loadJson function, which is obfuscated to another function:

function h(e, t, n)

which uses XMLHttpRequest to retrieve the JSON and parse it.

So putting it together, a search for "61 hours" results in hitting this URL:

https://www.fantasticfiction.com/dbs/books2?q.parser=structured&q=(and%20db:%27FF%27%20searchstr:%2761%20hours%27)&start=0&size=20&sort=visits_uk%20desc&return=booktype,title,atitle,vtitle,year,pfn,hasimage,authorsinfo,seriesinfo,db,imageloc,imageurl_amazon,imageurl_amazonuk,imageurl_amazonca,genrepage,series_links,vtitlecountry,hidevtitle

which should return this response:

{"status":{"rid":"l4uHja8zm+OBAQrUR7k=","time-ms":0},"hits":{"found":1,"start":0,"hit":[{"id":"w253217","fields":{"booktype":"1","title":"61 Hours","pfn":"c/lee-child/61-hours.htm","year":"2010","authorsinfo":"c/lee-child|Lee Child|15807|FF","genrepage":["T"],"series_links":["/c/lee-child/jack-reacher/"],"seriesinfo":"Jack Reacher|14","imageurl_amazon":"https://m.media-amazon.com/images/I/51PdZTNGZ5L._SL500_.jpg","imageurl_amazonuk":"https://m.media-amazon.com/images/I/41UR4mMa8CS._SL500_.jpg","imageurl_amazonca":"https://m.media-amazon.com/images/I/51PdZTNGZ5L._SL500_.jpg","db":["FF"]}}]}}

Ive tried replicating the query in Postman, trying to line up what I thought were the most important headers and get the same behaviour I see when run from calibre - 403 forbidden without the Referer header, and blank response with it.

So there must be something else needing to be passed or some other difference I haven't spotted. Anyone up for the challenge of figuring it out?

[Nightliss]

Hey @kiwidude.

I was able to get the results in the Postman by adding `Cookie` header with `aws-waf-token` from the browser search request. Hope this helps coz I'm missing this source a lot.

[kiwidude]

Originally Posted by Nightliss

Hey @kiwidude.

I was able to get the results in the Postman by adding `Cookie` header with `aws-waf-token` from the browser search request. Hope this helps coz I'm missing this source a lot.

Nice find Nightliss, indeed that seems to be the missing link. The next problem being how to generate that cookie value - pasting in a value from my own browser will only work for 4 days until it expires

For those others wondering what is going on - it is pretty well documented if you google "aws-waf-token" that it is an anti-scraping measure to protect against botnets. From my quick look it seems there is some javascript that would be executing on the page when you visit it as a "human" which is responsible for returning the cookie to say yup, you are ok to proceed. That then gets sent with your subsequent requests to the site.

Right now the plugin code doesn't execute javascript on the page, so there is no ability for it to have a cookie generated.

Again this is beyond my dev capacity at the moment due to the day job to look into, if someone comes up with a workaround please feel free to reach out.

[Nightliss]

Originally Posted by kiwidude

Right now the plugin code doesn't execute javascript on the page, so there is no ability for it to have a cookie generated.

Any chance you could upload a version here where we can copy/paste the cookie from the browser in the UI/config files? I think that's much better than not having this plugin working at all. From my cursory glance it doesn't seem to be easy way to generate the token. Maybe with something like puppeteer or playwright but don't think it's viable here. Not familiar with python and calibre plugins.