[GUI Plugin] Author Watch

[Sirtel]

Originally Posted by vasbinde

Would any of you be interested in dramatically enhancing this plugin with a separate "Scan Helper" that would be a full, JavaScript capable headless browser in a docker container? It would run constantly, scanning periodically (e.g. 1x per week) to see if new books are found from multiple sources and would then add it to a REST API accessible SQL DB. The plugin would then easily, as an option, communicate with the new Scan Helper to suck down data, as opposed to direct pulls. Avoiding a lot of the JavaScript requirements for anti-scanning bots, etc., whilst caching everything so that your pulls of data are very friendly for the sites we're using as data sources.

This would require some configuration, but could be much more robust for getting complete and accurate data.

I would be interested if it solved the problem of newly released books on Amazon not turning up. And I definitely don't want to add any URLs manually, too much work with more than hundred authors.

I'm not interested in finding missing books, personally. Only new releases.

Sorry, I haven't had time to run diagnostics yet.

[sgmoore]

Originally Posted by vasbinde

Would any of you be interested in dramatically enhancing this plugin with a separate "Scan Helper" that would be a full, JavaScript capable headless browser in a docker container?

I have to be honest and say I am not sure it would be 'worth' it.
I don't find a new 'favourite' author very often and my current favourites don't release a new book every week or month, so I don't like the idea of a process continually running (and using up memory) all the time.

Avoiding a lot of the JavaScript requirements for anti-scanning bots, etc.,

You do realise that one of the things anti-scanning bots do is look to see if you are running a headless browser. How do you get around that?

[vasbinde]

Originally Posted by sgmoore

I have to be honest and say I am not sure it would be 'worth' it.
I don't find a new 'favourite' author very often and my current favourites don't release a new book every week or month, so I don't like the idea of a process continually running (and using up memory) all the time.



You do realise that one of the things anti-scanning bots do is look to see if you are running a headless browser. How do you get around that?

There are certain "stealth" techniques that can help a little. That said, I would LOVE to have a normalized, API accessible list of Author ebooks, with book language, Title, Series, etc. all enumerated and available either free of charge or for a nominal fee. That would make this process a LOT easier and avoid the screen scraping entirely.