[GUI Plugin] Author Watch

[Sirtel]

Originally Posted by vasbinde

Would any of you be interested in dramatically enhancing this plugin with a separate "Scan Helper" that would be a full, JavaScript capable headless browser in a docker container? It would run constantly, scanning periodically (e.g. 1x per week) to see if new books are found from multiple sources and would then add it to a REST API accessible SQL DB. The plugin would then easily, as an option, communicate with the new Scan Helper to suck down data, as opposed to direct pulls. Avoiding a lot of the JavaScript requirements for anti-scanning bots, etc., whilst caching everything so that your pulls of data are very friendly for the sites we're using as data sources.

This would require some configuration, but could be much more robust for getting complete and accurate data.

I would be interested if it solved the problem of newly released books on Amazon not turning up. And I definitely don't want to add any URLs manually, too much work with more than hundred authors.

I'm not interested in finding missing books, personally. Only new releases.

Sorry, I haven't had time to run diagnostics yet.

[sgmoore]

Originally Posted by vasbinde

Would any of you be interested in dramatically enhancing this plugin with a separate "Scan Helper" that would be a full, JavaScript capable headless browser in a docker container?

I have to be honest and say I am not sure it would be 'worth' it.
I don't find a new 'favourite' author very often and my current favourites don't release a new book every week or month, so I don't like the idea of a process continually running (and using up memory) all the time.

Avoiding a lot of the JavaScript requirements for anti-scanning bots, etc.,

You do realise that one of the things anti-scanning bots do is look to see if you are running a headless browser. How do you get around that?

[vasbinde]

Originally Posted by sgmoore

I have to be honest and say I am not sure it would be 'worth' it.
I don't find a new 'favourite' author very often and my current favourites don't release a new book every week or month, so I don't like the idea of a process continually running (and using up memory) all the time.



You do realise that one of the things anti-scanning bots do is look to see if you are running a headless browser. How do you get around that?

There are certain "stealth" techniques that can help a little. That said, I would LOVE to have a normalized, API accessible list of Author ebooks, with book language, Title, Series, etc. all enumerated and available either free of charge or for a nominal fee. That would make this process a LOT easier and avoid the screen scraping entirely.

[vasbinde]

If you are interested in this plugin, please note that I have been working the past weeks on a new architecture that will allow people with the desire, and the hardware, to run a full fledged mySQL DB with an external containerized tool, called "Scan Helper". This tool will do things like download the ~100GB of OpenLibrary data dumps once a month and make that available to Scan Helper so that when you add an author, none of the requests hit ANY metadata sources other than your database. Admittedly, it can go stale every 29 days, however, it does NOT hammer our various metadata sources and leverages a supported and recognized official mechanism by OpenLibrary.

In addition, all of the data from that source, and others is exposed via a REST API that the plugin can pull down easily.

This works for my use case as I have the hardware to spare for this effort, however, I know that very few people want that extensive of an architectural model. As such, the original, standalone method will be improved, although it will be limited due to the blocking involved.

I am also considering putting in a limiter into the config of the plugin when it's in standalone mode, limiting the plugin to no more than X amount of queries per day per source. The default would be 25-50: I have not decided on a number yet. The idea being that the plugin, even in standalone mode, heavily caches data so as to allow you to build up your own database of information about your favorite authors, only really querying the metadata sources to see if anything new is there and ONLY downloading new works, rather than everything all over again.

The goal is to be as kind as possible to these sources, while still filling my own unmet need.

I'll publish the updates when they are ready.

I hope you are all doing well!