MobileRead Forums - View Single Post

vasbinde · 08-31-2026, 01:55 AM

If you are interested in this plugin, please note that I have been working the past weeks on a new architecture that will allow people with the desire, and the hardware, to run a full fledged mySQL DB with an external containerized tool, called "Scan Helper". This tool will do things like download the ~100GB of OpenLibrary data dumps once a month and make that available to Scan Helper so that when you add an author, none of the requests hit ANY metadata sources other than your database. Admittedly, it can go stale every 29 days, however, it does NOT hammer our various metadata sources and leverages a supported and recognized official mechanism by OpenLibrary.

In addition, all of the data from that source, and others is exposed via a REST API that the plugin can pull down easily.

This works for my use case as I have the hardware to spare for this effort, however, I know that very few people want that extensive of an architectural model. As such, the original, standalone method will be improved, although it will be limited due to the blocking involved.

I am also considering putting in a limiter into the config of the plugin when it's in standalone mode, limiting the plugin to no more than X amount of queries per day per source. The default would be 25-50: I have not decided on a number yet. The idea being that the plugin, even in standalone mode, heavily caches data so as to allow you to build up your own database of information about your favorite authors, only really querying the metadata sources to see if anything new is there and ONLY downloading new works, rather than everything all over again.

The goal is to be as kind as possible to these sources, while still filling my own unmet need.

I'll publish the updates when they are ready.

I hope you are all doing well!

08-31-2026, 01:55 AM	#19
vasbinde Enthusiast Posts: 33 Karma: 10 Join Date: Jan 2008 Device: Kobo Libra Colour / iPad Pro / Various Phones	Not Vanished - Heads down on Scan Helper If you are interested in this plugin, please note that I have been working the past weeks on a new architecture that will allow people with the desire, and the hardware, to run a full fledged mySQL DB with an external containerized tool, called "Scan Helper". This tool will do things like download the ~100GB of OpenLibrary data dumps once a month and make that available to Scan Helper so that when you add an author, none of the requests hit ANY metadata sources other than your database. Admittedly, it can go stale every 29 days, however, it does NOT hammer our various metadata sources and leverages a supported and recognized official mechanism by OpenLibrary. In addition, all of the data from that source, and others is exposed via a REST API that the plugin can pull down easily. This works for my use case as I have the hardware to spare for this effort, however, I know that very few people want that extensive of an architectural model. As such, the original, standalone method will be improved, although it will be limited due to the blocking involved. I am also considering putting in a limiter into the config of the plugin when it's in standalone mode, limiting the plugin to no more than X amount of queries per day per source. The default would be 25-50: I have not decided on a number yet. The idea being that the plugin, even in standalone mode, heavily caches data so as to allow you to build up your own database of information about your favorite authors, only really querying the metadata sources to see if anything new is there and ONLY downloading new works, rather than everything all over again. The goal is to be as kind as possible to these sources, while still filling my own unmet need. I'll publish the updates when they are ready. I hope you are all doing well!