[Metadata Source Plugin] DriveThruRPG

[memory_dump]

I wrote this plugin to bring some semblance of order to my nearly 2k TTRPG PDF collection. A lot of those PDFs are purchased from DriveThruRPG, and they have an API, so I thought "how hard could it be?". I'm satisfied with how it's working, and figure it was time to share.

And when I say wrote, I mean I took kiwidude's FictionDB plugin and mangled it into this.

Main Features:

• Can retrieve Title, Author(s), ISBN, Comments, Publisher, Publication Date, and Cover
• Configure whether to add DriveThruRPG item categories and filters into the book Tags
• Configure whether to add Artists, Editors and/or Contributors into the book Author(s)
• Retrieves a DriveThruRPG id, which can be used to directly jump to the web page for a specific book (from the book details pane)

I'll post a link to a GitHub repo as soon as I get that set up, so you can point and laugh at my code.

[memory_dump]

This is a placeholder post for potential future test builds

[memory_dump]

Well, apparently my karma or post count or whatever is still too low to edit messages, so here's the GitHub link in a new post.

https://github.com/quickwick/driveth...calibre-plugin

[Epsilon Rose]

This plugin seems like it could be really useful for adding a bunch of TTRPG material, but there are a few things I'm unclear on and a few requests I'd like to make.

First, you said that it's possible to add DriveThruRPG item categories and filters, but what exactly does that cover? Whatever they cover, downloading the meta data seems to pull in a lot of duplicate and not particularly helpful tags, so it might be nice if we could have a bit more fine grained control over what gets imported, at the settings level. I'm hoping there might be a way to clean up some of the manual cleanup that way.

Beyond that, it would also be really useful if there was a defined way to include the rule system in such a way that you could tell it's a rule system without comparing it to a list of rule systems. The goal would be to have some way to easily sort or filter by rule system, just like you can sort and filter by author and publisher. My naive suggestion would be co-opting the Series field, since it's a dedicated field and TTRPGs are unlikely to make use of it, but I'm rather new to Calibre and I don't know if something else, like structured tags, would be better.

By the same token, would it be possible to automatically add a TTRPG tag to any content that has its meta data downloaded, so it would be easier to separate out TTRPG material from regular books.

Finally, the whole process seems very manual, especially if you need to find the DriveThruRPG ID, manually copy it into the Id field, run the download for a single file, and then manually clean up the tags field. (And it's even worse if you can't copy and paste, for unrelated reasons). Is there any way to automate the process a bit more, or at least make it a bit less individual?

[memory_dump]

First off, thanks for the feedback. It's good to know at least one person out there has tried it.

Originally Posted by Epsilon Rose

First, you said that it's possible to add DriveThruRPG item categories and filters, but what exactly does that cover? Whatever they cover, downloading the meta data seems to pull in a lot of duplicate and not particularly helpful tags, so it might be nice if we could have a bit more fine grained control over what gets imported, at the settings level. I'm hoping there might be a way to clean up some of the manual cleanup that way.

There are two fields included in the item metadata returned from the DriveThruRPG API that include tag-like entries: 'Categories' and 'Filters'. I added options in the plugin settings to include the contents of one or both into the Tags. In this case, we're entirely at the mercy of what's included in the API output. I agree that they are sometimes not particularly helpful, which is why I made them optional. Do you have some specific suggestions about how to include fine-grained control?

Originally Posted by Epsilon Rose

Beyond that, it would also be really useful if there was a defined way to include the rule system in such a way that you could tell it's a rule system without comparing it to a list of rule systems. The goal would be to have some way to easily sort or filter by rule system, just like you can sort and filter by author and publisher. My naive suggestion would be co-opting the Series field, since it's a dedicated field and TTRPGs are unlikely to make use of it, but I'm rather new to Calibre and I don't know if something else, like structured tags, would be better.

I agree, it would be useful. My solution for this was to add two custom columns to my RPG Calibre library. One column is of the Type "Text, but with a fixed set of permitted values". I populated that custom column with values that make sense to me based on the items in my library: Adventure, Art, Bestiary, Character Sheet, Map, Quickstart, Reference, Rulebook, and Supplement. After I add a new item, I just click the column and pick the appropriate value from the drop-down. The second custom column is of Type "Comma separated text, like tags" and I use that one to enter System names (like '5e D&D', 'DCC RPG', 'Traveller', 'Troika!', etc.).

Originally Posted by Epsilon Rose

By the same token, would it be possible to automatically add a TTRPG tag to any content that has its meta data downloaded, so it would be easier to separate out TTRPG material from regular books.

Maybe? Probably? I solved this by making a Calibre library just for my RPG books & materials, separate from my regular books. It's also trivial to select multiple books, Edit metadata in bulk, and add a tag to all the books.

Originally Posted by Epsilon Rose

Finally, the whole process seems very manual, especially if you need to find the DriveThruRPG ID, manually copy it into the Id field, run the download for a single file, and then manually clean up the tags field. (And it's even worse if you can't copy and paste, for unrelated reasons). Is there any way to automate the process a bit more, or at least make it a bit less individual?

For one, you don't have to acquire the DTRPG Id and copy it into the Id field, you can always try your luck with search by Title. I usually start with Title search, and then switch to manually entering the Id if Title search fails. You can also try your luck with selecting multiple titles and downloading metadata for all of them at once, but I don't know how that would interact with the "Keep more than one entry per souce" option, and frankly I'm afraid to try.

[tradjincal]

I was looking for something my use TTRPG PDF collection and I found your plugin. I think it is very interesting. I think I have the same suggestion as @Epsilon Rose.

Originally Posted by memory_dump

There are two fields included in the item metadata returned from the DriveThruRPG API that include tag-like entries: 'Categories' and 'Filters'. I added options in the plugin settings to include the contents of one or both into the Tags. In this case, we're entirely at the mercy of what's included in the API output. I agree that they are sometimes not particularly helpful, which is why I made them optional. Do you have some specific suggestions about how to include fine-grained control?

I think if we can specify tags (text or regular expression) to ignore global settings, it should be enough (for example, I can ignore all PDF tag as it comme already with format field)

Originally Posted by memory_dump

I agree, it would be useful. My solution for this was to add two custom columns to my RPG Calibre library. One column is of the Type "Text, but with a fixed set of permitted values". I populated that custom column with values that make sense to me based on the items in my library: Adventure, Art, Bestiary, Character Sheet, Map, Quickstart, Reference, Rulebook, and Supplement. After I add a new item, I just click the column and pick the appropriate value from the drop-down. The second custom column is of Type "Comma separated text, like tags" and I use that one to enter System names (like '5e D&D', 'DCC RPG', 'Traveller', 'Troika!', etc.).

Maybe you can fill the series field as game type. For example, for Catalyst Game Labs you have shadowrun 6, shadowrun 5 "series". Not sure if it can be retrieve with the api though but it can be an idea.

One more question, how do you manage titles which provide several files?

[memory_dump]

Originally Posted by tradjincal

I think if we can specify tags (text or regular expression) to ignore global settings, it should be enough (for example, I can ignore all PDF tag as it comme already with format field)

This looks doable using the built-in Metadata download feature "Create rules to transform tags/authors/publishers". You can create a rule to automatically remove specific tags.

Originally Posted by tradjincal

Maybe you can fill the series field as game type. For example, for Catalyst Game Labs you have shadowrun 6, shadowrun 5 "series". Not sure if it can be retrieve with the api though but it can be an idea.

There is a "Rule System" field on DTRPG product pages, but that information doesn't appear to be returned in the current API call I'm making. I might be able to grab it with a separate API call, though that isn't idea. Unfortunately, DTRPG has changed something on their end and the API documentation has disappeared. Until they fix that, I'm stuck with what's available from the current API call.

Originally Posted by tradjincal

One more question, how do you manage titles which provide several files?

I add each file as a separate entry in Calibre, and pull the same product information for each entry, then manually tweak the Title afterwards to indicate what the specific file is (Character Sheet, Map, etc).

[memory_dump]

edit: accidentally double-posted due to forum being weird

[Mela]

Oh, that's wonderful. I'm so happy right now. Works like a charm.

[OldeMusicke]

This plugin looks really promising for me. I have over 1800 DriveThruRPG titles. My early experiments with the plugin look great, including genre and rules in the tags in the downloaded metadata.

Calibre gives me its generic warning that metadata sources may block excessive requests. DriveThruRPG is my only enabled metadata source.

This makes me reluctant to download metadata for too many titles at a time, especially if it might disrupt or block my access to the DriveThruRPG website.

Is there a recommended size or frequency for downloading metadata? Is DriveThruRPG likely to throttle or block me?

Thanks.

[memory_dump]

Originally Posted by OldeMusicke

Is there a recommended size or frequency for downloading metadata? Is DriveThruRPG likely to throttle or block me.

The API documentation makes no mention of rate limits, but there must be some. I wouldn't try to download all 1800 at once; maybe in chunks of about 50? Staying under 60 requests per minute should be safe.

For me, the limiting factor is the quality of API search. I find I have to do one item at a time, to make sure it's connecting to the correct DTRPG record. If you can get good results for multiple items simultaneously, consider me impressed.

[OldeMusicke]

I exported my purchased library to CSV, extracted each product ID from its URL, and used Calibre’s Import List plugin to populate ID:drivethrurpg. I'm hoping that addresses a product matching problem. I tried downloading metadata for a batch of 26 and everything looks good.

[maddz]

Originally Posted by OldeMusicke

I exported my purchased library to CSV, extracted each product ID from its URL, and used Calibre’s Import List plugin to populate ID:drivethrurpg. I'm hoping that addresses a product matching problem. I tried downloading metadata for a batch of 26 and everything looks good.

How did you manage that? I have the app and there doesn't seem to be a download titles option, just download files... Even on the website there's only export files.

Short of scraping all the links from multiple pages, I don't see an automated way to do it.

[OldeMusicke]

Originally Posted by maddz

How did you manage that? I have the app and there doesn't seem to be a download titles option, just download files... Even on the website there's only export files.

Short of scraping all the links from multiple pages, I don't see an automated way to do it.

I hope I can recreate the steps correctly!

First, I ran dtrpg_convert.py. I followed the steps as listed in its comments. A small correction to the comments: It says "Run: `python dtrpg_convert.py data.json export.json`" and then "Your data is in `export.csv` file." That should actually be export.csv in the Python command line.

The CSV file contains the columns Title, Publisher, Updated, and URL. Fortunately, the URL format is completely consistent and it contains the product number, so I pulled the CSV into Excel and used the Excel formula =TEXTBEFORE(TEXTAFTER(D2,"/product/"),"/")

That gave me an additional column ("Product") with the DTRPG product number (and only the product number). I saved the result as a CSV file.

I added the Import List addon in Calibre and selected the CSV file. I found I had to change the encoding from utf-8 to Windows-1252 because one DTRPG title included odd double quotes. You might not run into this. (Side note: Later checking revealed that even though the quotes in text were funny, the product ID was correct and the downloaded metadata wasn't a problem.)

I had Delimiter set to Other with a comma in the box (because it's a CSV). I also checked "Skip first row" because I had a header row in my CSV files. I checked Unquote and Tidy, which seems to have worked out for me; I suspect whether you do or don't check Unquote and Tidy is a combination of your list of titles and your preferences.

I believe I had to hit the Preview button (in Import List) before I could take the following step.

Set the Identifier (still in Import List) to drivethrurpg.

I mapped the Import List columns as follows: Title in column 1, Authors in column 0 (because the CSV didn't include authors), Publisher in column 2, and ID:drivethrurpg in column 5 (because that's where I had the product numbers).\

The next step was to hit Next. I selected all the imported entries and clicked the "Create empty books" button. This creates books in Calibre without additional metadata.

In Calibre's Prefernces / Plugins, I made DriveThruRPG my only Metadata source. I'm hoping that has made the metadata download faster and more accurate.

I then selected a group of records and hit ctrl-D to download metadata. I used the "Download Both" button to get both the metadata and the covers.

You can then review the results and commit them. So far, I haven't run into any problems with product mismatches. I *think* that's because it's using the real DTRPG product number instead of a title search that might or might not get the right title.

Because of Calibre's warning against excessive downloads, I've done only 50 or so entries so far. I haven't run into any download problems. All the products matched correctly.

[OldeMusicke]

Originally Posted by maddz

How did you manage that? I have the app and there doesn't seem to be a download titles option, just download files... Even on the website there's only export files.

Short of scraping all the links from multiple pages, I don't see an automated way to do it.

I'm new to this forum so apparently I did something wrong in my earlier detailed reply because it vanished as soon as I tried to post it.

Trying again...

I extracted the DTRPG product number from the URL in the CSV file. I used this formula in Excel: =TEXTBEFORE(TEXTAFTER(D2,"/product/"),"/")
Of course, that's the formula that extracted from the URL in cell D2. Successive rows used D3, D4, etc.

Fortunately, the URL structure was completely consistent, so the formula was successful in retrieving the DTRPG product numbers. Note that the resulting column contained only the product numbers.

I then saved the result as a CSV file.

In Import List, I selected the CSV file. I had to hit the Preview button before I could take this next step...

I switched Match Method to Identifier: drivethrurpg. It had been Title/Author by default. The Identifier button had been grayed out until I got the Preview.

I mapped the columns (still in Import List) like this: Title in column 1, Authors in column 0, Publisher in column 2, and ID:drivethrurpg in column 5 (corresponding to how I structured the CSV file).

I clicked Next.

I selected all the entries and clicked "Create Empty Books."

I then selected a batch of entries and used ctrl-D to Download Metadata.

When prompted, I clicked "Download Both" to get the metadata and the covers.

I've done about 50 records so far. All of the products have matched correctly. I suspect that's because it can match the product number exactly instead of hoping that a title/author search gets the right thing on the first try.

Here's hoping that a) the above steps are reasonably complete, accurate, and clear, and b) this time, I don't toss my reply into the bit bucket.