ID types available/Custom IDs?

[StillReading]

I've just started using Calibre a few days ago, and I'm amazed by the program. I quickly came to a point where I'm having more questions than answers though, I hope this forum can help a little to remedy that (; .

I'm wondering which types of IDs (amazon, amazon_de, isbn, etc.) are available in Calibre. I couldn't find any info on that matter on the manual or on the web.

Also, I'm wondering if it's possible to add custom ID types without making changes to the core program. For instance, I'm using worldswithoutend.com as a source for meta data. This site is using a single numeric ID as an "identifier" for authors and books, and probably other data. I'd rather just have the numeric ID in the Calibre database as a reference, and somehow "build" an URL out of that data. Adding a custom column would work I guess, but that'd be a rather clunky solution.

[chaley]

Originally Posted by StillReading

I've just started using Calibre a few days ago, and I'm amazed by the program. I quickly came to a point where I'm having more questions than answers though, I hope this forum can help a little to remedy that (; .

I'm wondering which types of IDs (amazon, amazon_de, isbn, etc.) are available in Calibre. I couldn't find any info on that matter on the manual or on the web.

Not clear what you mean by "available". Identifiers are supported by metadata source plugins. If you mean "supported out of the box" to make URLs, then from looking at the code I think the answer is amazon, douban, edelweiss, google, isbndb, openlibrary, overdrive, and ozon. There are many (!) metadata sources available as plugins (see https://www.mobileread.com/forums/sho...d.php?t=118764 and scroll down to metadata sources). Most if not all of these support identifiers.

Also, I'm wondering if it's possible to add custom ID types without making changes to the core program. For instance, I'm using worldswithoutend.com as a source for meta data. This site is using a single numeric ID as an "identifier" for authors and books, and probably other data. I'd rather just have the numeric ID in the Calibre database as a reference, and somehow "build" an URL out of that data. Adding a custom column would work I guess, but that'd be a rather clunky solution.

You can use anything you want as an ID name. If you want to have it converted to URLs then you will need to make a metadata source plugin for it.

Note that "url" and "uri" are supported identifier types.

[davidfor]

You can create whatever id you want. So, you could put in "worldswithoutend:1234" and calibre will store and display it. But, to translate that to a URL, you need a Metadata Source plugin. With that, calibre can search for the book on the site and retrieve the metadata and cover. Creating a metadata source plugin isn't particularly hard, especially if the site is well designed. You need some Python and HTML knowledge. If you are interested in creating one, have a look at any of the metadata source plugins listed in the plugin index. They can be used as a starting point.

The alternative is to use a URL identifier. With this, you use "url:https://worldswithoutend.com/novel.asp?id=2858". When you select the book, the details panel will show the "worldswithoutend.com" and clicking it will open the page.

[StillReading]

Thanks for all that information! All of these solutions aren't feasible for me though, unfortunately.

Let me explain, maybe the info is of interest to the development community. Background: I'm an IT professional, with a little experience in coding/scripting. I never coded anything in Python, though.

Suggested solutions:

1) code a metadata plugin: not feasible for me at all. I had a look at the code of other metadata plugins, and it seems it's not enough to adapt existing code. I'd have to build the plugin from the ground up, and learn Python (and other stuff, like how to properly use RegEx) while doing so.

2) URL/URI identifier: AFAIK, only one such identifier can be used for a book. This isn't a problem per se, but I've already "used up" that one URL/URI for many books for a different purpose. Besides, The "raw data" I'd need are the numerical IDs alone, the rest of the URL would be superflous data.

I currently have no idea for a solution/workaround. I'll probably go with the custom database field, though this makes editing data a PITA, and is probably not optimal in terms of performance. In a perfect world, adding custom IDs would be exposed via the GUI, allowing stupid users like me to simply add a new identifier (like amazon_de etc.), an URL portion, and then just add the new identifier and id number to the "ID" field.

[BetterRed]

Originally Posted by StillReading

I'll probably go with the custom database field, though this makes editing data a PITA, and is probably not optimal in terms of performance.

Where is your evidence that using a custom column is not 'optimal in terms of performance'. Why is editing a custom column a PITA, does your keyboard not have an F2 key?

Originally Posted by StillReading

In a perfect world, adding custom IDs would be exposed via the GUI, allowing stupid users like me to simply add a new identifier (like amazon_de etc.), an URL portion, and then just add the new identifier and id number to the "ID" field.

Well it would be a perfect world if you were willing to create a suitable PI to suit your particular needs.

AFAIK most (all ?) of the optional plugins that are available were developed by individuals to fulfil their particular need, in their own time at there own expense. They continue to maintain them on a likewise basis - and in many cases they willingly and enthusiastically enhance their plug-ins to meet other users needs - again in their own time and at their own expense.

It's not unknown for people to pay to have a PI developed, and there's is the Plugin ideas... thread at which you could ask if anyone's interested in doing for free in their own time and at their expense.

BR

[StillReading]

BetterRed, not sure why you're going passive-agressive on me. I was merely pointing out a problem (which may be specific to my use case), not criticizing anyone or anything in particular. I do think though that some of Calibres functionality is "hidden" behind layers of technical stuff, which makes it very hard for users like me to use Calibre to its full potential.

I'd love to have the time to learn Python and develop plugins, unfortunately I don't.

Also, I'm pretty sure the more (custom) fields you have in a database, the slower queries get. I know from experience that this is true for SQLite and others. Also, it stands to reason - the more data you have to process, the longer the runtime will be. But that's off-topic.

Anyway, I'll play around with Calibre a bit more, I'm still learning new stuff every day. Maybe I'll be able to code a simple plugin at some point. Thanks for the info about the "Plugin Ideas" area, didn't read any post there yet.

[BetterRed]

@StillReading - Your previous post came across to me as a 'whinge', you made unsubstantiated claims about performance, complained that having to enter data was a Pain in the Ass, and finished with a snide remark about how 'in a perfect world all would be the way you want it to be...". You also told us that whilst your an IT professional you're not prepared to acquire the skills to write your own plugin.

calibre does not use the SQLite data as you probably think it does.

I have a one library of ~50,000 books, there are 9 custom columns, 2 of which are long text that in some cases contain long screeds of HTML. I can switch to that library in 1-2 seconds, queries are almost instantaneous.

I have another library consisting of almost 10,000 video and audio interviews, again there are a number of custom columns, in fact that library makes use of more custom columns than it does builtin columns! The library itself is 600GB, I can switch to to that library in 1-2 seconds. If I want all the Charlie Rose interviews and the current 'book' happens to be one of his, then I can press Alt/A and all 300+ are displayed instantly.

My computer is modest a i5 with 6G of Ram with WD spinning disks, no tricks like RAM disks. The drive housing the libraries is compressed, it is watched by my AV, all files are fully indexed and wherever possible using Ifilter dlls. I cannot discern any impact on Calibres performance by having those services running.

Again you say "I'm pretty sure the more (custom) fields you have in a database, the slower queries get". But how much slower - a few microseconds, a few seconds, a few minutes. I'm pretty sure your statement is speculation, based on what are probably false assumptions.

If you wrote a script using calibres command line facilities you could build a library of say 50,000 books quite quickly. Then you could perform some empirical tests to examine your hypothesis as to calibres performance.

Failing that you could have asked how big can a library be, what impact on performance do different custom column types have, is performance impacted by the number of tags etc, etc.

Anyway I'm sorry we have gotten off on the wrong foot, its actually quite unusual for me, as others could attest - maybe I'm having a bad hair day.

Try a few things, if they don't work satisfactorily try a different idea. I had one issue when I started using calibre that bugged me. How to deal with extra material for authors and books, it took me months to come up with the solution I'm using now.

BTW I'm 40+ years in IT and calibre is probably the most flexible and extendible products of its genre (record keeping) I've come across and that's only a fraction of what it does.

BR

[PeterT]

Something to remember is that while calibre does store data in an SQL database, Kovid has mentioned it is read into memory and searches etc. are performed against the Indore data, and not the underlying database.

Also while on mega sized databases being accessed by many users simultaneously there might be performance issues, on this a single user, small database I would not be overly concerned.

[davidfor]

As the others have said, the performance of adding a column with the URL in it is probably not going to be noticeable. Adding a new id to every book might have as much affect when scrolling through a large library.

In any case, attached is a metadata source plugin for the Worlds Without Ends site. It supports searching the site and downloading the standard metadata and covers. The identifier it uses is "wwend" with the number from the URL. It will display as "WorldsWithoutEnds" in the details pane.

I have done some testing, but it needs more testing before release. There is a good chance that I have missed something. If there is a problem, press the "View log" button and then "Copy to clipboard" on the next dialog. Post the contents so I can see what went wrong.

And of course, tell me if it all works fine.

[StillReading]

davidfor, thank you so much for this! Will give the plugin a spin later and report any issues.

EDIT: checked a few books, working fine so far! Even "series" data is being retrieved, which is great. Will test more books and metadata retrieval and report again. Question: do you think it would be possible to retrieve "Awards" and "Lists" information as well? I'm happy with the current functionality, Awards and Lists into custom fields would be the cherry on top though (; .

[davidfor]

Good to hear it is working.

The other information is a bit harder. The metadata source plugins can only update the core metadata. They can't update custom columns.