Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 09-16-2014, 06:11 AM   #1
StillReading
Junior Member
StillReading began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Sep 2014
Device: none
ID types available/Custom IDs?

I've just started using Calibre a few days ago, and I'm amazed by the program. I quickly came to a point where I'm having more questions than answers though, I hope this forum can help a little to remedy that (; .

I'm wondering which types of IDs (amazon, amazon_de, isbn, etc.) are available in Calibre. I couldn't find any info on that matter on the manual or on the web.

Also, I'm wondering if it's possible to add custom ID types without making changes to the core program. For instance, I'm using worldswithoutend.com as a source for meta data. This site is using a single numeric ID as an "identifier" for authors and books, and probably other data. I'd rather just have the numeric ID in the Calibre database as a reference, and somehow "build" an URL out of that data. Adding a custom column would work I guess, but that'd be a rather clunky solution.
StillReading is offline   Reply With Quote
Old 09-16-2014, 06:45 AM   #2
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,428
Karma: 831552
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by StillReading View Post
I've just started using Calibre a few days ago, and I'm amazed by the program. I quickly came to a point where I'm having more questions than answers though, I hope this forum can help a little to remedy that (; .

I'm wondering which types of IDs (amazon, amazon_de, isbn, etc.) are available in Calibre. I couldn't find any info on that matter on the manual or on the web.
Not clear what you mean by "available". Identifiers are supported by metadata source plugins. If you mean "supported out of the box" to make URLs, then from looking at the code I think the answer is amazon, douban, edelweiss, google, isbndb, openlibrary, overdrive, and ozon. There are many (!) metadata sources available as plugins (see http://www.mobileread.com/forums/sho...d.php?t=118764 and scroll down to metadata sources). Most if not all of these support identifiers.
Quote:
Also, I'm wondering if it's possible to add custom ID types without making changes to the core program. For instance, I'm using worldswithoutend.com as a source for meta data. This site is using a single numeric ID as an "identifier" for authors and books, and probably other data. I'd rather just have the numeric ID in the Calibre database as a reference, and somehow "build" an URL out of that data. Adding a custom column would work I guess, but that'd be a rather clunky solution.
You can use anything you want as an ID name. If you want to have it converted to URLs then you will need to make a metadata source plugin for it.

Note that "url" and "uri" are supported identifier types.
chaley is offline   Reply With Quote
 
Enthusiast
Old 09-16-2014, 06:58 AM   #3
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 5,457
Karma: 5446662
Join Date: Jul 2011
Location: Australia
Device: Kobo Touch, Kobo Glo
You can create whatever id you want. So, you could put in "worldswithoutend:1234" and calibre will store and display it. But, to translate that to a URL, you need a Metadata Source plugin. With that, calibre can search for the book on the site and retrieve the metadata and cover. Creating a metadata source plugin isn't particularly hard, especially if the site is well designed. You need some Python and HTML knowledge. If you are interested in creating one, have a look at any of the metadata source plugins listed in the plugin index. They can be used as a starting point.

The alternative is to use a URL identifier. With this, you use "url:https://worldswithoutend.com/novel.asp?id=2858". When you select the book, the details panel will show the "worldswithoutend.com" and clicking it will open the page.

Last edited by davidfor; 09-16-2014 at 07:00 AM. Reason: Beaten by chaley
davidfor is online now   Reply With Quote
Old 09-16-2014, 08:24 AM   #4
StillReading
Junior Member
StillReading began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Sep 2014
Device: none
Thanks for all that information! All of these solutions aren't feasible for me though, unfortunately.

Let me explain, maybe the info is of interest to the development community. Background: I'm an IT professional, with a little experience in coding/scripting. I never coded anything in Python, though.

Suggested solutions:

1) code a metadata plugin: not feasible for me at all. I had a look at the code of other metadata plugins, and it seems it's not enough to adapt existing code. I'd have to build the plugin from the ground up, and learn Python (and other stuff, like how to properly use RegEx) while doing so.

2) URL/URI identifier: AFAIK, only one such identifier can be used for a book. This isn't a problem per se, but I've already "used up" that one URL/URI for many books for a different purpose. Besides, The "raw data" I'd need are the numerical IDs alone, the rest of the URL would be superflous data.

I currently have no idea for a solution/workaround. I'll probably go with the custom database field, though this makes editing data a PITA, and is probably not optimal in terms of performance. In a perfect world, adding custom IDs would be exposed via the GUI, allowing stupid users like me to simply add a new identifier (like amazon_de etc.), an URL portion, and then just add the new identifier and id number to the "ID" field.
StillReading is offline   Reply With Quote
Old 09-16-2014, 05:30 PM   #5
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,410
Karma: 1957866
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by StillReading View Post
I'll probably go with the custom database field, though this makes editing data a PITA, and is probably not optimal in terms of performance.
Where is your evidence that using a custom column is not 'optimal in terms of performance'. Why is editing a custom column a PITA, does your keyboard not have an F2 key?

Quote:
Originally Posted by StillReading View Post
In a perfect world, adding custom IDs would be exposed via the GUI, allowing stupid users like me to simply add a new identifier (like amazon_de etc.), an URL portion, and then just add the new identifier and id number to the "ID" field.
Well it would be a perfect world if you were willing to create a suitable PI to suit your particular needs.

AFAIK most (all ?) of the optional plugins that are available were developed by individuals to fulfil their particular need, in their own time at there own expense. They continue to maintain them on a likewise basis - and in many cases they willingly and enthusiastically enhance their plug-ins to meet other users needs - again in their own time and at their own expense.

It's not unknown for people to pay to have a PI developed, and there's is the Plugin ideas... thread at which you could ask if anyone's interested in doing for free in their own time and at their expense.

BR
BetterRed is offline   Reply With Quote
Old 09-17-2014, 06:25 AM   #6
StillReading
Junior Member
StillReading began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Sep 2014
Device: none
BetterRed, not sure why you're going passive-agressive on me. I was merely pointing out a problem (which may be specific to my use case), not criticizing anyone or anything in particular. I do think though that some of Calibres functionality is "hidden" behind layers of technical stuff, which makes it very hard for users like me to use Calibre to its full potential.

I'd love to have the time to learn Python and develop plugins, unfortunately I don't.

Also, I'm pretty sure the more (custom) fields you have in a database, the slower queries get. I know from experience that this is true for SQLite and others. Also, it stands to reason - the more data you have to process, the longer the runtime will be. But that's off-topic.

Anyway, I'll play around with Calibre a bit more, I'm still learning new stuff every day. Maybe I'll be able to code a simple plugin at some point. Thanks for the info about the "Plugin Ideas" area, didn't read any post there yet.
StillReading is offline   Reply With Quote
Old 09-17-2014, 07:46 AM   #7
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,410
Karma: 1957866
Join Date: Mar 2012
Location: NSW Australia
Device: none
@StillReading - Your previous post came across to me as a 'whinge', you made unsubstantiated claims about performance, complained that having to enter data was a Pain in the Ass, and finished with a snide remark about how 'in a perfect world all would be the way you want it to be...". You also told us that whilst your an IT professional you're not prepared to acquire the skills to write your own plugin.

calibre does not use the SQLite data as you probably think it does.

I have a one library of ~50,000 books, there are 9 custom columns, 2 of which are long text that in some cases contain long screeds of HTML. I can switch to that library in 1-2 seconds, queries are almost instantaneous.

I have another library consisting of almost 10,000 video and audio interviews, again there are a number of custom columns, in fact that library makes use of more custom columns than it does builtin columns! The library itself is 600GB, I can switch to to that library in 1-2 seconds. If I want all the Charlie Rose interviews and the current 'book' happens to be one of his, then I can press Alt/A and all 300+ are displayed instantly.

My computer is modest a i5 with 6G of Ram with WD spinning disks, no tricks like RAM disks. The drive housing the libraries is compressed, it is watched by my AV, all files are fully indexed and wherever possible using Ifilter dlls. I cannot discern any impact on Calibres performance by having those services running.

Again you say "I'm pretty sure the more (custom) fields you have in a database, the slower queries get". But how much slower - a few microseconds, a few seconds, a few minutes. I'm pretty sure your statement is speculation, based on what are probably false assumptions.

If you wrote a script using calibres command line facilities you could build a library of say 50,000 books quite quickly. Then you could perform some empirical tests to examine your hypothesis as to calibres performance.

Failing that you could have asked how big can a library be, what impact on performance do different custom column types have, is performance impacted by the number of tags etc, etc.

Anyway I'm sorry we have gotten off on the wrong foot, its actually quite unusual for me, as others could attest - maybe I'm having a bad hair day.

Try a few things, if they don't work satisfactorily try a different idea. I had one issue when I started using calibre that bugged me. How to deal with extra material for authors and books, it took me months to come up with the solution I'm using now.

BTW I'm 40+ years in IT and calibre is probably the most flexible and extendible products of its genre (record keeping) I've come across and that's only a fraction of what it does.

BR
BetterRed is offline   Reply With Quote
Old 09-17-2014, 07:46 AM   #8
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 6,769
Karma: 43931916
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
Something to remember is that while calibre does store data in an SQL database, Kovid has mentioned it is read into memory and searches etc. are performed against the Indore data, and not the underlying database.

Also while on mega sized databases being accessed by many users simultaneously there might be performance issues, on this a single user, small database I would not be overly concerned.
PeterT is offline   Reply With Quote
Old Today, 07:34 AM   #9
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 5,457
Karma: 5446662
Join Date: Jul 2011
Location: Australia
Device: Kobo Touch, Kobo Glo
As the others have said, the performance of adding a column with the URL in it is probably not going to be noticeable. Adding a new id to every book might have as much affect when scrolling through a large library.

In any case, attached is a metadata source plugin for the Worlds Without Ends site. It supports searching the site and downloading the standard metadata and covers. The identifier it uses is "wwend" with the number from the URL. It will display as "WorldsWithoutEnds" in the details pane.

I have done some testing, but it needs more testing before release. There is a good chance that I have missed something. If there is a problem, press the "View log" button and then "Copy to clipboard" on the next dialog. Post the contents so I can see what went wrong.

And of course, tell me if it all works fine.
Attached Files
File Type: zip WorldsWithoutEnd.zip (6.2 KB, 0 views)

Last edited by davidfor; Today at 07:40 AM. Reason: Repeat after me: Remember to attach the file
davidfor is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Custom Column: Ids SoulStyle Calibre 0 10-12-2011 12:36 PM
IDs jlutes Library Management 1 08-02-2011 12:30 AM
Can custom book data be displayed in a custom column? kiwidude Development 9 03-02-2011 05:35 AM
ids down ? reaver121 iRex 0 01-16-2010 09:53 AM


All times are GMT -4. The time now is 07:56 AM.


MobileRead.com is a privately owned, operated and funded community.