Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 11-04-2018, 07:57 PM   #1
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 300
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
[Metadata Source Plugin] wikidata

This plugin attempts to find a book in Wikipedia's wikidata and download metadata.

Features supported in version 1.1:
  • Search wikidata database by author, title and combinations of author and title, isbn, gutenberg ID, and wikidata ID
  • Import first publication date, gutenberg ID
  • Import several properties (instance, genre) as tags
  • Automatic detection and conversion of Overdrive linked gutenberg IDs
  • Link wikidata ID to make browsing of additional wikidata info easy
  • Link gutenberg ID to gutenberg website for easy book import
  • After finding a book, offer images linked in wikidata as covers

Constructive criticism for this plugin would be greatly appreciated.
No new releases are planned for this plugin unless features are requested.

Examples of books that are in wikipedia but are not found by this plugin (or have metadata you want to import) are very welcome!

Version History:
Spoiler:

Version 1.1.0 - 15 Nov 2018
This will be the last release of this until someone asks for a new feature (or finds a bug).

Added cover download capability
Add option to save translated or found gutenberg IDs
Add option to use wikidata Q codes or descriptions for instance/genre tags
- existing calibre filtering of tags works well with this

bugs fixed:
fixed crash on book search with unknown author
Use all gutenberg IDs found in wikidata
removed internal commas from tags

bugs:
Only finds first overdrive gutenberg ID attached to a book in existing metadata
(no intent to fix this unless someone complains, it works well enough)

Version 1.0.0 - 12 Nov 2018

Features added:
Save wikidata "instance of" and "genre" properties as tags (optionally)
support gutenberg book IDs (finding, searching by, saving)
additional more accurate searches
inexact searches with better sorting (ie., author words, title words)
support for languages other than english
use more inclusive wikidata literature and publication types
add support for keep_dup

Bugs fixed:
better formatting of exception errors
finds more books successfully

Version 0.5.0 - 9 Nov 2018
New features:
Add support for saving and using the wikidata ID
add options: slow search, ignore wikidata, language(default=en)
much improved book hit rate:
search for all subclasses of written works instead of just books
search for alternate titles too
find books without publication dates too

Bugs fixed:
Display the actual title and authors found
improved logging of search attempts
better selection of correct relevant book from fuzzy matches

Version 0.1.0 - 4 Nov 2018
Initial version, pubdate only, limited search options


Known bugs:
Spoiler:
  • Calibre only keeps the first book returned if multiple are found (feature request needed to fix this, bug #1802293 )
  • Only the first gutenberg ID from Overdrive is used for matching, but all of them found in wikidata will be imported.
  • Sometimes finds works that are not written works when using inexact searches (so examine results carefully if you turn this on!)


Possible future features:
No work is planned on any of these unless someone asks for them.
Spoiler:
  • add additional wikidata property fields (see links in forum thread below)
  • add rare date fields ("first inception", "first performance" (play))
  • remove subtitles from titles during title search to try to find more titles successfully
  • Other wikidata properties (publisher, , series, number of pages, etc.)
  • other identifiers and handles to link for them: LoC, NLA, gutenberg, BNE_ID, Encyclopaedia Britannica online, Freebase_ID, etc.)
  • prioritize found books by wikidata media type, and blacklist some (i.e., songs)
  • further refinement of book search is possible (but is it necessary?)
Attached Files
File Type: zip Wikidata.zip (43.2 KB, 156 views)

Last edited by compurandom; 11-16-2018 at 01:11 AM. Reason: new release
compurandom is offline   Reply With Quote
Old 11-05-2018, 12:06 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 33,617
Karma: 10209576
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I have added it to the index, however it will not show up in calibre because the plugin files are ina sub-directory in the zip file, they should be at the top level.
kovidgoyal is offline   Reply With Quote
Advert
Old 11-05-2018, 12:43 AM   #3
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 300
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
Quote:
Originally Posted by kovidgoyal View Post
I have added it to the index, however it will not show up in calibre because the plugin files are ina sub-directory in the zip file, they should be at the top level.
Thanks, fixed, let me know if I got any other parts of the formatting wrong.
compurandom is offline   Reply With Quote
Old 11-05-2018, 03:22 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 33,617
Karma: 10209576
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
the rest looks fine.
kovidgoyal is offline   Reply With Quote
Old 11-06-2018, 10:45 PM   #5
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 300
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
I just ran this on the portion of my library that should be in wikipedia.
It found 118 titles, and didn't find 87. So I have some test cases for things it doesn't find.

I suspect a lot of those are not books -- maybe poems, short stories, etc.
This currently only finds books. I'll figure out how to add the others slowly.

Next version I release will have support for saving and using the wikidata identifier.

Covers is a possibility as well.
compurandom is offline   Reply With Quote
Advert
Old 11-09-2018, 08:25 AM   #6
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 300
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
Wikidata is a rich source of metadata, for example
https://www.wikidata.org/wiki/Q1219561

http://tinyurl.com/yd3m69x6

Would anyone be interested in having more of that metadata imported?

For example, wikidata tags works as novels, plays, poems, etc.
It tags these literary works with one or more genre tags as well.
I could import this metadata either into the existing tags column or a user defined column.

Is there other metadata that people would want?

Last edited by compurandom; 11-11-2018 at 01:01 PM.
compurandom is offline   Reply With Quote
Old 11-16-2018, 01:09 AM   #7
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 300
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
The latest version now supports covers and I've tweaked the options and refined some of the searches. It finds most books on the first try now. Books in my library that it still can't find are either not in wikidata or have issues with author and title spelling (i.e., I'll fix by editing my library.)

I consider this plugin "finished" and will not develop it further unless I get feedback.
compurandom is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Metadata Source Plugin] Comicvine chewt0y Plugins 60 06-29-2018 06:26 PM
[Metadata Source Plugin] BiblioNETka.pl fenuks Plugins 3 03-09-2017 09:26 PM
[Metadata Source Plugin] Biblionet.gr wrangly Plugins 7 12-30-2016 07:20 AM
[Metadata Source Plugin] Skoob rodrigoccurvo Plugins 10 02-07-2015 08:20 AM
[Metadata Source Plugin] DIZZIE_NL Pr.BarnArt Plugins 7 08-11-2014 02:48 PM


All times are GMT -4. The time now is 05:13 AM.


MobileRead.com is a privately owned, operated and funded community.