![]() |
#1 |
Connoisseur
![]() Posts: 94
Karma: 40
Join Date: Mar 2020
Location: Belgium (sorry, I am from the Walloon side of the country and I speak french only)
Device: PW3, Kobo Libra H2O
|
[Metadata Source Plugin] noosfere DB
[Metadata Source Plugin] noosfere DB
This plugin gets the metadata for books from the noosfere site. The noosfere site The noosfere site says in french: "nooSFere est une association régie par la loi de 1901, à but non lucratif. Elle a pour but de promouvoir la science-fiction parue en langue francaise, essentiellement au travers de ce site web." This translate to: "nooSFere is a non-profit association governed by the law of 1901. Its aim is to promote science fiction published in the French language, mainly through this website." The noosfere site is NOT selling any books, it is a huge source of information about books and volumes (a volume being a particular edition of the book where isbn, cover, publisher may be different between the volumes of a same book) Note: this site is written in French about books/volumes written in French for French speaking readers. Main Features I choose to include in the volume's comment, the resume (quatrième de couverture), the critics and the table of content (sommaire). Optionally, one can include in the comments some other information such as awards attached to the book, film made out of the books, etc.... (provided the info is available. ![]() I choose to have all link working as noosfere propose a lot more information. I find it invaluable when I build a catalogue.
Installation Notes:
Configuration: Several options to modify/tune the behaviour off the plugin.
Sources: https://github.com/lrpirlet/cal-noosfere.git Noosfere_util, a companion plugin I have developped a plugin that will allow a user of noosfere DB to choose in an internet browser the edition he best like. The plugin will write the nsfr_id of the selected lines according to the user choice. The modified lines will be put in a set whose metadata can be downloaded via noosfere DB. Finaly, if the user has requested overloading the publisher field, a utility will dispatch the information... see https://www.mobileread.com/forums/sh...00#post4208700 Since version 1.0.0, noosfere_util is less important: see explanation Last comment: This work is open source… I enjoyed writing it, if you feel like this work should be retributed, choose a charity and give them some with a mention like "Thanks to Louis Richard" or "Merci à Louis Richard" or some similar in your language. That will build up my (unpublished) reputation… �� What charity? My worse nightmares involve fire, so I give for deeply burn children… My wife is afraid of cancer so she gives to cancer research, we both feel bad about starving people so we give to "resto du Coeur"… Sadly, there is always someone who needs help and who will not be able to pay back (except maybe with a thought for the unknown who helped). So there is plenty of choice. En Français Spoiler:
Version History: Spoiler:
The zip Last edited by lrpirlet; 05-12-2023 at 11:17 AM. Reason: Announce a new version |
![]() |
![]() |
![]() |
#2 |
Connoisseur
![]() Posts: 94
Karma: 40
Join Date: Mar 2020
Location: Belgium (sorry, I am from the Walloon side of the country and I speak french only)
Device: PW3, Kobo Libra H2O
|
My process to dispatch information from extended publisher
Be careful! you're making a mistake? you don't understand me? that's not my problem...
Make a good backup before destroying everything... You have been warned... There is no official way to populate user-added columns from a source Metadata extension. To stay on the road, I decided to give the possibility to extend the field publisher with complementary information. If we choose this option we will have publisher§collection€seriel-code in the publisher column. To use this option, I created a column "collection" (Lookup name: #collection, type: Text, column shown in the Tag browser) "coll_srl" (Lookup name: #coll_srl, type: Long text, like comments, not shown in the Tag browser interpret this column as: Short text, like a title) After the metadata research work, we can rectify by hand ... OR use edit metadata in bulk .. !!!.backup.!!! preparation: use edit metadata in bulk 'Find and replace' tab search field: publisher Search for: .*§(.*)€(.*) Replace with: \2 destination field: #Coll_srl HERE WE SAVE and we give a name easy to find such as: "from publisher to #coll_str (separator § €)" 'Find and replace' tab search field: publisher Search for: (.*)§(.*) Replace with: \ 1 Destination field: Publisher HERE WE SAVE and we give a name easy to find such as: "from publisher to publisher (we erase everything from €)" 'Find and replace' tab search field: publisher Search for: .*§(.*) Replace with: \1 Destination field: #collection HERE WE SAVE and we give a name easy to find such as: "from publisher to #collection (separator § )" 'Find and replace' tab search field: publisher Search for: (.*)§(.*) Replace with: \ 1 Destination field: Publisher HERE WE SAVE and we give a name easy to find such as: "from publisher to publisher (we erase everything from §)" We can now take action. Did you make a backup? no? do it... in Calibre search do search § and € then verify that what we want to rectify is selected and nothing else edit metadata in bulk 'Find and replace' tab copy a line from a "Book" box to the "Your test" box from Load search/replace: choose "from publisher to #coll_srl (separator § €)" check that the expected result is indeed the one obtained under testing the result ... Apply ditto with "from publisher to publisher (we erase everything from €)" in Calibre search do search § then verify that what we want to rectify is selected and nothing else edit metadata in bulk 'Find and replace' tab copy a line from a "Book" box to the "Your test" box from Load search/replace: choose "from publisher to #collection (separator §)" check that the expected result is indeed the one obtained under testing the result ... Apply ditto with "from publisher to publisher (we erase everything from §)" we restore the backup if we screwed up (but we didn't screw up... yeahhh...) |
![]() |
![]() |
Advert | |
|
![]() |
#3 |
Connoisseur
![]() Posts: 94
Karma: 40
Join Date: Mar 2020
Location: Belgium (sorry, I am from the Walloon side of the country and I speak french only)
Device: PW3, Kobo Libra H2O
|
Problem-Solution... Evolution of the plugin
Code:
SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)
Thinks I have been trying to allow the reader to choose what volume to select, that I could NOT solve. I tried to make the pluggin to display all the volumes... When Calibre finds several metadata about a book, then those metadata do get merged, even if nfsr_id is different... (git branch push_top_vol_indx-from-worker-to-__init__ in the sources) I tried to make the pluggin display the volumes presentation page to allow the reader to choose the volume... that does not seems possible in the metadata plugin context (git branch interractive_mode_via_spawn_and_que in the sources)
Set the nsfr_id to point to the desired url... see entry number #8 A new version (V 0.9.5) of this plugin replaced v0.9.3. It is available here while testing goes. This will be removed when I publish version 1.0.0. I feel it is needed as testing proved that refactoring of choice was a necessity (and a few bugs fixed... and some improvements added). A new version of this plugin (v 0.9.6) corrects the algoritm for the publication date.
Now, to add or modify an id , just copy and paste the newly selected url on the icon to the left of the ids field in the metadata editor... See the entry #13
The added functionnality in v1.0.0 wasn't working sometimes due to inconsistencies in the character case ( https://www.noosfere.org/livres/niourf.asp?NumLivre=<nbr> or https://www.noosfere.org/livres/niourf.asp?numlivre=<nbr> were used ?randomly?)... fixed I designed a better detection of the numeration of the series that will now give a number when several omnibus editions exists in a series. vol_serie_seq : (omn1) ==> 0.1 vol_serie_seq : 2a ==> 2.01 vol_serie_seq : 2bis ==> 2 (sorry, I could not think of a way to show "bis" in a float expression that would be unique.) Accessorily when a book in a series is missing an arrow to point an adjacent volume, this new code bypasses the problem (https://www.noosfere.org/livres/niou...?numlivre=4757) Code is more modular, less "spaghetti" ![]() Previous versions Last edited by lrpirlet; 05-12-2023 at 11:12 AM. Reason: more infos about this pluggin |
![]() |
![]() |
![]() |
#4 |
Connoisseur
![]() Posts: 94
Karma: 40
Join Date: Mar 2020
Location: Belgium (sorry, I am from the Walloon side of the country and I speak french only)
Device: PW3, Kobo Libra H2O
|
Note
I'll do my best to answer any question, to correct any mistakes...
I wrote this in order to teach myself a new language, even if I used to be a hardware engineer understanding rather than writting software... I realize that I may code like a noob, you'll excuse me. I think it works well enough. However, I make this public as a contribution to this community. Be warned, I am not as available as I would like and may need some time in order to answer. Last edited by lrpirlet; 10-08-2021 at 04:19 AM. |
![]() |
![]() |
![]() |
#5 |
Junior Member
![]() Posts: 9
Karma: 10
Join Date: Mar 2019
Location: Paris, France
Device: Kobo Aura Edition 2
|
Hello
english : Thanks you lrpirlet !! that's THE plugin I always wanted But... I've some issues sometimes : even with the relevant ISBN (ie : the one recorded on the edition I have in the noosfere DB), it's another edition which is selected So, I propose the following improvment ![]() - to present all editions (with cover, and details already displayed) and then user can choose the editions he wants what do you think ? french: un énorme merci lrpirlet !! J'ai toujours recherché ce genre de plugin ![]() il fonctionne très bien, néanmoins, dans certains cas : le plugin me ramène une édition incorrecte (par ex, il me ramène l'éditions Pocket, alors que j'ai dans Calibre l'isbn correspondant à l'éditions Gallimard). j'ai tenté plusieurs paramétrages comme tu l'indiques en haut du thread, mais sans succès (meme en forcant l'éditeur à GALLIMARD, ca n'a pas marché) sinon, voici une évolution que j'adorerai ![]() - le plugin ramène toutes les éditions présentes dans noosfere et en plus d'afficher le détails, d'afficher la couverture - l'utilisateur peut ensuite sélectionner l'édition qu'il souhaite ![]() encore bravo et merci pour ce plugin génial Marco24 |
![]() |
![]() |
Advert | |
|
![]() |
#6 |
Junior Member
![]() Posts: 9
Karma: 10
Join Date: Mar 2019
Location: Paris, France
Device: Kobo Aura Edition 2
|
Hello,
please note that the plugin is not working today ![]() SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123) then, is it possible to also download the price ? ![]() (example : https://www.noosfere.org/livres/niou...vre=2146621053) Prix obtenus Hugo, Roman, 1966 Locus, Meilleur roman de tous les temps, 1975 Locus, Meilleur roman de tous les temps, 1987 Locus, Meilleur roman de tous les temps, 1998 Nebula, Roman, 1965 Finally, I'm a member of nossfere association perhaps can we discuss together in order to deploy API REST ? |
![]() |
![]() |
![]() |
#7 | |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,138
Karma: 78857258
Join Date: Nov 2007
Location: Toronto
Device: Libra H2O, Libra Colour
|
See this post and see if it helps
Quote:
Sent from my Pixel 4a using Tapatalk |
|
![]() |
![]() |
![]() |
#8 | |
Connoisseur
![]() Posts: 94
Karma: 40
Join Date: Mar 2020
Location: Belgium (sorry, I am from the Walloon side of the country and I speak french only)
Device: PW3, Kobo Libra H2O
|
Quote:
![]() ![]() ![]() Your proposed improvement is just NOT possible within the context of a plugin. In reality, this was exactly what I was aiming to, but I never was able to display the noosfere recap of all the volume... Probably because the python library does not allow recursivity. When i discovered that I was getting the wrong volume because the ISBN was common to several volume, I introduced a an identifier (nsfr_id) made of the book and the volume reference in the noosfere DB: bk$numeric$vl$numeric… the initial idea was to be able to access both the book and the volumes. Because I could not make an interactive choice, the need for the BK$numeric part became useless, I made no effort to keep it… VL$numeric is quite enough to uniquely refer to the volume. example: Boule de foudre de LIU Cixin... has for URL: https://www.noosfere.org/livres/niou...vre=2146606157 If I want to get the noosfere data from my plugin, I just edit in the metadata the Ids and add "nsfr_id:vl$2146606157" to get this noosfere DB entry. If Noosfere is finding the wrong one on the first try I edit the nsfr_id so that it contains the page I want. Hope it helps... |
|
![]() |
![]() |
![]() |
#9 |
Junior Member
![]() Posts: 9
Karma: 10
Join Date: Mar 2019
Location: Paris, France
Device: Kobo Aura Edition 2
|
Hi PeterT
Many thanks !! it works ![]() |
![]() |
![]() |
![]() |
#10 | |
Connoisseur
![]() Posts: 94
Karma: 40
Join Date: Mar 2020
Location: Belgium (sorry, I am from the Walloon side of the country and I speak french only)
Device: PW3, Kobo Libra H2O
|
Quote:
Since I am back from holidays in front of my real "big" computer, I would like to try to help you. But you got to help me by giving me the reference of the metadata that "fails" you. I may be able to provide a solution, but I need to get facts. What books (give me an URL in noosfere) fails to return the correct edition when you force the editor to "GALLIMARD"? In French Spoiler:
|
|
![]() |
![]() |
![]() |
#11 |
Junior Member
![]() Posts: 9
Karma: 10
Join Date: Mar 2019
Location: Paris, France
Device: Kobo Aura Edition 2
|
Hi lrpirlet,
In french it would by easier for both of us ![]() je n'ai pas pu retrouver l'ex avec l'éditeur forcé car j'ai entre temps trouvé un moyen de contournement (qui consiste à renseigner dans Calibre, champ ids nsfr_id:bk$aaaa$vl$bbbbbbb en prenant aaaa la réf de l'oeuvre dans l'url noosfere et bbbbbbb la réf de l'édition) MAIS j'ai trouvé un autre ex qui pourrait peut-être t'intéresser : en recherchant Le Monde Lavalite de Philip José FARMER, isbn 2-266-03637-8 : https://www.noosfere.org/livres/Edit...p?numitem=5085 dans noosfere, j'ai bien 3 résultats (dans cet ex, les couv sont identiques, mais j'avais déjà identifié des isbn identique mais avec des couv différentes, c'est trivial, mais j'y attache son importance LOL) même en paramétrant le plugin sur l'oeuvre la plus récente (ou d'autres paramètres), le plugin me récupère toujours l'édition de Janvier 1991 et non celle de Décembre 1997 en fait, d'après les logs, en fait, il ne récupère qu'une seule édition, et non les 3 portant le même isbn : Code:
Running identify query with parameters: {'title': 'Le monde Lavalite', 'authors': ['Philip José Farmer'], 'identifiers': {'isbn': '2266036378', 'goodreads': '18191754'}, 'timeout': 30} Using plugins: noosfere DB (0, 9, 0) The log from individual plugins is below ****************************** noosfere DB (0, 9, 0) ****************************** Found 1 results Downloading from noosfere DB took 2.3268990516662598 --- Title : Le Monde Lavalite Author(s) : Philip José Farmer Publisher : POCKET Tags : Science-Fiction Series : Les Faiseurs d'univers / la Saga des Hommes-Dieux #5 Languages : fra Published : 1986-09-14T22:00:00+00:00 Identifiers : nsfr_id:bk$5085$vl$2146597649, isbn:2266036378 sinon, un truc qui m'intéresserait serait de récupérer les infos sur les prix obtenus et les adaptations TV/films et pour finir, je suis adhérent de Noosfere : j'ai parlé de ton plugin à l'asso, du coup, pour éviter que le plugin ne fonctionne plus en cas de changement de l'interface, du module de recherche etc, ca t'intéresserait qu'on mette en place une API ? |
![]() |
![]() |
![]() |
#12 | ||
Connoisseur
![]() Posts: 94
Karma: 40
Join Date: Mar 2020
Location: Belgium (sorry, I am from the Walloon side of the country and I speak french only)
Device: PW3, Kobo Libra H2O
|
I guess you did NOT delete the nfsr_id field... In that case, my doc is incomplete
That is absolutely true... However, I thing we should either move the thread to the french forum.... I have been very frustrated, once, to find an answer in german (I do not understand it
![]() Allow me to quickly translate your input in the next quote Quote:
Please confirm... If the id field contains any valid nfsr_id, then if will bypass EVERYTHING and go directly to the volume pointed by the nfsr_id. The resulting effect is to overwrite everything, including the title, the author and even the ISBN... Now, I think that the algorithm that chooses the volume should be better documented. I was aiming to include a maximum of information about the book, and that implies choosing the "best" volume, with some (maybe too) small tuning via the noosfere DB configuration... Here is the algorithm: Code:
def ret_top_vol_indx(self, url, book_title): # cette fonction reçoit l'URL du livre qui contient plusieurs volumes du même auteur, # dont certains ont le même ISBN et généralement le même titres. # # Ces volumes diffèrent par l'éditeur, la date d'édition ou de réédition, l'image de couverture, le 4me de couverture, la critique. # MON choix se base sur un système de points sur les indications du site # résumé présent: r 1pt # critique présente: c 1pt # semble pas trop correct car CS n'existe pas même si, quand # critique de la série cs 1pt # une critique existe, elle est reprise pour tous les volumes # sommaire des nouvelles présentes: s 1pt # information vérifiée v 1pt # titre identique t 1pt # image présente p 1pt # isbn présent i 50pt sauf préférence # isbn présent et identique a calibre 100pt sauf préférence # le nombre de point sera augmenté de telle manière a choisir le volume chez l'éditeur le plus représenté... MON choix # en cas d'égalité, le plus ancien reçoit la préférence # # This gets the book's URL, there many volume may be present with (or not) same ISBN, same title. # if the book only has one volume, then we bypass ret_top_vol_indx # # the volumes are different by the publisher, edition's or re-edition's date, cover, resume, critic... # MY choice is based on a point system based on the site's flag # resume available: r 1pt # critic available: c 1pt # maybe incorrect as sometimes, when a critic exists # series critic: cs 1pt # it is distributed to all volume without indication # summary of novel in the book: s 1pt # verified information v 1pt # same title as requested t 1pt # cover available p 1pt # isbn available i 50pt unless overwritten by the priority choice # isbn available et same as requested 100pt unless overwritten by the priority choice # the score will be increased so that the volume will be chosen to the most present publisher ... MY choice # in case of equality the oldest win # Quote:
If the association agrees to let me integrate more information, I see only one problem: will I have the time? Translated with www.DeepL.com/Translator (free version) |
||
![]() |
![]() |
![]() |
#13 |
Connoisseur
![]() Posts: 94
Karma: 40
Join Date: Mar 2020
Location: Belgium (sorry, I am from the Walloon side of the country and I speak french only)
Device: PW3, Kobo Libra H2O
|
New functionalities - Nouvelles fonctionnalités
It seems I was missing some functionality by not providing 2 methods:
BOTH methods are now implemented...
This made noosfere_util a little less necessary, but if I want to distribute the information in the editor, or if I want to clean up the metadata before uploading it (some fields are never deleted, noosfere_util does it). I can always use the noosfere_util web browser to select the exact volume I want... En Français... Il semble qu'il manquait de certaines fonctionnalités en ne fournissant pas 2 méthodes :
Cela a rendu noosfere_util un peu moins nécessaire, mais si je veux distribuer l'information dans l'éditeur, ou si je veux nettoyer les métadonnées avant de les télécharger (certains champs ne sont jamais supprimés, noosfere_util le fait). Je peux toujours utiliser le navigateur web noosfere_util pour sélectionner exactement le volume que je veux... Last edited by lrpirlet; 10-31-2022 at 05:03 PM. |
![]() |
![]() |
![]() |
Thread Tools | Search this Thread |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
[Metadata Source Plugin] Baen | kiwidude | Plugins | 4 | 10-16-2022 10:55 AM |
[GUI Plugin] Noosfere_util, a companion plugin to noosfere DB | lrpirlet | Plugins | 2 | 08-18-2022 03:15 PM |
Read a book's metadata in a Metadata source plugin? | J-H | Development | 2 | 03-30-2021 09:08 AM |
[Metadata Source Plugin] Empty Plugin? (Fake Identifier) | mneimeyer | Plugins | 3 | 11-11-2019 08:07 PM |
[Metadata Source Plugin] Aladin.co.kr (KO) | sseeookk | Plugins | 1 | 04-10-2014 08:34 PM |