[GUI-Plugin] Library Codes-SRU

[davidjoseph1]

Originally Posted by frustratedhacker

Well, you'd have some extra work, but it'd make it easier to contribute and to see what's being changed and so on. The copyright status of the code we may never know, since its original author hasn't logged in in a while. Still, the code was given open-source, for free and without mention of any license, not to mention that you've made a substantial enough change in the code to consider it partly yours. Anyways, it was just an idea. If it doesn't make sense to you or you think it'll be too much work, that's fine.

I've been away from this project for a few months and I'll take the weekend to put it up on Github if I can. I use LC in conjunction with the metadata plugin SRU because of some issues you touch on below.

Originally Posted by frustratedhacker

1. Got subject heading (LCSH) extraction working.

I would love to see your code. I haven't been able to wrap my head around ETREE and make that extraction work, and there are a bunch of different LC codes that contain alternate LCSH and Subject headings.

Originally Posted by frustratedhacker

2. I found recently that there seems to be character limit to the author + title search. Here's an example (try removing "Joel" or "Century" and you'll get a result):

http://lx2.loc.gov:210/LCDB?query=dc...imumRecords=10

That's interesting. I haven't seen that, and the underlying SRU query language doesn't mention character limits

Originally Posted by frustratedhacker

3. Sometimes, in author + title searches, less exact search terms help in finding additional records of the same book, which may have complementary data. Adding a submenu to the author + title search that opens a popup with the search terms, so the user can change them would be a nice addition. Here's an example of what I'm talking about:

http://lx2.loc.gov:210/LCDB?query=dc...imumRecords=10

Notice how I used only the author's last name, since one of the records shortens the other names while the other doesn't. That gets me a subject heading. It's not much, but in other book, tweaking the search terms got me a DDC code.

Interesting. I have had issues with multiple items returned for a single search, and LC doesn't know how to deal.

Originally Posted by frustratedhacker

What other functionalities are you interested in reviving? If it's the "DDC/LCC to genre mapping", I'm interested in that one also (mostly for LCC codes). It needs a different approach, in my opinion, but it's difficult...

Basically restoring the VIAF and LC Subject and Name Authorities lookup. There's a lot of extended value in those cross-references for further research and curiosity.

I would like to see the code for LCSH extraction!

[frustratedhacker]

Originally Posted by davidjoseph1

I would love to see your code. I haven't been able to wrap my head around ETREE and make that extraction work, and there are a bunch of different LC codes that contain alternate LCSH and Subject headings.

I spoke too soon about the LCSH extractor. It wasn't as complete as I thought. Now, after lots of testing, I think it's working pretty good, although I've made other changes which maybe you're not comfortable with. I still have to make a few changes and tests and I'll post it this weekend, most likely.

Originally Posted by davidjoseph1

That's interesting. I haven't seen that, and the underlying SRU query language doesn't mention character limits

That's right, but at the moment of posting that, that query wasn't returning anything, not even an error. When I was doing tests, days after that, it worked fine. Maybe it was a server problem.

Originally Posted by davidjoseph1

Basically restoring the VIAF and LC Subject and Name Authorities lookup. There's a lot of extended value in those cross-references for further research and curiosity.

The VIAF thing I've seen it in the code. It's just getting the author's VIAF id, right? I'm a bit lost with the LC Subject and Name Authorities lookup you mention. Can you specify where in the code is it?

For the VIAF, I discovered the viapy python library (https://github.com/Princeton-CDH/viapy), but you I saw in the code you can get a json response by changing the header in the request (https://github.com/Princeton-CDH/via...apy/api.py#L45). You can try this in the REPL:

import requests

def get_viaf_id_auto(name):
    resp = requests.get(
        "http://viaf.org/viaf/AutoSuggest",
        params={"query": name},
        headers={"accept": "application/json"}
    )
    data = resp.json()
    res = data.get("result", [])
    if not res:
        return None
    first_result = res[0]
    first_result_viafid = res[0]["viafid"]
    return first_result, first_result_viafid

first_result, first_result_viafid = get_viaf_id_auto("Jane Austen")

print("First result:", first_result, "\n")
print("First result's VIAF id:", first_result_viafid)

[frustratedhacker]

Here it is. I'll say no more but add some notes I've been writing while doing this. The second and third points are important for its functioning, since I added an external library (sorry).

• Changed files are ui.py, classify_web_service_webscraping.py and __init__.py (to add the lib folder to the path). Most changes are marked with a "### Bookmark" comment.
  
• Added a record filtering for the title and author search.
  
• The record filtering code uses the library "thefuzz" (https://github.com/seatgeek/thefuzz), so you have to install that or add it to the lib folder, if you can't/don't want to install it. I'll add another zip with the library included in the plugin (had to do it for me because I installed Calibre as a Flatpak). If you don't trust me, you can download it with pip.
  
• Changed the way the script processes the title and author to form the URL; it was failing with some titles and authors. I also made it a bit more "loose" to avoid not getting matches.
  
• "dc.identifier" isn't working now (might be a temporary problem). Changed it to "bath.isbn".
  
• Shortly after my last post I added an option to change title and author's name for the query. Might be useful to remove extra text from the title (e.g: Third Edition, Book 2, Volume III) to get matches (tip: if there's a misspelled or incomplete word in the title, you probably won't get a match).
  
• Added DDC edition extraction. It doesn't seem to bother the genre mapping function; the problem seems to be the forward slashes.
  
• Added some code to handle multi-author books. It's hacky but it seems to work. Probably needs more testing.
  
• A problem: The LCSH extractor grabs all headings from all filtered records and removes duplicates, but sometimes, two records might have slightly changed headings, which get passed to the final list of headings.

[GalacticHull]

@frustratedhacker

My primary use of the forum is for tools like this and I'm entirely ignorant of coding, making it difficult to even ask an appropriate question. So I do not often post here.

Just wanted to say thank you for taking this project up. It saves me an extraordinary amount of time and energy. Appreciated.

[frustratedhacker]

Originally Posted by GalacticHull

@frustratedhacker

My primary use of the forum is for tools like this and I'm entirely ignorant of coding, making it difficult to even ask an appropriate question. So I do not often post here.

Just wanted to say thank you for taking this project up. It saves me an extraordinary amount of time and energy. Appreciated.

Sorry for the late reply, I'm not an active user here either and I've been busy. I'm glad it's useful to you. Feedback is also a form of contribution and from you post I see it's working for you, so thanks for that.

Also, for anyone trying this, if you aren't getting results or if you are but suspect they aren't good, you can enable Calibre's debug mode to get detailed log messages and see what the plugin is doing.

[readabit]

LC comes preloaded for DDC mappings, but a template .csv file for LCC mappings is attached below.

I'm confused -- I'm not seeing where there is a template file (though maybe I'm just overlooking it )?

I'm wanting my #lc_genre field to use LCC Genres, but it sounds like that is not automatically included (and does not seem to be working for me in my initial tests with an item that does have an LCC).

All that aside, I am SO grateful you have created this plugin!! I am working toward using Calibre for my physical library (I already use it for my digital one), so I'd like to use LCC in particular with my physical books.

[davidjoseph1]

This fork of Library Codes was created because OCLC took down their open-source web portal for sharing individual records - Classify- in the summer of 2023.

This was bewildering at the time, but now I know why.
Baker and Taylor, a book wholesaler and distributor that provides library acquisition and collections management, engaged in a four year project to steal OCLC WorldCat data to create its own cataloguing service - BTCat.

Here's the complaint from OCLC that was filed in the spring.


We were caught in the crossfire of that litigation.

[readabit]

Originally Posted by davidjoseph1

This fork of Library Codes was created because OCLC took down their open-source web portal for sharing individual records - Classify- in the summer of 2023.

Is this a general update you are sharing, or is it specifically in response to my questions about how to get LCC Genres working? I'm not immediately understanding the connection if it is in response to the later, nor understanding the full implications if so. Thanks for bearing with my questions!

Originally Posted by readabit

I'm confused -- I'm not seeing where there is a template file (though maybe I'm just overlooking it )?

I'm wanting my #lc_genre field to use LCC Genres, but it sounds like that is not automatically included (and does not seem to be working for me in my initial tests with an item that does have an LCC).

All that aside, I am SO grateful you have created this plugin!! I am working toward using Calibre for my physical library (I already use it for my digital one), so I'd like to use LCC in particular with my physical books.

Originally Posted by frustratedhacker

I spoke too soon about the LCSH extractor. It wasn't as complete as I thought. Now, after lots of testing, I think it's working pretty good...

Is what's being worked on above by frustratedhacker relevant to my interest (of using LCC Genres), and if so, should I use those versions instead, or will it all be folded together into the main version (in the first post in this thread)?

[qucur]

Hi

Why doesn't this plugin work. Every time I try to use it, it gives this error:
"Nothing Found For Any Selected Book. Visit: Http://classify.oclc.org/classify2/"

Thanks in advance for reply.

[SidJnsn]

The plugin did not work for me with ISBN's, so I made some changes in the oclc_classify_webscraping_stdnbr(paramtype, paramvalue) function that use a corrected API call. The changes also support addition diagnostics and handling on multi-line responses. Currently, it will take the longest DDC and LCC, assuming them to be the most detailed. The code is structured to easily support a different algo for selection.

I'm not looking to step on any toes here, but I haven't seen davidjoseph1 here in awhile (totally understandable - life intervenes). Also tried to send this via PM. So what I'm doing is attaching the new zip file here, with numbering of 2.04.1 (the current official version is 2.04). In the future, if desired, the single file mentioned above can be swapped into the master version.

To install:
- Preferences->Advanced->Plugins->Load Plugin from File
- The new plugin will install new or will overwrite your existing plugin.
- This plugin can always be overwritten by installing the latest LC-SRU version.