[Eskoe]

Originally Posted by kiwidude

Thanks for giving it a run so quickly Eskoe and your logs, very helpful. It allowed me to confirm that you were exclusively hitting the "old" html pages with all those books - i.e. they would have coincidentally still worked with the old 1.5.x plugin. So at least I know that side of things is still working well, I will wait a little longer for feedback from others before releasing in the hope they happen to hit the new code more.

Unfortunately there is no guaranteed way I found to force it to hit one version of the Goodreads book page over another at the moment (which made testing my changes a pain). It is genuinely a heads or tails coin flip every time. I had to sit there doing Download -> View Log -> Cancel over and over each time I wanted to test something with specific page variants...

Incidentally if you are bored/curious and want to know what to look for in the logs - if you see "_legacy" in those log files then it is the "old" website page design (e.g. "parse_isbn_legacy: xxxx"). Whereas with the new web page the log will just say "parse_isbn: xxxx". There are other lines in there which point out more explicitly but that is the quickest way to skim read the answer.

Not a problem! And definitely curious, that's quite interesting. I guess I somehow lucked (unlucked?) into only getting the legacy, explains why I got failures so often lol.

I double checked with a bunch of attempts, it's definitely both with and without the "_legacy"

Spoiler:

#################### Log for A critical cinema : interviews with independent filmmakers ####################
Running identify query with parameters:
{'title': 'A critical cinema : interviews with independent filmmakers', 'authors': ['MacDonald, Scott, 1942-'], 'identifiers': {'goodreads': '599683'}, 'timeout': 30}
Using plugins: Goodreads (1, 6, 1)
The log from individual plugins is below

****************************** Goodreads (1, 6, 1) ******************************
Found 1 results
Downloading from Goodreads took 2.9740753173828125


---
Title : A Critical Cinema 3: Interviews With Independent Filmmakers
Author(s) : Scott MacDonald
Publisher : University of California Press
Tags : Film
Languages : eng
Rating : 2.2
Published : 1998-01-01T00:00:00+00:00
Identifiers : goodreads:599683, isbn:9780520209435
Comments :
A Critical Cinema 3 continues Scott MacDonald's compilation of personal interviews and public discussions with major contributors to independent filmmaking and film awareness. An informative exchange with Amos Vogel, whose Cinema 16 Society drew American filmgoers into a broader sense of film history, is followed by interviews reflecting a wide range of approaches to filmmaking. Sally Potter discusses her popular feature, Orlando , in relation to the experimental work that preceded it, and Canadian independent John Porter argues compellingly for small-gauge, Super-8mm filmmaking. Ken Jacobs discusses the "Nervous System" apparatus with which he transforms old film footage into new forms of motion picture art; Jordan Belson describes his Vortex Concerts, ancestors of modern laser light shows; and Elias Merhige talks about going beneath the "rational structure of meaning" in Begotten.
A Critical Cinema 3 presents independent cinema as an international and multiethnic phenomenon. MacDonald interviews filmmakers from Sweden, France, Italy, Austria, Armenia, India, the Philippines, and Japan and examines the work of African Americans, European Americans, Asian Americans, and Hispanics. He provides an introductory overview of each interviewee, as well as detailed film/videographies and selected bibliographies. With its predecessors, A Critical Cinema (California, 1988) and A Critical Cinema 2 (California, 1992), this is the most extensive, in-depth exploration of independent cinema available in English.

identify - start. title=A critical cinema : interviews with independent filmmakers, authors=['MacDonald, Scott, 1942-'], identifiers={'goodreads': '599683'}
Goodreads book url: 'https://www.goodreads.com/book/show/599683'
Trying to parse book json for 2022 web page format
Page is legacy html format as NO Json found
parse_title: title_text='A Critical Cinema 3: Interviews with Independent Filmmakers'
parse_authors_legacy - author: Scott MacDonald
parse_authors_legacy - author: Scott MacDonald
parse_details - goodreads_id: 599683, mi: Title : A Critical Cinema 3: Interviews with Independent Filmmakers
Author(s) : Scott MacDonald
parse_rating_legacy: rating_value= 4.38
parse_language_legacy: Have language node
parse_language_legacy: raw= English
parse_language_legacy: ans= eng

************************************************** ******************************
The identify phase took 3.10 seconds
The longest time (2.974075) was taken by: Goodreads
Merging results from different sources
We have 1 merged results, merging took: 0.00 seconds

****************************** Goodreads Covers ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36')]
Downloaded cover: 267x400
Took 0.8618631362915039 seconds
Downloading cover from: https://i.gr-assets.com/images/S/com...05l/599683.jpg

************************************************** ******************************

#################### Log for The life and adventures of Lazarillo de Tormes ####################
Running identify query with parameters:
{'title': 'The life and adventures of Lazarillo de Tormes', 'authors': ['Lazarillo de Tormes. Roscoe, Thomas, 1791-1871. Le Sage, Alain René, 1668-1747. Brady, John Henry. Alemán, Mateo, 1547-1614.'], 'identifiers': {'goodreads': '7574111'}, 'timeout': 30}
Using plugins: Goodreads (1, 6, 1)
The log from individual plugins is below

****************************** Goodreads (1, 6, 1) ******************************
Found 1 results
Downloading from Goodreads took 1.8510007858276367


---
Title : The Life and Adventures of Lazarillo De Tormes / the Life and Adventures of Guzman D'Alfarache, or the Spanish Rogue
Author(s) : Thomas Roscoe
Publisher : J.C. Nimmo and Bain
Languages : eng
Rating :
Published : 1599-01-01T08:00:00+00:00
Identifiers : goodreads:7574111
Comments :
Volume: 1 General Books publication date: 2009 Original publication date: 1881 Original Publisher: J.C. Nimmo and Bain Subjects: Fiction / Literary Literary Collections / Essays Literary Criticism / Caribbean

identify - start. title=The life and adventures of Lazarillo de Tormes, authors=['Lazarillo de Tormes. Roscoe, Thomas, 1791-1871. Le Sage, Alain René, 1668-1747. Brady, John Henry. Alemán, Mateo, 1547-1614.'], identifiers={'goodreads': '7574111'}
Goodreads book url: 'https://www.goodreads.com/book/show/7574111'
Trying to parse book json for 2022 web page format
Json script node found, page in 2022+ html format
parse_title: The Life and Adventures of Lazarillo de Tormes / The Life and Adventures of Guzman D'Alfarache, or The Spanish Rogue
parse_authors - author: Thomas Roscoe
parse_authors - author: Anonymous
parse_details - goodreads_id: 7574111, mi: Title : The Life and Adventures of Lazarillo de Tormes / The Life and Adventures of Guzman D'Alfarache, or The Spanish Rogue
Author(s) : Thomas Roscoe
parse_isbn: None
parse_asin: None
parse_rating: 0.0
Error parsing cover for url: 'https://www.goodreads.com/book/show/7574111'
Traceback (most recent call last):
File "mechanize\_mechanize.py", line 266, in _mech_open
AttributeError: 'str' object has no attribute 'get_full_url'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.goodreads.worker", line 259, in parse_details
File "calibre_plugins.goodreads.worker", line 510, in parse_cover
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 273, in _mech_open
mechanize._mechanize.BrowserStateError: can't fetch relative reference: not viewing any document

parse_publisher_and_date: First published date: 1599-01-01 08:00:00+00:00
parse_language: ans= eng

************************************************** ******************************
The identify phase took 2.04 seconds
The longest time (1.851001) was taken by: Goodreads
Merging results from different sources
We have 1 merged results, merging took: 0.00 seconds

****************************** Goodreads Covers ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36')]
Failed to download valid cover
Took 1.968559980392456 seconds
No cached cover found, running identify
identify - start. title=The life and adventures of Lazarillo de Tormes, authors=['Lazarillo de Tormes. Roscoe, Thomas, 1791-1871. Le Sage, Alain René, 1668-1747. Brady, John Henry. Alemán, Mateo, 1547-1614.'], identifiers={'goodreads': '7574111'}
Goodreads book url: 'https://www.goodreads.com/book/show/7574111'
Trying to parse book json for 2022 web page format
Json script node found, page in 2022+ html format
parse_title: The Life and Adventures of Lazarillo de Tormes / The Life and Adventures of Guzman D'Alfarache, or The Spanish Rogue
parse_authors - author: Thomas Roscoe
parse_authors - author: Anonymous
parse_details - goodreads_id: 7574111, mi: Title : The Life and Adventures of Lazarillo de Tormes / The Life and Adventures of Guzman D'Alfarache, or The Spanish Rogue
Author(s) : Thomas Roscoe
parse_isbn: None
parse_asin: None
parse_rating: 0.0
Error parsing cover for url: 'https://www.goodreads.com/book/show/7574111'
Traceback (most recent call last):
File "mechanize\_mechanize.py", line 266, in _mech_open
AttributeError: 'str' object has no attribute 'get_full_url'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.goodreads.worker", line 259, in parse_details
File "calibre_plugins.goodreads.worker", line 510, in parse_cover
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 273, in _mech_open
mechanize._mechanize.BrowserStateError: can't fetch relative reference: not viewing any document

parse_publisher_and_date: First published date: 1599-01-01 08:00:00+00:00
parse_language: ans= eng
No cover found

************************************************** ******************************

#################### Log for A Season in Hell & Illuminations ####################
Running identify query with parameters:
{'title': 'A Season in Hell & Illuminations', 'authors': ['Arthur Rimbaud'], 'identifiers': {'goodreads': '168496'}, 'timeout': 30}
Using plugins: Goodreads (1, 6, 1)
The log from individual plugins is below

****************************** Goodreads (1, 6, 1) ******************************
Found 1 results
Downloading from Goodreads took 2.938692331314087


---
Title : A Season in Hell & Illuminations
Author(s) : Arthur Rimbaud
Publisher : Modern Library
Tags : Poetry, Classics
Languages : eng
Rating : 2.1
Published : 1873-01-01T08:00:00+00:00
Identifiers : goodreads:168496, isbn:9780679643272
Comments :
“The definitive translation for our time.”

–Edward Hirsch
From Dante’s Inferno to Sartre’s No Exit, writers have been fascinated by visions of damnation. Within that rich literature of suffering, Arthur Rimbaud’s A Season in Hell–written when the poet was nineteen–provides an astonishing example of the grapple with self.
As a companion to Rimbaud’s journey, readers could have no better guide than Wyatt Mason. One of our most talented young translators and critics, Mason’s new version of A Season in Hell renders the music and mystery of Rimbaud’s tale of Hell on Earth with exceptional finesse and power.
This bilingual edition includes maps, a helpful chronology of Rimbaud’s life, and the unfinished suite of prose poems, Illuminations and A Season in Hell cement Rimbaud’s reputation as one of the foremost, and most influential, writers in French literature.

identify - start. title=A Season in Hell & Illuminations, authors=['Arthur Rimbaud'], identifiers={'goodreads': '168496'}
Goodreads book url: 'https://www.goodreads.com/book/show/168496'
Trying to parse book json for 2022 web page format
Json script node found, page in 2022+ html format
parse_title: A Season in Hell & Illuminations
parse_authors - author: Arthur Rimbaud
parse_authors - author: Wyatt Mason
parse_details - goodreads_id: 168496, mi: Title : A Season in Hell & Illuminations
Author(s) : Arthur Rimbaud
parse_isbn: 9780679643272
parse_asin: None
parse_rating: 4.3
parse_publisher_and_date: First published date: 1873-01-01 08:00:00+00:00
parse_language: ans= eng

************************************************** ******************************
The identify phase took 3.06 seconds
The longest time (2.938692) was taken by: Goodreads
Merging results from different sources
We have 1 merged results, merging took: 0.00 seconds

****************************** Goodreads Covers ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36')]
Downloaded cover: 292x450
Took 0.5784306526184082 seconds
Downloading cover from: https://images-na.ssl-images-amazon....37i/168496.jpg

************************************************** ******************************

(The failure was because the book didn't have a cover).

Still haven't run into a problem, everything's working great. Can't thank you enough