Metadata download plugin help with text encoding disorder

[Ubiquity]

Hello, this is for me difficult to trace, I'm writing metadata download plugin and got stuck at extracting metadata from book details page.
Doing the testing with this book Serhii Plokhy - Chernobyl: The History of a Nuclear Catastrophe

Firstly, the author field seems correctly extracted to authors string and it prints to log as 'Serhii Plokhy', but when constructing a Metadata structure by

mi = Metadata(title, authors)

the print(mi) at end of fetching prints

Author(s)           : S & e & r & h & i & i &   & P & l & o & k & h & y

This is weird encoding and I even can't gues from what to what I should convert. The more weird it is that title, fetched exactly the same way is stored in metadata properly. I'm still assuming the web page is in UTF-8 and so are interpreted the Python strings internally.

Secondly, which may be related when parsing other book details like publisher, tags etc. from details table, the data are stored in table

<tr>
  <td>name</td>
  <td>value</td>
<tr>

I'm iterating through the table and feed mi values deciding the field by extracted name literals. This works when name is only lower ascii, details having name containing acutes or diacritics aren't matched by corresponding names in plugin code. This points out that name is in wrong code page, but again fetched name literals are printed to log in proper form.

Yet another difficulty with debugging, I'm not able to figure out where log.info(...), log.debug(...) and log.error(...) commands print. Calling calibre-debug -opens a textual log after closing Calibre, but the log doesn't contain any debug info printed by anu of these commands. What only works for me is using print(...) instead which appears in %temp%/calibre_XXXXXX/*.log files. I need a clue how to debug log properly.

[kovidgoyal]

That indicates you are setting the value of authors to a string instead of a list of strings. And the output of the log statements will go into the metadata dwnload log, which you get by clicking the view log button on the download dialog.

[eschwartz]

Originally Posted by kovidgoyal

That indicates you are setting the value of authors to a string instead of a list of strings.

As Kovid said. But also keep in mind that when you try to use a string in a place where a list is expected, the string will be turned into a list by splitting each character. Then each character is treated as a separate author name, joined by "&", and potentially gets sorted alphabetically.

[Ubiquity]

I wouldnot say that my artist string is in list form. Fetching it and joining to reult string which initializes Metadata structure

authors = root.xpath('//h2[@class="authornames"]/span/a/text()')
if authors:
  authors = ' & '.join(authors).strip()
.
.
.
mi = Metadata(title, authors)

My issue seem to relate with unknown codepage in which Calibre operates the plugin.

If I parse the detail page in testing standalone script, all acuted string literals print out in proper form. If I print same string constants from plugin source, they print out in broken form, representing higher ascii characters in two byte unreadable chunks. But both plugin sources and testing script are saved in UTF-8 encoding.

[thiago.eec]

Originally Posted by Ubiquity

I wouldnot say that my artist string is in list form. Fetching it and joining to reult string which initializes Metadata structure

authors = root.xpath('//h2[@class="authornames"]/span/a/text()')
if authors:
  authors = ' & '.join(authors).strip()
.
.
.
mi = Metadata(title, authors)

What @eschwartz said is that calibre expects authors as a list, so you should format it as such. If you pass a string, then you will have this odd behavior you described.

So, you should initialize your variable as a list, and then append the values, like this example:

        authors = []
        for author_node in author_nodes:
            authors.append(author_node.text_content().strip())

[eschwartz]

Originally Posted by Ubiquity

I wouldnot say that my artist string is in list form. Fetching it and joining to reult string which initializes Metadata structure

authors = root.xpath('//h2[@class="authornames"]/span/a/text()')
if authors:
  authors = ' & '.join(authors).strip()
.
.
.
mi = Metadata(title, authors)

So that is exactly the problem. DO NOT JOIN THEM. Metadata() requires that you *not* join them yourself using ' & '.join(), so... you have explicitly converted your isinstance(authors, list) from a form that is useful and good, to a form that is useless and bad.

Originally Posted by Ubiquity

My issue seem to relate with unknown codepage in which Calibre operates the plugin.

If I parse the detail page in testing standalone script, all acuted string literals print out in proper form. If I print same string constants from plugin source, they print out in broken form, representing higher ascii characters in two byte unreadable chunks. But both plugin sources and testing script are saved in UTF-8 encoding.

You posted about multiple unrelated issues. I didn't say anything about this issue.