02-18-2011, 09:21 PM | #1 |
Calibre Plugins Developer
Posts: 4,637
Karma: 2162064
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
|
How to handle badly formed xml from web page?
The Goodreads API I use for my Calibre Goodreads sync plugin uses (mainly) xml responses to return the results. However I have found a situation where the xml being returned is "badly formed". It can't be displayed in a web browser, due to the error, nor can it be parsed using ElementTree.
I have traced the problem down to a particular field in the xml which seems to have corrupted content - it is missing the opening <![CDATA[ within the xml text (though it has the closing ]]>). Spoiler:
I've raised this just now as a bug on the Goodreads API forums, but given they don't seem to be very actively responding to issues I want to try to handle this case myself. That particular description field doesn't happen to be one I need the values of. Currently I am using ElementTree to load the http content and retrieve elements, but of course it blows up trying to use et.fromstring() when badly formed, as below: Code:
root = et.fromstring(content)
reviews_node = root.find('reviews')
if reviews_node is not None:
total = int(reviews_node.attrib.get('total'))
end = int(reviews_node.attrib.get('end'))
book_nodes = reviews_node.findall('review/book')
for book_node in book_nodes:
book = {}
goodreads_id = book_node.findtext('id')
book['goodreads_id'] = goodreads_id
isbn = book_node.findtext('isbn13')
book['goodreads_isbn'] = isbn
(title, series) = self.convert_goodreads_title_with_series(book_node.findtext('title').strip())
book['goodreads_title'] = title
book['goodreads_series'] = series
# Grab the first author only for now
book['goodreads_author'] = book_node.findtext('authors/author/name')
|
02-18-2011, 09:25 PM | #2 |
creator of calibre
Posts: 43,856
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Use a recovering parser, grep the calibre source code for RECOVER_PARSER to see examples of its use.
|
02-18-2011, 09:38 PM | #3 | |
Calibre Plugins Developer
Posts: 4,637
Karma: 2162064
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
|
Quote:
Perhaps I shall just "gracefully" handle the error with an error dialog and have to wait for Goodreads to pull finger and fix it their side. It has only occurred with one particular book so far but if it happens for one there are bound to be others. |
|
02-18-2011, 10:19 PM | #4 |
creator of calibre
Posts: 43,856
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You can also try using beautifulstonesoup, may be more robust. If it parses successfully, then you can use it to serialize back to xml which should fix the problems for lxml.
But before doing so you will have to give it a list of the self closing tags. |
02-18-2011, 10:45 PM | #5 |
Calibre Plugins Developer
Posts: 4,637
Karma: 2162064
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
|
Ok, I decided to write out the offending http content to a file, and I discovered I was wrong about the cause being a missing CDATA opening element (it must have gone missing somehow when I printed to a debug window).
I have attached the xml file. I believe the problem is perhaps the "special characters" inside the description fields within CDATA. The parse error says line 32 column 25 which makes it look like there is some sort of encoding issue? It wouldn't be the first time with Goodreads as chaley will attest to - they have a habit of sending headers saying 'utf-8' and then putting non utf-8 characters in. I am already decoding using .decode('utf-8, errors=replace). However while that trick worked for my html web scraping issues it still isn't sufficient for the xml parser to work as coded currently (or the recovery parser). |
02-18-2011, 11:53 PM | #6 |
creator of calibre
Posts: 43,856
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Code:
from calibre.utils.cleantext import clean_ascii_chars raw = br.open(url).read() etree.fromstring(clean_ascii_chars(raw)) |
02-19-2011, 12:05 AM | #7 |
Calibre Plugins Developer
Posts: 4,637
Karma: 2162064
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
|
Thanks Kovid - that has it working now. Brilliant. |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Web Server XML data | gweminence | Calibre | 12 | 02-17-2011 10:11 AM |
Web-page uploading to Kindle(DX) | si14 | Amazon Kindle | 2 | 10-02-2010 09:25 AM |
Reading web on Sony? Demystification needed badly | reak | Sony Reader | 5 | 12-11-2007 03:35 AM |
PRS-500 XML hack for clock, joystick page turns, history AND navigation!!!!! | Bob Russell | Sony Reader Dev Corner | 2 | 08-21-2007 12:49 PM |
The first web page of the Internet | Colin Dunstan | Lounge | 1 | 01-08-2005 11:02 AM |