index_to_soup: how can I sanitize the html using markupMassage

[oecherprinte]

Hi,

I am writing a recipe for a web page that includes an error which completely confuses beautiful soup. When I use the convenience function index_to_soup I can generate beautiful soup from an html file. However, I would have to use the markupMassage feature of beautiful soup to remove some errors from the html file before converting it into beautiful soup:

http://www.crummy.com/software/Beaut...mentation.html

Are there any parameters or other mechanisms to pass the markupMassage list to index_to_soup? I have notices that the function does something with the makupMassage Parameter when generating the beautiful soup:

  massage = list(BeautifulSoup.MARKUP_MASSAGE)
        enc = 'cp1252' if callable(self.encoding) or self.encoding is None else self.encoding
        massage.append((re.compile(r'&(\S+?);'), lambda match:
            entity_to_unicode(match, encoding=enc)))
        return BeautifulSoup(_raw, markupMassage=massage)

So there must be some way of passing my personal markupMassage list to index_to_soup?

Thanks,

Jens

[Starson17]

Originally Posted by oecherprinte

Are there any parameters or other mechanisms to pass the markupMassage list to index_to_soup? I have notices that the function does something with the makupMassage Parameter when generating the beautiful soup:

  massage = list(BeautifulSoup.MARKUP_MASSAGE)
        enc = 'cp1252' if callable(self.encoding) or self.encoding is None else self.encoding
        massage.append((re.compile(r'&(\S+?);'), lambda match:
            entity_to_unicode(match, encoding=enc)))
        return BeautifulSoup(_raw, markupMassage=massage)

So there must be some way of passing my personal markupMassage list to index_to_soup?

Thanks,

Jens

Look at news.py for the def of index_to_soup.
I'll paste it here:

Spoiler:

    def index_to_soup(self, url_or_raw, raw=False):
        '''
        Convenience method that takes an URL to the index page and returns
        a `BeautifulSoup <http://www.crummy.com/software/BeautifulSoup/documentation.html>`_
        of it.

        `url_or_raw`: Either a URL or the downloaded index page as a string
        '''
        if re.match(r'\w+://', url_or_raw):
            open_func = getattr(self.browser, 'open_novisit', self.browser.open)
            with closing(open_func(url_or_raw)) as f:
                _raw = f.read()
            if not _raw:
                raise RuntimeError('Could not fetch index from %s'%url_or_raw)
        else:
            _raw = url_or_raw
        if raw:
            return _raw
        if not isinstance(_raw, unicode) and self.encoding:
            if callable(self.encoding):
                _raw = self.encoding(_raw)
            else:
                _raw = _raw.decode(self.encoding, 'replace')
        massage = list(BeautifulSoup.MARKUP_MASSAGE)
        enc = 'cp1252' if callable(self.encoding) or self.encoding is None else self.encoding
        massage.append((re.compile(r'&(\S+?);'), lambda match:
            entity_to_unicode(match, encoding=enc)))
        return BeautifulSoup(_raw, markupMassage=massage)

You should be able to modify the massage = list for whatever you need.

[oecherprinte]

@Starson17:

Thanks. Of course I could change the index_to_soup function.
But I have the impression that I can set the variable

BeautifulSoup.MARKUP_MASSAGE

somewhere externally e.g. in the header of the recipe?

[oecherprinte]

It's a nightmare. My first contact to python is really messy. What terrible language to debug ...

Anyway, I tried fumbling around with setting a "BeautifulSoup.MARKUP_MASSAGE" variable in my recipe which did not work. Then I tried to copy the "index_to_soup" function into my recipe and renamed it to "my_index_to_soup". I copied all the imports from the "calibre.web.feeds.news" file. Now I get the error message:

"ValueError: too many values to unpack"

for the line "return BeautifulSoup(_raw, markupMassage=massage)"

without even touching the code. I am giving up now.

Isn't there an easy way to automatically remove erroneous html code from a file before transferring it into beautiful soup? Maybe the developer could help (by the way: I already donated via paypal last week :-) ). I could imagine that many recipe programmers are facing this problem.

Cheers,

Jens

P.S.: My specific problem is the line "<!#BeginList>" in the html file which makes beautiful soup think that the remainder of the file is a single tag ... (sigh)

[oecherprinte]

Nevermind. I just solved the problem:

As I said I just copied the index_to_soup function into my new recipe and renamed it to my_index_to_soup. Then I added the following lines before the "return" statement:

        #remove erroneous strings from input file
        massage.append((re.compile("<!#BeginList>"), lambda match:''))
        massage.append((re.compile("<!#EndList>"), lambda match:''))

and voila, the junk is removed ...