NYTimes - unclosed comment tag

[bobbysteel]

Hi Kovid -
The NYTimes recipe has a collection of --> (sometimes twice) leftover in many articles, which seems to be an unmatched comment tag. I've tried replacing in preprocess and postprocess_html but can't seem to figure out how to do so. Any ideas the best way to remove this from articles?

    def postprocess_html(self, soup, first_fetch):
        findcomment = soup.findAll(text = re.compile('--gt&;'))
        for comment in findcomment:
            fixed_text = unicode(comment).replace('--gt&;', '')
            comment.replace_with(fixed_text)
        return soup

Doesn't seem to work and returns articles with nothing but the -->

[kovidgoyal]

THe easiest way to get rid of comments is to use preprocess_regexps and simply replace them with an empty string. something like

preprocess_regexps = [(re.compile(r'(?s)<!--.*?-->'), lambda m: '')]

[bobbysteel]

Great thanks. Shall I check this in once I've tested?

[kovidgoyal]

Sure.

[bobbysteel]

Also I'm also getting a very occasional error on the existing recipe

has_supplemental = article.find(**classes('story-body-supplemental')) is not None

is generating

Could not fetch link https://www.nytimes.com/2018/05/18/watching/cheers-best-episodes.html
Traceback (most recent call last):
  File "site-packages\calibre\web\fetch\simple.py", line 520, in process_links
  File "site-packages\calibre\web\fetch\simple.py", line 227, in get_soup
  File "<string>", line 107, in preprocess_html
AttributeError: 'NoneType' object has no attribute 'find'

Is there a way to make this not fail?

[bobbysteel]

Hm, I also can't seem to get the result from the regexp you suggested.

    preprocess_regexps = [(re.compile(r'(?s)-->'), lambda m: '')]
    preprocess_regexps = [(re.compile(r'(?s)-->'), lambda m: '')]

Still returns this in the final HTML:

<p class="calibre_2">--&gt;--&gt;  </p>

[kovidgoyal]

That's not the regexp I suggested, you need both the opening and closing parts of the comment. If you want to debug it or do the reaplcing manually you can implement preprocess_raw_html in the recipe.

And I committed a fix for the AttributeError

[bobbysteel]

I tried the version you suggested, but shortened as it seems to be missing only the closing part of the comment no?

[bobbysteel]

Doh ok i tried again and it worked. Silly me.

[bobbysteel]

Weird. Kovid, the comment tag is coming back now. Why is it missing just the closing tag? Is there a parsing error somewhere missing the multi-line comments somehow? Your regexp misses it too even with DOTALL enabled.

[kovidgoyal]

Unless the NYT's markup is invaid (i.e. contains nested comments) that regexp will remove all comments. Implement preprocess_raw_html() in the recipe and save the raw html and look at it in an editor. Track down where the closing comment is coming from by compainr it to the final HTML generated byt he recipe.

[bobbysteel]

How do I save the raw HTML easily from command line? Is there syntax to drop it in to a temp folder?
Thx

[kovidgoyal]

open('/path/to/tempfile.html', 'wb').write(raw_html)

[bobbysteel]

just looking at a random article it doesn't seem to be unmatched anywhere I can see. Just lots of comments with one space in between. i can't understand the source why this regex would fail with only 7 comment tags on page.

[bobbysteel]

Originally Posted by bobbysteel

How do I save the raw HTML easily from command line? Is there syntax to drop it in to a temp folder?
Thx

where do i drop that line? in preprocess subroutine it fails.