using auto_cleanup and manual clean up together

[scissors]

After weeks of tinkering withe the Daily Mirror recipe, I went back to the start and found auto_cleanup was doing a really good job - with a couple of exceptions

1) The articles by and date text are erased after the headline.
2)The text "Advertisement >>" is left intact.

The article source for the date is

Spoiler:

<h1> UK's IMF contribution limit '&pound;40bn' </h1>
<div class="article-attr">

<div class="byline append-1">


<a class="published" href="http://www.mirror.co.uk/news/latest/2011/11/06/"
title="Find all articles published on 6/11/2011 to the Latest section">
6/11/2011
</a>

</div>

so I thought using

auto_cleanup_keep = '//a[@class="published"]'

or

auto_cleanup_keep = '//*[@class="published"]'

would mean the date got left in - it wasn't.

I also tried

preprocess_regexps = [
(re.compile(r'Advertisement >>', re.IGNORECASE | re.DOTALL), lambda match: '')]

to just delete "Advertisement >>" so even if a class was created by calibre it would be empty. Again no success.

Is the call being ignored because autocleanup is being used?

It would be nice to fix this as the file created is smaller than my butchery and seems formatted in a cleaner way.

Here's the simplified code as it stands

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe
import re
from calibre.utils.magick import Image, PixelWand
class AdvancedUserRecipe1306061239(BasicNewsRecipe):
    title          = u'The Daily Mirror 2'
    description = 'News as provide by The Daily Mirror -UK'

    __author__ = 'Dave Asbury'
    # last updated 30/10/11
    language = 'en_GB'

    cover_url = 'http://yookeo.com/screens/m/i/mirror.co.uk.jpg'

    masthead_url = 'http://www.nmauk.co.uk/nma/images/daily_mirror.gif'


    oldest_article = 2
    max_articles_per_feed = 3
    remove_empty_feeds = True
    remove_javascript     = True
    no_stylesheets = True
    
    preprocess_regexps = [
    (re.compile(r'Advertisement >>', re.IGNORECASE | re.DOTALL), lambda match: '')]
    
    extra_css  = '''
	body{ text-align: justify; font-family:Arial,Helvetica,sans-serif; font-size:11px; font-size-adjust:none; font-stretch:normal; font-style:normal; font-variant:normal; font-weight:normal;}
                    h1{ font-size:16px;}
                   	'''
    auto_cleanup = True
    #auto_cleanup_keep = '//div[@class="article-attr"]'
    auto_cleanup_keep = '//a[@class="published"]'
    
    
    
    feeds          = [

        (u'News', u'http://www.mirror.co.uk/news/rss.xml')
       
           # example of commented out feed not needed ,(u'Travel','http://www.mirror.co.uk/advice/travel/rss.xml')
  ]

    def postprocess_html(self, soup, first):
        #process all the images
        for tag in soup.findAll(lambda tag: tag.name.lower()=='img' and tag.has_key('src')):
            iurl = tag['src']
            img = Image()
            img.open(iurl)
            if img < 0:
                raise RuntimeError('Out of memory')
            img.type = "GrayscaleType"
            img.save(iurl)
        return soup

[kovidgoyal]

auto_cleanup_keep will typically fail if you put it on a low level element like an <a> tag. Instead find the <div> the a is in and try keeping that.

[scissors]

Originally Posted by kovidgoyal

auto_cleanup_keep will typically fail if you put it on a low level element like an <a> tag. Instead find the <div> the a is in and try keeping that.

Hi Kovid.

I tried the div above, and it's parent and both together. no good.

Also I thought that the use of the * as in
auto_cleanup_keep = '//*[@class="important"]'

meant all elements would be saved regardless of the tag it's attached to.

ALso, is

preprocess_regexps = [
(re.compile(r'Advertisement >>', re.IGNORECASE | re.DOTALL), lambda match: '')]

not deleting instances of "Advertisement >>" because auto clean up overides it?

Can you do auto clean up followed by manual for any stray elements that get through.

+++++++++++++++++

BTW the whole reason Ive gone down this path is I discovered the text/paragraph after the first image in an article is being displayed to the right of the image (in the original Daily Mirror recipe).On my prs300 it's getting "displayed" off screen. I can't find a method to insert a crlf after the image/ before the image caption.

[kovidgoyal]

stick

img { display:block}

in the extra_css

and set

conversion_options = { 'linearize_tables' : True }

[scissors]

Originally Posted by kovidgoyal

stick

img { display:block}

in the extra_css

and set

conversion_options = { 'linearize_tables' : True }

no. still the same

[kovidgoyal]

Use the --debug-pipeline option and post one of the downloaded html files that display this issue (alo add no_stylesheets = True) to your recipe.