postprocess_html

[marbs]

i have a recipe that i am working on.
it has a few tags in the middle od the article text like this:
<p>&nbsp;&nbsp;</p>
and some like this:
<p>&nbsp;</p>
there is now way to remove them with remove_tag.
thought of something like this:

Spoiler:

    def postprocess_html(self, soup):
        print 'the soup is', soup
        for tag in soup.findAll(name=['p']):
            print tag
            text= tag.contents()
            print text
            if text = ' ':
                tag.extract ()

once i add this function, the recipe does not give me any articles. am i using it right?

on an other recipe i am working on i want to use the description form the rss feed replace with a tag in the article it self.

can i the description as one of the variables that postprocess_html gets?
what is the name of the description variable in calibre?
something along the lines of

Spoiler:

    def postprocess_html(self, soup, description):
        print 'the soup is', soup
        for tag in soup.findAll(name=['td']):
            print tag
            text= tag.id
            print text
            if text ='titleContainer1':
                tag.replaceWith (description)

or something like that?

[Starson17]

Originally Posted by marbs

thought of something like this:

Spoiler:

    def postprocess_html(self, soup):
        print 'the soup is', soup
        for tag in soup.findAll(name=['p']):
            print tag
            text= tag.contents()
            print text
            if text = ' ':
                tag.extract ()

once i add this function, the recipe does not give me any articles. am i using it right?

You are using "=" the assignment operator. You want "==" the comparison operator. You could also use

    preprocess_regexps = [
        (re.compile(r'<p>&nbsp;</p>', re.DOTALL|re.IGNORECASE), lambda match: '')
        ]

can i the description as one of the variables that postprocess_html gets?
what is the name of the description variable in calibre?

The name is "text_summary", available in parse_feeds

    def parse_feeds (self): 
      feeds = BasicNewsRecipe.parse_feeds(self) 
      for feed in feeds:
        for article in feed.articles[:]:
          print 'article.text_summary is: ', article.text_summary
      return feeds

I've never used it outside of parse_feeds, that part is up to you.

[marbs]

the preprocess_regexps was a stroke of genius. very much!

as for the pre/postprocess_html, i just cant get the article soup and the article.text_summary to meet in one function. have any ideas?

[Starson17]

Put the article.text_summary into a list with append in parse_feeds, increment a counter in postprocess_html to extract it?

BTW, I did test this. I put an article_title_list = [] and index n = 0 outside the functions, then used self.article_title_list.append(article.text_summar y) inside parse_feeds and
self.article_title_list[self.n] inside postprocess_html to access it and self.n = self.n + 1 inside postprocess_html to increment the index. I didn't check to see that the order of the created list correctly matched the order the articles were accessed by postprocess_html, so you may have to deal with that.

[marbs]

got the idea. i think i can figure that out. let me work on it a bit...

any new thoughts on my maya recipe?

[Starson17]

Originally Posted by marbs

any new thoughts on my maya recipe?

Yes. What is that site?

[marbs]

what site? i mean this thread

[Starson17]

Originally Posted by marbs

what site? i mean this thread

I know. The recipe is for a site. What is that site?

[marbs]

the site is "maya.tase.co.il".

i feel i should explain more exactly what it is. it is the local stock exchange reports. on the 1st page that opens you get all the reports from all the companies for that day.

you may want all the reports from yesterday.

you might also want all the reports from one company going back to 1/1/2000.

i thought this would be possible, but they are really not making my life easy.

[marbs]

how do i use extra_css to make the article it self go rtl. i am missing the XXXX. extra_css='XXXX{direction: rtl;}'

[Starson17]

Originally Posted by marbs

how do i use extra_css to make the article it self go rtl. i am missing the XXXX. extra_css='XXXX{direction: rtl;}'

I assume you want right-to-left, and I have no idea how to do that. It's not something I do a lot.

[marbs]

i have done it before. its my curse.

in any case all i need to know is how to change the CSS on the article body it self. in other words, the article body name...

[Starson17]

Originally Posted by marbs

i have done it before. its my curse.

in any case all i need to know is how to change the CSS on the article body it self. in other words, the article body name...

    extra_css             = ' body{whatever} '

[marbs]

ill give it a try.

you haven't shared your idea about maya yet.

[Starson17]

Put the article.text_summary into a list with append in parse_feeds, increment a counter in postprocess_html to extract it?

BTW, I did test this. I put an article_title_list = [] and index n = 0 outside the functions, then used self.article_title_list.append(article.text_summar y) inside parse_feeds and
self.article_title_list[self.n] inside postprocess_html to access it and self.n = self.n + 1 inside postprocess_html to increment the index. I didn't check to see that the order of the created list correctly matched the order the articles were accessed by postprocess_html, so you may have to deal with that.