Fetching taking too much time

[Bortolotto]

Hi buddies!

I've made a recipe that is taking too much time to process (about 30 min.)
May you take a look at it?

Below is the recipe and attached the execution log.

class PortalR7(BasicNewsRecipe):
    title                  = 'Noticias R7'
    __author__             = 'Diniz Bortolotto'
    description            = 'Noticias Portal R7'
    oldest_article         = 2
    max_articles_per_feed  = 20
    encoding               = 'utf8'
    publisher              = 'Rede Record'
    category               = 'news, Brazil'
    language               = 'pt_BR'
    publication_type       = 'newsportal'
    feeds                  = [
                              (u'Brasil', u'http://www.r7.com/data/rss/brasil.xml'), 
                              (u'Economia', u'http://www.r7.com/data/rss/economia.xml'), 
                              (u'Internacional', u'http://www.r7.com/data/rss/internacional.xml'), 
                              (u'Tecnologia e Ci\xeancia', u'http://www.r7.com/data/rss/tecnologiaCiencia.xml')
                             ]

    reverse_article_order  = True
    remove_tags            = [
                              dict(name='ul', attrs={'class':'controles'}),
                              dict(name='div', attrs={'class':'materia_banner'}),
                              dict(name='ul', attrs={'class':'relacionados'})
                             ]
    keep_only_tags         = [
                              dict(name='div', attrs={'class':'materia'})
                             ]

[BRGriff]

I've made a recipe that is taking too much time to process (about 30 min.)
May you take a look at it?

I am addressing ONLY your complaint of the time it takes to download your articles! Since I do not read Spanish, I can not even tell if my assistance is of any help. I have done some minor work on your recipe and the material downloaded in 3 minutes (being converted into .mobi format for a Kindle). There are MANY tags and other extraneous material that still should be removed, but I will leave that to you as, again, I am only addressing the time issue.

See if the attached recipe is of any benefit and let me know. If you have further questions about your recipe, other than time it takes to download, submit another post regarding those issues.

[BRGriff]

I decided to work a little further and assist in removing some of the extraneous tags. There are still many more but, as I said, I do not read Spanish so I can not tell what is relevant and what is not.

[Bortolotto]

Hi BRGriff!!

First of all, I want to say a big "Thank you!!".

Considering your first reply, I made a new recipe (below).

Now, that new version takes around 4 minutes to fetch and create MOBI output. That is really better than first version.

So, I believe this new recipe can be usefull for all friends that are able to read in Brazilian Portuguese (not Spanish ).

The RSS source is a brazilian, well known, news portal called R7.com.
It belongs to a broadcasting corporation called "Rede Record".

import re

class PortalR7(BasicNewsRecipe):
    title                  = 'Noticias R7'
    __author__             = 'Diniz Bortolotto'
    description            = 'Noticias Portal R7'
    oldest_article         = 2
    max_articles_per_feed  = 20
    encoding               = 'utf8'
    publisher              = 'Rede Record'
    category               = 'news, Brazil'
    language               = 'pt_BR'
    publication_type       = 'newsportal'
    use_embedded_content   = False
    no_stylesheets         = True
    remove_javascript      = True
    remove_attributes      = ['style']

    feeds                  = [
                              (u'Brasil', u'http://www.r7.com/data/rss/brasil.xml'), 
                              (u'Economia', u'http://www.r7.com/data/rss/economia.xml'), 
                              (u'Internacional', u'http://www.r7.com/data/rss/internacional.xml'), 
                              (u'Tecnologia e Ci\xeancia', u'http://www.r7.com/data/rss/tecnologiaCiencia.xml')
                             ]
    reverse_article_order  = True

    keep_only_tags         = [dict(name='div', attrs={'class':'materia'})]
    remove_tags            = [
                              dict(id=['espalhe', 'report-erro']),
                              dict(name='ul', attrs={'class':'controles'}),
                              dict(name='ul', attrs={'class':'relacionados'}),
                              dict(name='div', attrs={'class':'materia_banner'}),
                              dict(name='div', attrs={'class':'materia_controles'})
                             ]

    preprocess_regexps     = [
                              (re.compile(r'<div class="materia">.*<div class="materia_cabecalho">',re.DOTALL|re.IGNORECASE),
                              lambda match: '<div class="materia"><div class="materia_cabecalho">')
                             ]

What do you think about my new recipe?

[BRGriff]

It seems to work very well and downloaded in 2 minutes 11 seconds for me. It could have been the time of day, my computer network or one of a dozen other factors. But even 4 minutes seems a vast improvement. You may wish to publish your recipe under the forum for Recipes.

Forgive my lack of knowledge as to the language. It is all Greek to me!!!

If I have been of any benefit, please do not forget to add to my Karma rating.

[Bortolotto]

Originally Posted by BRGriff

If I have been of any benefit, please do not forget to add to my Karma rating.

Yes, you helped me a lot! But, what is a "Karma rating"?

[BRGriff]

At the bottom of every user's name and info about the user is a blue button "Karma". By clicking on it, you add to that user's rating, indicating his or her helpfulness within the community.

Best of luck!!! You'll do well as member of this forum since it seems you have experience in Python coding and know your way around a Recipe.