Recipe Request: New left review

luiscc · 03-07-2011, 04:24 PM

Hello,

Is it possible to create a recipe to new left review? http://newleftreview.org/
For the majority of the articles a subscrption (login/password) is needed but some are available for free.

Kind Regards.

luiscc · 03-08-2011, 03:48 PM

I managed to create a recipe to NLR(based on oneilpt example), it works fairly well although it has a minor problem with some symbols, (they are replaced others). But that is not a big deal for me. I know the code is far from perfect.

At the end tried to had the code to insert login/password, but when fetching keep receiving an annoying message.

Here is the CODE:

Code:

'''
http://newleftreview.org
'''

from calibre.web.feeds.news import BasicNewsRecipe

class NewLeftReview(BasicNewsRecipe):
    title                 = 'New Left Review2'
    description           = ''
    publisher             = 'NLR'
    category              = 'analisys, politics, left'
    needs_subscription = True
    no_stylesheets        = True
    encoding              = 'cp1252'
    use_embedded_content  = False
    language              = 'en'
    publication_type      = 'magazine'    

    
    masthead_url     = 'http://newleftreview.org/images/logo.gif'
    index                 = 'http://newleftreview.org/'
    
    conversion_options = {
                          'comment'      : description
                        , 'tags'         : category
                        , 'publisher'    : publisher
                        , 'language'     : language
                        }

    remove_attributes=['width','height']
    remove_tags_before = dict(name='h4', attrs={'id':'publisher'}) 
    remove_tags=[dict(name='ul', attrs={'id':'articlelogolist'}),
	 dict(name='div', attrs={'id':'bottom'})     
	 ]

    def get_browser(self):
        br = BasicNewsRecipe.get_browser()
        if self.username is not None and self.password is not None:
            br.open('http://www.newleftreview.org/?page=login')
            br.select_form(name='login')
            br['USERID']   = self.username
            br['PASSWORD'] = self.password
            br.submit()
        return br

    def parse_index(self):
        articles = []
        soup = self.index_to_soup(self.index)
        cover = None
        feeds = []
        for section in soup.findAll('h4', attrs={'class':'center'}):
            section_title = self.tag_to_string(section.find('title'))
            articles = []
            for post in section.findAll('a', href=True):
                url = post['href']
                if url.startswith('?'):
                  url = 'http://newleftreview.org/'+url
                  title = self.tag_to_string(post)
                  self.log()
                  self.log('--> post:  ', post)                  self.log('--> url:   ', url)
                  self.log('--> title: ', title)
                  articles.append({'title':title, 'url':url})
            if articles:
                feeds.append((section_title, articles))
        return feeds

and here the ERROR message

Code:

Fetch news from New Left Review2
Resolved conversion options
calibre version: 0.7.47
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'enable_heuristics': False,
 'extra_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x05AA49F0>,
 'insert_blank_line': False,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_ignore_margins': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindleOutput object at 0x05AA4D10>,
 'page_breaks_before': None,
 'password': 'riotejo',
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'rescale_images': False,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'username': 'luiscc',
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Python function terminated unexpectedly
  no form matching name 'login' (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 103, in main
  File "site.py", line 85, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 110, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 904, in run
  File "site-packages\calibre\customize\conversion.py", line 204, in __call__
  File "site-packages\calibre\web\feeds\input.py", line 101, in convert
  File "site-packages\calibre\web\feeds\news.py", line 631, in __init__
  File "c:\users\lus~1.pro\appdata\local\temp\calibre_0.7.47_tmp_nxbsa9\calibre_0.7.47_gz04zi_recipes\recipe0.py", line 43, in get_browser
    br.select_form(name='login')
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 524, in select_form
mechanize._mechanize.FormNotFoundError: no form matching name 'login'

Can anyone help, please?

Starson17 · 03-08-2011, 04:15 PM

Quote:

Originally Posted by luiscc

Can anyone help, please?

You have told your recipe to find a form named "login" with this line:

Code:

br.select_form(name='login')

Your error is simple:

Code:

no form matching name 'login'

Try something like this:

Code:

br.select_form(nr=0)

Increment nr until you hit your form. That assumes that you are on the right login page, that it uses a login form, that you have the right control names USERID and PASSWORD, etc.

luiscc · 03-08-2011, 05:27 PM

Thanks Starson17, it solved that problem. But now i have a new one, I get that userID is "no control matching name",

ERROR message

Code:

Fetch news from New Left Review2
Resolved conversion options
calibre version: 0.7.47
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'enable_heuristics': False,
 'extra_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x05A149F0>,
 'insert_blank_line': False,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_ignore_margins': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindleOutput object at 0x05A14D10>,
 'page_breaks_before': None,
 'password': 'riotejo',
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'rescale_images': False,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'username': 'luiscc',
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Python function terminated unexpectedly
  no control matching name 'USERID' (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 103, in main
  File "site.py", line 85, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 110, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 904, in run
  File "site-packages\calibre\customize\conversion.py", line 204, in __call__
  File "site-packages\calibre\web\feeds\input.py", line 101, in convert
  File "site-packages\calibre\web\feeds\news.py", line 631, in __init__
  File "c:\users\lus~1.pro\appdata\local\temp\calibre_0.7.47_tmp_nxbsa9\calibre_0.7.47_cedlus_recipes\recipe0.py", line 47, in get_browser
    br['USERID']   = self.username
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_form.py", line 2780, in __setitem__
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_form.py", line 3101, in find_control
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_form.py", line 3185, in _find_control
mechanize._form.ControlNotFoundError: no control matching name 'USERID'

i found here a reasonable explanation. I tried to adapt their solution, but I'm really new in python and recipes and couldnt do it

Is there any other way?

By the way is

Code:

br.select_form(nr=1)

Starson17 · 03-09-2011, 07:52 AM

Quote:

Originally Posted by luiscc

Thanks Starson17, it solved that problem. But now i have a new one, I get that userID is "no control matching name"

That's why I wrote that you have to "have the right control names USERID and PASSWORD, etc." What are the control names for your form? Look at the source code for your page to find out.

luiscc · 03-09-2011, 09:46 AM

Quote:

What are the control names for your form? Look at the source code for your page to find out.

In fact I already did that, and thought everything was right but it was not, 'userId' apparantely should be changed by 'referenenum'. But the problem is not solved. I changed the order of password and userid in the code (referencenum from now on) and the answer was

Quote:

_find_control
mechanize._form.ControlNotFoundError: no control matching name 'PASSWORD'

and i'm sure there is a password control.

I think the same problem was found here, (at least the html code as 2 cases of the error they notice, the use of a "<br/>" tag which should be "<br /", with the space) i followed one recomendation there and did "print br.form.controls" in my code the answer was

Quote:

print br.form.controls
AttributeError: 'NoneType' object has no attribute 'controls'

I probably must addapt the solution they found, but that i cant do, is too much for me.

Starson17 · 03-09-2011, 12:06 PM

Quote:

Originally Posted by luiscc

In fact I already did that, and thought everything was right but it was not, 'userId' apparantely should be changed by 'referenenum'.

I looked at your login page. It's "referencenum," not "referenenum."

Quote:

and i'm sure there is a password control.

There is, but IIRC, controls are case sensitive...
if you got this error message:

Code:

_find_control
mechanize._form.ControlNotFoundError: no control matching name 'PASSWORD'

you may need to use "password" not "PASSWORD" as it's lower case on that page.

luiscc · 03-09-2011, 12:50 PM

Thank you starson17 you have been very kind.

Quote:

I looked at your login page. It's "referencenum," not "referenenum."

yes I know, it was a typo.

Quote:

you may need to use "password" not "PASSWORD" as it's lower case on that page

I did it, the error maintains.

I really think the problem is on the HTML page, because they use "<br/>" instead of "<br /", as i told before this situation is described here

Code is this:

Code:

'''
http://newleftreview.org
'''

from calibre.web.feeds.news import BasicNewsRecipe
import re
from mechanize import Browser


class NewLeftReview(BasicNewsRecipe):
    title                 = 'New Left Review2'
    description           = ''
    publisher             = 'NLR'
    category              = 'analisys, politics, left'
    needs_subscription = True
    no_stylesheets        = True
    encoding              = 'cp1252'
    use_embedded_content  = False
    language              = 'en'
    publication_type      = 'magazine'    

    
    masthead_url     = 'http://newleftreview.org/images/logo.gif'
    index                 = 'http://newleftreview.org/'
    
    conversion_options = {
                          'comment'      : description
                        , 'tags'         : category
                        , 'publisher'    : publisher
                        , 'language'     : language
                        }

    remove_attributes=['width','height']
    remove_tags_before = dict(name='h4', attrs={'id':'publisher'}) 
    remove_tags=[dict(name='ul', attrs={'id':'articlelogolist'}),
	 dict(name='div', attrs={'id':'bottom'})     
	 ]

    def get_browser(self):
        br = BasicNewsRecipe.get_browser()
        if self.username is not None and self.password is not None:
            br.open('http://www.newleftreview.org/?page=login')
            br.select_form(nr=1)
            br['password'] = self.password
            br['referencenum']   = self.username
            br.submit()
        return br

    def parse_index(self):
        articles = []
        soup = self.index_to_soup(self.index)
        cover = None
        feeds = []
        for section in soup.findAll('h4', attrs={'class':'center'}):
            section_title = self.tag_to_string(section.find('title'))
            articles = []
            for post in section.findAll('a', href=True):
                url = post['href']
                if url.startswith('?'):
                  url = 'http://newleftreview.org/'+url
                  title = self.tag_to_string(post)
                  self.log()
                  self.log('--> post:  ', post)
                  self.log('--> url:   ', url)
                  self.log('--> title: ', title)
                  articles.append({'title':title, 'url':url})
            if articles:
                feeds.append((section_title, articles))
        return feeds

The error

Code:

Fetch news from New Left Review2
Resolved conversion options
calibre version: 0.7.47
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'enable_heuristics': False,
 'extra_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x05ACD9F0>,
 'insert_blank_line': False,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_ignore_margins': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindleOutput object at 0x05ACDD10>,
 'page_breaks_before': None,
 'password': 'riotejo',
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'rescale_images': False,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'username': 'luiscc',
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Python function terminated unexpectedly
  no control matching name 'password' (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 103, in main
  File "site.py", line 85, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 110, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 904, in run
  File "site-packages\calibre\customize\conversion.py", line 204, in __call__
  File "site-packages\calibre\web\feeds\input.py", line 101, in convert
  File "site-packages\calibre\web\feeds\news.py", line 631, in __init__
  File "c:\users\lus~1.pro\appdata\local\temp\calibre_0.7.47_tmp_psb_n8\calibre_0.7.47_jabvrj_recipes\recipe0.py", line 47, in get_browser
    br['password'] = self.password
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_form.py", line 2780, in __setitem__
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_form.py", line 3101, in find_control
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_form.py", line 3185, in _find_control
mechanize._form.ControlNotFoundError: no control matching name 'password'

Starson17 · 03-09-2011, 01:21 PM

Quote:

Originally Posted by luiscc

Thank you starson17 you have been very kind.

This is an interesting puzzle. I have less time than I used to, but if you want me to take a look, PM me a user/pass, and I'll see if I can solve it for you.

sneakyfox · 05-29-2014, 03:02 PM

I am very interested to hear if this recipe was ever made to work properly?

It would be great with a recipe for New Left Review!

03-07-2011, 04:24 PM	#1
luiscc Junior Member Posts: 9 Karma: 10 Join Date: Feb 2011 Device: kindle	Recipe Request: New left review Hello, Is it possible to create a recipe to new left review? http://newleftreview.org/ For the majority of the articles a subscrption (login/password) is needed but some are available for free. Kind Regards.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Request: for recipe	deppeler	Recipes	4	01-12-2011 09:29 AM
recipe request	Torx	Recipes	0	12-20-2010 08:33 AM
Request for recipe	sumper	Recipes	2	10-11-2010 02:25 AM
Recipe request please	aessedai44	Recipes	2	10-06-2010 01:07 AM
Request for Recipe	ddavtian	Calibre	2	11-24-2008 02:43 AM

05-29-2014, 03:02 PM	#10
sneakyfox Junior Member Posts: 3 Karma: 10 Join Date: May 2014 Device: Sony Ebook Reader	I am very interested to hear if this recipe was ever made to work properly? It would be great with a recipe for New Left Review!

Advert

Advert