Recipe for Science Magazine

[keyi.fang]

Hi All,

Can someone help me setup the recipe for Science Magazine?
I am trying to download the RSS Feeds from 'http://science.sciencemag.org/rss/twis.xml'
but encounter problems logging in using the following code.

Spoiler:

Following is the code I mm working on:

#!/usr/bin/env python2

__license__ = 'GPL v3'
__copyright__ = '2008, Darko Miletic <darko.miletic at gmail.com>'
'''
sciencemag.org
'''
from calibre.web.feeds.news import BasicNewsRecipe


class ScienceAAS(BasicNewsRecipe):
title = u'Science AAAS - Work In Progress'
__author__ = u'Darko Miletic'
language = 'en'

description = u'The best in science news, commentary, and research'
oldest_article = 7
max_articles_per_feed = 100
no_stylesheets = True
use_embedded_content = False
simultaneous_downloads = 1
delay = 1
timefmt = ' [%A, %d %B, %Y]'
needs_subscription = True

def get_browser(self):
br = BasicNewsRecipe.get_browser(self)
if self.username is not None and self.password is not None:
br.open('https://signin.aaas.org/oxauth/login')
br.select_form('loginForm')
br['username'] = self.username
br['password'] = self.password
br.submit()
return br

keep_only_tags = [dict(name='div', attrs={'id': 'content-block'})]

feeds = [('Research highlights from the current issue of Science Magazine',
'http://science.sciencemag.org/rss/twis.xml')]

and the Following is the error returned:

Fetch news from Science AAAS - Work In Progress
Resolved conversion options
calibre version: 2.78.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0455BE50>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhiteOutp ut object at 0x04561230>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Python function terminated unexpectedly
no form matching name 'loginForm' (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 195, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 1065, in run
File "site-packages\calibre\customize\conversion.py", line 245, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\recipe_ input.py", line 118, in convert
File "site-packages\calibre\web\feeds\news.py", line 887, in __init__
File "<string>", line 30, in get_browser
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_mechanize.py", line 524, in select_form
mechanize._mechanize.FormNotFoundError: no form matching name 'loginForm'

[kovidgoyal]

Use select_form(nr=0) (might need to change 0 to 1 or 2 to get the right form.

[keyi.fang]

i tired nr=0, 1 and 2 all returning the following error msg:

Spoiler:

Fetch news from Science AAAS - Work In Progress
Resolved conversion options
calibre version: 2.78.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x044AAE70>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhiteOutp ut object at 0x044B0250>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Python function terminated unexpectedly
no form matching nr 2 (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 195, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 1065, in run
File "site-packages\calibre\customize\conversion.py", line 245, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\recipe_ input.py", line 118, in convert
File "site-packages\calibre\web\feeds\news.py", line 887, in __init__
File "<string>", line 30, in get_browser
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_mechanize.py", line 524, in select_form
mechanize._mechanize.FormNotFoundError: no form matching nr 2

[kovidgoyal]

They cannot all be returning that error, since that error refers to nr=2. If you are getting a form not found error with all values of nr, it means there is no form on the page.

[keyi.fang]

yes. I have tired all values of nr.

Studying the website using the development tools there is a form titled 'loginForm' in 'https://signin.aaas.org/oxauth/login'

How to specify the form?

[Image violates Posting Guidelines for size - MODERATOR]

[kovidgoyal]

What you see in a browser and what you get in mechanize can be two very different things. Either save the html mechanize gets and check that, or just manually send the post request that the form would generate, which you can get by using the dev tools in your browser to intercept the POST request.

[keyi.fang]

Thanks for your reply.

hmm..Let me explore your suggestions abit.

[Dr. Drib]

Moderator Notice

Please take a few moments to read our Posting Guidelines, in particular the part about the maximum image size allowed (600x600 pixels). Images that violate those guidelines may be deleted.

It is every member's responsibility to read our Posting Guidelines and follow them.

Thanks.

[COLOR="Green"]