Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 02-23-2011, 07:43 PM   #1
Nanoox
Kindlicious.
Nanoox began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Dec 2010
Location: Monterrey, Mexico
Device: Kindle 3 (2010)
Truncation of the NYTimes Headlines recipe

I've searched the forums and it doesn't appear that anyone's reported this (or perhaps it's only happening to me!).

I run the NYTimes Headlines recipe daily at 6:30am.

Since last Friday (I think -- it's been about a week), the articles cease to load as of the New York/Region section. In other words, articles are fine for the first 7-8 sections, and then articles appear blank with a "Log In to Download" message. This can even happen within the same section; e.g. the first article in the Arts section will have content, and the other two will show the "Log In" message.

This was working fine for me until last week. I believe that I have my log in information set for that recipe (and the complete NYTimes recipe which I run once a week on Fridays with no issues), but will check Calibre on my home Mac when I get back home.

Has anyone else seen this behavior? Is there perhaps some content throttling starting to be imposed by NYTimes?

Thanks.
Nanoox is offline   Reply With Quote
Old 02-23-2011, 09:57 PM   #2
Nanoox
Kindlicious.
Nanoox began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Dec 2010
Location: Monterrey, Mexico
Device: Kindle 3 (2010)
I've checked it in Calibre and it has no username and pw field, unlike the full NYTimes recipe.

Perhaps this one needs to be updated with a username field to avoid throttling downloads?
Nanoox is offline   Reply With Quote
 
Enthusiast
Old 02-23-2011, 10:14 PM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,104
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Sounds like the NY Times now requires a password for the headlines articles as well, I'll change the recipe for the next release.
kovidgoyal is offline   Reply With Quote
Old 02-23-2011, 11:33 PM   #4
Nanoox
Kindlicious.
Nanoox began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Dec 2010
Location: Monterrey, Mexico
Device: Kindle 3 (2010)
Thanks Kovid! I look forward to it.
Nanoox is offline   Reply With Quote
Old 02-24-2011, 08:22 AM   #5
Nanoox
Kindlicious.
Nanoox began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Dec 2010
Location: Monterrey, Mexico
Device: Kindle 3 (2010)
And here's the log:


Code:
calibre, version 0.7.46
ERROR: Conversion Error: <b>Failed</b>: Fetch news from New York Times Headlines

Fetch news from New York Times Headlines
Resolved conversion options
calibre version: 0.7.46
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'enable_heuristics': False,
 'extra_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x108487ad0>,
 'insert_blank_line': False,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_ignore_margins': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindleOutput object at 0x108488150>,
 'page_breaks_before': None,
 'password': None,
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'rescale_images': False,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'username': None,
 'verbose': 2}
Python function terminated unexpectedly: The "New York Times Headlines" recipe needs a username and password.
InputFormatPlugin: Recipe Input running
Traceback (most recent call last):
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 147, in main
    return run_entry_point()
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 116, in run_entry_point
    return getattr(pmod, func)()
  File "site-packages/calibre/utils/ipc/worker.py", line 110, in main
  File "site-packages/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 904, in run
  File "site-packages/calibre/customize/conversion.py", line 204, in __call__
  File "site-packages/calibre/web/feeds/input.py", line 101, in convert
  File "site-packages/calibre/web/feeds/news.py", line 629, in __init__
ValueError: The "New York Times Headlines" recipe needs a username and password.
Nanoox is offline   Reply With Quote
Old 03-03-2011, 07:00 PM   #6
ChappyOnIce
Junior Member
ChappyOnIce began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Feb 2011
Device: Kindle 3
I get this for several whole sections on the NY Times headlines recipe. However, if I go to the website, I can access all the articles on the headlines page that show up with "Please log in" in Calibre, without needing to log in. I think there may be other issues at work than needing a login.
ChappyOnIce is offline   Reply With Quote
Old 03-04-2011, 09:14 AM   #7
ChappyOnIce
Junior Member
ChappyOnIce began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Feb 2011
Device: Kindle 3
After poking around some more, it appears that the NY Times only allows so many accesses per browser session. I used Chrome to access the headlines page and after being able to pull up several articles, I finally got a login prompt. I then got the same prompt for the articles I was previously able to open. Then I opened IE, pulled up the same headlines page and started counting as I accessed the articles. On the 12th one, I was prompted to login.
ChappyOnIce is offline   Reply With Quote
Old 03-05-2011, 10:49 PM   #8
sdow1
Connoisseur
sdow1 began at the beginning.
 
Posts: 51
Karma: 10
Join Date: Apr 2010
Location: new york city
Device: nook, ipad
The Times is in the process of erecting its paywall (which they claim is going to launch any day now), so it's possible the changes we're seeing are the result of new barriers they're erecting "under the hood" to make the paywall work without actually having to pay quite yet.

From my understanding, the "casual" browser will probably never have to pay, but the paywall is designed to get $$ from people who read the whole paper online every day, and who don't already pay for a paper subscription. I imagine that calibre users would count in the latter.

My info is probably not helpful on a technical level, but it might be worth waiting a few days, or weeks (as annoying as it is) for the paywall to actually kick in before spending a ton of time revamping the recipe - if they're making a boatload of changes to the site, it might be easier in the long run to wait for the dust to settle.
sdow1 is offline   Reply With Quote
Reply

Tags
bug, headlines, nytimes

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Recipe works when mocked up as Python file, fails when converted to Recipe ode Recipes 7 09-04-2011 04:57 AM
MacWorld recipe - only headlines - no articles simonz Recipes 4 06-04-2011 09:02 AM
eBook vs. Hardcover: Beyond the Headlines DMcCunney News 26 07-21-2010 07:32 PM
Daily Rotation - More Tech headlines than you can imagine! Bob Russell Lounge 1 09-24-2005 04:36 AM


All times are GMT -4. The time now is 06:54 AM.


MobileRead.com is a privately owned, operated and funded community.