Calibre WSJ Problem

[jsl21]

Anyone else have a problem downloading the WSJ today? I keep getting the following error. It worked fine before today.

ERROR: Conversion Error: <b>Failed</b>: Fetch news from The Wall Street Journal (US)

InputFormatPlugin: Recipe Input running
Python function terminated unexpectedly
'NoneType' object has no attribute 'findNext' (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 103, in main
File "site.py", line 85, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 99, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 24, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 815, in run
File "site-packages\calibre\customize\conversion.py", line 211, in __call__
File "site-packages\calibre\web\feeds\input.py", line 104, in convert
File "site-packages\calibre\web\feeds\news.py", line 701, in download
File "site-packages\calibre\web\feeds\news.py", line 854, in build_index
File "c:\docume~1\jeff~1.lox\locals~1\temp\calibre_0.7. 2_vrq3le_recipes\recipe0.py", line 90, in parse_index
table = left_column.findNext('table')
AttributeError: 'NoneType' object has no attribute 'findNext'

[kovidgoyal]

Try it again (the website changed and I've updated the recipe)

[moosejons_dad]

Originally Posted by kovidgoyal

Try it again (the website changed and I've updated the recipe)

I just tried the WSJ and the good news is that it works. However, the format of the paper is completely different from what we were getting before the change. The headings were as follows:Front Section, Opinion, Greater New York, Market Place and Personal Journal. It looks like they have changed their RSS feeds too.

Any thoughts on how to get it back to the way it was especially adding in "Greater New York" and "Personal Journal"?

Thanks

[kovidgoyal]

The subscription based on is the same as before. The RSS one depends on what RSS the wsj puts out, not a lot I can do about that.

[jsl21]

Thanks Kovid! It works now but I am having a weird problem. I've been using a custom recipe (to remove the date from the title). I removed my old custom recipe and added it back in to get what I thought would be the updated recipe but it does not work when I run this version (even if I make no changes to it). I must not be understanding how the built in recipes work. Did you just fix it today? If so, how did I get it without downloading a new version. Thanks in advance.

P.S. The new version does seem to get only the first section now. I was going to try to fool around with the recipe to fix this but first I need to get a usable custom recipe

[kovidgoyal]

The latest version of the recipe is always available here: http://bazaar.launchpad.net/~kovid/c...pes/wsj.recipe

[moosejons_dad]

Originally Posted by kovidgoyal

The subscription based on is the same as before. The RSS one depends on what RSS the wsj puts out, not a lot I can do about that.

Thanks for your advice.
I did however download the free edition of the WSJ today as well as the subscription version and found that the subscription version has 41 articles and the free edition had 265. My only question is why such a difference in the two versions and why would anyone continue to pay for the Journal when the free edition gives more news?
Thanx...

[kovidgoyal]

the subscription version takes the articles linked to in the todays paper page on the wsj website. You will find many of the articles in the free version to be duplicated between successive downloads. PLus inthe free version you get partial articles.

[jsl21]

moosejons_dad,

I think this is also because they changed the format of the paid version so what is being downloaded is only the first of six sections. Each section is now on a separate web page instead of the single page that it used to be on. I've modified the recipe to get each of the six sections as six separate files. I can send them to you although it isn't very pretty yet as I am not so good at programming this. I'm going to try to make it into a single file when I get a chance.

[moosejons_dad]

Originally Posted by jsl21

moosejons_dad,

I think this is also because they changed the format of the paid version so what is being downloaded is only the first of six sections. Each section is now on a separate web page instead of the single page that it used to be on. I've modified the recipe to get each of the six sections as six separate files. I can send them to you although it isn't very pretty yet as I am not so good at programming this. I'm going to try to make it into a single file when I get a chance.

JSL-thanks for the offer to share the six files but I would rather wait for the one file for the paid version.
https://www.mobileread.com/forums/sho...&highlight=nyt
This is a forum that works on recipes and perhaps you could send them the six files and they would help you make one file that works and therefore everyone would be able to use this.
Thanks again for responding.

[moosejons_dad]

Originally Posted by kovidgoyal

the subscription version takes the articles linked to in the todays paper page on the wsj website. You will find many of the articles in the free version to be duplicated between successive downloads. PLus inthe free version you get partial articles.

Thanks for the info...

[craftycoder]

It's definitely not working like it should. The pay site only has the stories from the front page and table of contents are essentially worthless as mine mostly have one story per section. Screen shot below.

Also, how can we integrate the PDF image LINKED on each page in the file as well? I'd like to see the image in the PDF as the cover rather than default Calibre cover.

[kovidgoyal]

Those are the only articles linked to in the todays paper page

http://online.wsj.com/itp

[craftycoder]

Not true my friend. The pay site is broken up into 6 pages. Listed below. Parsing is no fun. I'm working on a solution now. I've never worked in Python before so this will be slow for me. I've got to have my newspaper though.

BTW, thank you so much for taking the time to develop such a useful tool. Calibre earns an A, and I'm a tough software critic.

http://online.wsj.com/itp/20100617/us/pageone
http://online.wsj.com/itp/20100617/us/opinion
http://online.wsj.com/itp/20100617/us/newyork
http://online.wsj.com/itp/20100617/us/marketplace
http://online.wsj.com/itp/20100617/us/moneyandinvesting
http://online.wsj.com/itp/20100617/us/personaljournal

[kovidgoyal]

Ah not being a WSJ reader, I missed the sections. Fixed recipe uploaded. I leave polishing to the WSJ enthusiasts