WSJ downloads all links, but not all articles

[etmurphy]

Starting about a week ago (i.e. around March 26, 2022) the WSJ recipe downloads all links to articles in the Table of Contents, but does not actually download some of the articles. Typically the missing articles are the first three or four titles from the Front Page, U.S. News and World News sections. Fortunately the Opinion section has been reliably downloading all articles.

For example in the latest download from April 2, 2022 the following Front Page titles are included or not included (in this case every other one of the first six articles are missing):

- Ukraine Strikes Inside Russian Territory, Russians Say (missing)
- Russian Strategy in Ukraine Shifts After Setbacks, and a Lengthy War Looms (included)
- Roman Abramovich's Abrupt Transformation From Shunned Oligarch to Wartime Envoy (missing)
- They Attend Duke. And North Carolnia. Gulp. (included)
- Amazon Workers in New York Vote to Form Tech Giant's First U.S. Union (missing)
- U.S. March Jobs Report Shows Strong Hiring Momentum (included)
- Marketing Maverick Pitched Laser Mousetrap and Bone Fone Radio (included)
- Alex Mandl, Who Gave Up a Top AT&T Job to Lead a Startup, Has Died at Age 78 (included)
- Bar Code Pioneer Proved the Technology Was Robust (included)

Has anyone else experienced this problem or have ideas what might be the cause?

[kovidgoyal]

Post the urls to some of these missing articles.

[kaldrichjr]

Having same issue with both WSJ and New York Times recipes.

A couple examples of articles that are not dowloaded:
https://www.wsj.com/articles/egg-pri...ss_featst_pos1

https://www.wsj.com/articles/alaska-...ness_lead_pos7

The table of contents for each article will link to this article:
https://www.wsj.com/articles/hypebea...ts_pos2&page=1

Similar with NYT as well.

[etmurphy]

Originally Posted by kovidgoyal

Post the urls to some of these missing articles.

In the Monday, April 4 edition the first four Front Page articles are missing and all of the World News section. I cannot find the RSS Feed list of articles for the former, but following is the list page and URLs for the missing World News articles:

https://feeds.a.dj.com/rss/RSSWorldNews.xml

https://www.wsj.com/articles/oil-gas...pe-11649086062

https://www.wsj.com/articles/shangha...ts-11649053546

https://www.wsj.com/articles/release...ia-11649070001

https://www.wsj.com/articles/costa-r...on-11649073557

https://www.wsj.com/articles/the-wor...on-11649064601

The list page XML is also attached (since it will eventually cycle to tomorrow's listing).

Thank you for your help!

[kovidgoyal]

https://github.com/kovidgoyal/calibr...78c436a8a69b65

THe site seems to be in the middle of a transition to a new architecture, so there may be continuing issues till the transition is complete.

[magneticsubset]

Originally Posted by kovidgoyal

https://github.com/kovidgoyal/calibr...78c436a8a69b65

THe site seems to be in the middle of a transition to a new architecture, so there may be continuing issues till the transition is complete.

I used your updates and the recipie is not downloading the full articles. I have a wsj account. Today and yesterday for Frontpage it downloads just the first two paragraphs for all the articles.

[magneticsubset]

Upon further review there articles throughout the download that are just the first two paragraphs not just on the first page.

[kovidgoyal]

It will download two paragraphs if you are using the non-subscription recipe.

[magneticsubset]

Originally Posted by kovidgoyal

It will download two paragraphs if you are using the non-subscription recipe.

I am using the subscription recipie, I have a subscription and put in my login info. 60 - 80% of the articles fully downloaded. All the articles from the Frontpage on 4/10, 4/11, 4/12 and other sections had them spread out with the first two paragraphs.

[magneticsubset]

Today, the Frontpage Elon musk article, Ukraine zelensky and Moscow central banker articles didn't go past two paragraphs but the new relief article and investors turn cautious downloaded the whole article. Haven't gone past the front page section. Also is not the what's news section I know those are short this was frontpage.

[mjfriedman]

I continue to experience this issue and am using subscriber recipe with my credentials entered.

[kovidgoyal]

Well PM me your login details and the URLs for some articles that dont work an dI will see if I can reproduce.

[mjfriedman]

Sending credentials now. For today's paper, I receive the entire first article "Dow, S&P 500 Slide," but only the first two paragraphs of the second, "Inside Ukraine's Besieged Azovstal Steel Plant."

[ddavtian]

Kovid, did you get credentials for testing? If you need again, I can send mine to you. Last few days it was not downloading anything from WSJ. Today almost all articles have only first 2 paragraphs downloaded.

[kovidgoyal]

Yes I got them, but I am travelling, so looking at this is rather difficult for me till I return.