Difficulties with calibre news recipe to detect elements within angularjs

itssudipok · 07-27-2021, 12:40 PM

While preparing a customized news recipe for a website, I am not able to pick up the necessary elements such as the images, articles and so on. This happens with all elements with class 'ng-binding' or similar things.

However, the elements on the same page e.g., the main title with logo without this angularjs specifier, are being downloaded at ease.

Any help would be greatly appreciated. Thanks in advance!

kovidgoyal · 07-27-2021, 09:55 PM

those are probably loaded dynamically using javascript. Typically the data will be present embedded on the page as JSON, from where you can extract it, or you will need to use developer tools in a browser to detect what XHR requests are being made to load the data and send them in the recipe.

itssudipok · 07-28-2021, 09:56 AM

Thank you very much for your response!
I tried to follow your suggestions with whatever little knowledge I have, and it works well to some extent.

But I want to do the following:

1. I need the images related to the articles, which is not there right now. Like your 'The Hindu' recipe, I would like to place all the images at the beginning of the articles.

2. The articles show all the 'p' tags with 'span', which is understandable because I am directly feeding the XHR requests. I want to get rid of this.

3. I want to have more control on the json structure, I have no experience working with this. I tried to clean up things using 'remove_tags' and 'keep_only_tags', but got no luck yet.

4. And most importantly, I would like to know whether the whole process of getting the XHR links can be automated through the recipe itself or not.

You may also suggest some references, I would love to spend some time on this.

Thanks!

07-27-2021, 12:40 PM	#1
itssudipok Junior Member Posts: 6 Karma: 10 Join Date: Jul 2021 Location: West Bengal Device: Kindle Paper White 11th Gen	Difficulties with calibre news recipe to detect elements within angularjs While preparing a customized news recipe for a website, I am not able to pick up the necessary elements such as the images, articles and so on. This happens with all elements with class 'ng-binding' or similar things. However, the elements on the same page e.g., the main title with logo without this angularjs specifier, are being downloaded at ease. Any help would be greatly appreciated. Thanks in advance!

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre news recipe fetching articles from knappily.	patoliadixit	Recipes	7	09-23-2020 06:10 AM
Calibre News Source 'The Hindu' Recipe needs correction	snji	Recipes	12	10-05-2018 11:11 PM
Calibre News Recipe not downloading multiple page articles	Raptor	Recipes	0	02-03-2015 12:29 AM
Adding recipe to Calibre news sources	martin.tsanchev	Recipes	1	02-13-2012 12:57 PM
Adding recipe to Calibre news sources	martin.tsanchev	Recipes	1	08-11-2011 12:45 PM

07-27-2021, 09:55 PM	#2
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	those are probably loaded dynamically using javascript. Typically the data will be present embedded on the page as JSON, from where you can extract it, or you will need to use developer tools in a browser to detect what XHR requests are being made to load the data and send them in the recipe.

07-28-2021, 09:56 AM	#3
itssudipok Junior Member Posts: 6 Karma: 10 Join Date: Jul 2021 Location: West Bengal Device: Kindle Paper White 11th Gen	Thank you very much for your response! I tried to follow your suggestions with whatever little knowledge I have, and it works well to some extent. But I want to do the following: 1. I need the images related to the articles, which is not there right now. Like your 'The Hindu' recipe, I would like to place all the images at the beginning of the articles. 2. The articles show all the 'p' tags with 'span', which is understandable because I am directly feeding the XHR requests. I want to get rid of this. 3. I want to have more control on the json structure, I have no experience working with this. I tried to clean up things using 'remove_tags' and 'keep_only_tags', but got no luck yet. 4. And most importantly, I would like to know whether the whole process of getting the XHR links can be automated through the recipe itself or not. You may also suggest some references, I would love to spend some time on this. Thanks!