Customize article loading

bubak · 02-03-2024, 02:22 PM

Hi all,

I'm trying to write a recipe for a server where calibre loads only partial article. Interestingly, using wget or curl loads the whole article. So two questions: is there a way to find out why it differs and how to customize calibre API to get the whole article?
And if not, is there some recipe API function that accepts the URL and returns HTML or soup? Then I could call wget or curl from it.

kovidgoyal · 02-03-2024, 10:55 PM

You can override http access by implemneting get_browser in your recipe class to return an object with an open() method that returns a file like object from which the data is read.

bubak · 02-12-2024, 07:49 AM

I see, thanks. As a matter of fact, in turned out that the article contains all the text, even three times (!), but it is obfuscated by some preprocessing commands. So defining get_obfuscated_article() was enough.

02-03-2024, 02:22 PM	#1
bubak Connoisseur Posts: 63 Karma: 10 Join Date: Dec 2010 Device: kindle	Customize article loading Hi all, I'm trying to write a recipe for a server where calibre loads only partial article. Interestingly, using wget or curl loads the whole article. So two questions: is there a way to find out why it differs and how to customize calibre API to get the whole article? And if not, is there some recipe API function that accepts the URL and returns HTML or soup? Then I could call wget or curl from it.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Article link is hyperlinked into the text of every article of an RSS feed..	unkn0wn	Recipes	3	09-10-2021 03:50 AM
How do I customize this?	solid8546	Calibre	4	03-12-2017 04:14 PM
Yikes: "this article was downloaded by calibre" after every article	sheygetz	Calibre	11	05-29-2015 03:56 AM
headline of first article is shown, furthers are not in the article pages	DisasterArea	Recipes	1	02-02-2012 06:29 PM
Decorate article headings as hyperlinks to full article?	tomsem	Recipes	5	10-15-2010 09:30 PM

02-03-2024, 10:55 PM	#2
kovidgoyal creator of calibre Posts: 44,540 Karma: 24495948 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You can override http access by implemneting get_browser in your recipe class to return an object with an open() method that returns a file like object from which the data is read.

02-12-2024, 07:49 AM	#3
bubak Connoisseur Posts: 63 Karma: 10 Join Date: Dec 2010 Device: kindle	I see, thanks. As a matter of fact, in turned out that the article contains all the text, even three times (!), but it is obfuscated by some preprocessing commands. So defining get_obfuscated_article() was enough.

Advert