Referrer Help

yoss15 · 07-23-2012, 10:05 AM

I'm writing a recipe for a site which requires referers to access the articles properly. I set up the referer and the majority of the articles download fine but there are a lot which come out blank. I can't seem to find any pattern or reason why some aren't working.

Can anyone recommend some troubleshooting tips to figure out what the cause is?

Steven630 · 07-23-2012, 11:51 AM

I guess there's a limit to the referer trick. Use it too much, and it doesn't work. If that is the case, the same problem will occur even if you access the articles through a browser.How do you add referer to a recipe thoughï¼ŸSort of curious.

yoss15 · 07-23-2012, 01:33 PM

Yeah but it doesn't seem to go in any order. Its not like the first so many work and the rest stop working, so I'm wondering if there is someway to do it in two sessions or more, allowing it work.

Here is the code for the referer:

Code:

    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        cookies = mechanize.CookieJar()
        br = mechanize.build_opener(mechanize.HTTPCookieProcessor(cookies))
        br.addheaders = [('Referer','http://www."referralsite".com')]
        return br

Just replace "referralsite" with what you want the referer to be.

Steven630 · 07-24-2012, 12:04 AM

I tried the code. But it says global name 'mechanize' is not defined. Do I need to import something at the beginning of the recipe? I imported mechanize, but got a "NotImplementedError".

Perhaps you can try adding a delay between downloads?

yoss15 · 07-25-2012, 07:22 PM

You need an import mechanize statement at the beginning of the code.

Also oddly enough the issue only seems to come up when using the command line interface. I downloaded through the GUI with no issues, the only difference I know of being that the command line was .epub output and the GUI was .mobi.

Anyone have any idea why that might be?

07-23-2012, 10:05 AM	#1
yoss15 Enthusiast Posts: 37 Karma: 10 Join Date: Jul 2011 Device: Kindle	Referrer Help I'm writing a recipe for a site which requires referers to access the articles properly. I set up the referer and the majority of the articles download fine but there are a lot which come out blank. I can't seem to find any pattern or reason why some aren't working. Can anyone recommend some troubleshooting tips to figure out what the cause is?

07-23-2012, 01:33 PM	#3
yoss15 Enthusiast Posts: 37 Karma: 10 Join Date: Jul 2011 Device: Kindle	Yeah but it doesn't seem to go in any order. Its not like the first so many work and the rest stop working, so I'm wondering if there is someway to do it in two sessions or more, allowing it work. Here is the code for the referer: Code: def get_browser(self): br = BasicNewsRecipe.get_browser(self) cookies = mechanize.CookieJar() br = mechanize.build_opener(mechanize.HTTPCookieProcessor(cookies)) br.addheaders = [('Referer','http://www."referralsite".com')] return br Just replace "referralsite" with what you want the referer to be.

07-24-2012, 12:04 AM	#4
Steven630 Groupie Posts: 154 Karma: 10 Join Date: May 2012 Device: Kindle Paperwhite2	I tried the code. But it says global name 'mechanize' is not defined. Do I need to import something at the beginning of the recipe? I imported mechanize, but got a "NotImplementedError". Perhaps you can try adding a delay between downloads? Last edited by Steven630; 07-24-2012 at 12:12 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Help With Referrer	rth26	Recipes	3	04-10-2012 09:58 AM
Referrer problem with recipe	Starson17	Calibre	7	06-26-2010 04:47 PM

07-23-2012, 11:51 AM	#2
Steven630 Groupie Posts: 154 Karma: 10 Join Date: May 2012 Device: Kindle Paperwhite2	I guess there's a limit to the referer trick. Use it too much, and it doesn't work. If that is the case, the same problem will occur even if you access the articles through a browser.How do you add referer to a recipe thoughï¼ŸSort of curious.

07-25-2012, 07:22 PM	#5
yoss15 Enthusiast Posts: 37 Karma: 10 Join Date: Jul 2011 Device: Kindle	You need an import mechanize statement at the beginning of the code. Also oddly enough the issue only seems to come up when using the command line interface. I downloaded through the GUI with no issues, the only difference I know of being that the command line was .epub output and the GUI was .mobi. Anyone have any idea why that might be?

Advert

Advert