View Single Post
Old 03-12-2010, 02:14 PM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,120
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
When downloading articles it doesn't send any referrer. Each request leaves the browser state unchanged (this is so that the download can happen in multiple threads while using the same browser instance).

One possibility is to monkey path the open_novisit method on the browser instance to send the required referrer. so something like this

Code:
def get_browser(self):
    br = BasicNewsRecipe.get_browser(self)
    orig_open_novisit = br.open_novisit

    def my_open_no_visit(self, url, **kwargs):
        data = # add the referrer to the header
        return orig_open_novisit(url, data=data)

     br.open_novisit = my_open_no_visit
     return br
kovidgoyal is online now   Reply With Quote