[bobbysteel]

Hi Kovid (and others)
I'm looking at customizing a recipe now which requires a particular referer. I've found old code here suggesting something like the following would work.

    def get_browser(self):
        USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
        br = BasicNewsRecipe.get_browser(self, user_agent=USER_AGENT)
        orig_open_novisit = br.open_novisit
        br.set_debug_http(True)
        br.set_debug_responses(True)
        br.set_debug_redirects(True)
        def my_open_no_visit(url, **kwargs):
                req = mechanize.Request(url, headers = {'Referer':'https://www.facebook.com/'})
                return orig_open_novisit(req)
        br.open_novisit = my_open_no_visit
        return br

However when running this the logs show every other header but referer set. Is there some other change that's happened that would invalidate this?

1% Fetching feeds...
send: 'GET /XXXXXXXXXX HTTP/1.1\r\nAccept-Encoding: identity\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0\r\nHost: www.XXXXXXXX.com\r\nAccept: */*\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Content-Type-Options: nosniff
header: X-Download-Options: noopen
header: X-Xss-Protection: 1; mode=block
header: Cache-Control: max-age=0, no-cache, no-store, must-revalidate
header: Link: <//www.XXXXXXXXXX.com/assets/main.css>; as="style"; rel="preload"; nopush; as="script"; rel="preload"; nopush, as="script"; rel="preload"; nopush
header: Content-Type: text/html; charset=utf-8
header: Etag: W/"6f1ba-VqmRNsUPEdGWrd9DOPy5aS7TMf8"
header: Content-Length: 455098
header: Accept-Ranges: bytes
header: Date: Tue, 20 Feb 2018 12:04:26 GMT
header: Age: 95
header: Connection: close
header: Vary: flags, anonymous-user, edition, Accept-Encoding
header: Set-Cookie: Allocation=2d4c; Expires=Tue, 09 Mar 2021 18:00:00 GMT; Domain=.XXXXXXXX.com; Path=/
header: Set-Cookie: x-policy=-; Max-Age=60; Domain=.XXXX.com; Path=/
header: Set-Cookie: x-privileged-referer-model=(null); Max-Age=60; Domain=.XXXXX.com; Path=/
header: Set-Cookie: x-privileged-referer-phase=(null); Max-Age=60; Domain=.XXXX.com; Path=/
header: Strict-Transport-Security: max-age=2592000
in section:   Front Page & Second Front
title:   XXXXXXXXXXXXXXXX   url:  https://www.XXXXXXXXXXXXX.com/XXXXXXXXXXXXXXXXXXXX