Originally Posted by kovidgoyal
Try using the TamperData firefox extension to see exactly what happens when you fetch with a browser.
OK, the puzzle is solved - sort of. There were a couple of twists and turns. First, the difference was in the referrer. The site sends the image I don't want when the referrer is missing or not pointed to at least the root of the site.
I had tried blocking the referrer with the RefControl plugin, but it turns out that Firefox (or the plugin) will still send the referrer unless you shut it down first. That's why I was having trouble getting Firefox to emulate Calibre's recipe and that was tricky part #1.
The second tricky part was that TamperData seems to lie about the referrer. Apparently, it was showing the referrer FF would have sent, if not for the blocking of RefControl. Live HTTP Headers, however, was showing what was actually being sent.
For Firefox to get the same images that Calibre was getting, I had to clear the cache, <block> referrer with RefControl, then close FF and restart. (I was also removing cookies, but I'm not sure if that was necessary). To see what was really happening in FF, I had to watch with Live HTTP Headers.
What I'm not sure about is what referrer, if any, Calibre sends as a default. I haven't yet figured out how to watch the handshaking with mechanize. I tried some get_browser mods in the recipe to use the correct referrer, but so far it hasn't worked.