![]() |
#1 |
Wannabe Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 426
Karma: 2516674
Join Date: Apr 2011
Location: Geelong, Australia
Device: Kobo Libra 2, Kobo Aura 2, Sony PRS-T1, Sony PRS-350, Palm TX
|
AutoSport.com recipe
Hi folks,
Here's a recipe for autosport.com. Please be gentle, this is the first time I've written any python (although it's mostly cut'n'paste from other recipes). There's many RSS feeds available on the autosport site, but I've used only two (that I happen to be interested in). Is there a way you can enable/disable many feeds within a single recipe? I also suspect that the recipe might work without a subscription for parts of the autosport site but I haven't played around with that. This recipe uses the method here to only download stories once. Here's the recipe. Feedback appreciated! Spoiler:
Cheers, Simon. |
![]() |
![]() |
![]() |
#2 |
Wannabe Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 426
Karma: 2516674
Join Date: Apr 2011
Location: Geelong, Australia
Device: Kobo Libra 2, Kobo Aura 2, Sony PRS-T1, Sony PRS-350, Palm TX
|
Well, after many years of working perfectly it looks like Autosport has switched to some javascript based API mess outsourced via tinypass.com to log in. With only a vague understanding of python and getting completely lost in with APIs and callbacks after a short play with the Chrome developer tools, is reverse engineering something like this going to be overly difficult?
The login page is at https://www.autosport.com/userlogin. Edit: I've had a quick look at the WSJ recipe. That seems to look a lot simpler than what Autosport are doing. |
![]() |
![]() |
![]() |
#3 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 45,356
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
There is not straightforward way to fix JS login issues. You basically have to find the fields in the HTTP requests the browser sends, figure out how to fill them and send the same requests manually in the recipe.
|
![]() |
![]() |
![]() |
#4 |
Wannabe Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 426
Karma: 2516674
Join Date: Apr 2011
Location: Geelong, Australia
Device: Kobo Libra 2, Kobo Aura 2, Sony PRS-T1, Sony PRS-350, Palm TX
|
Thanks Kovid, I feared as much. I saw something like 650 HTTP requests to load the login page and actually log in
![]() |
![]() |
![]() |
![]() |
#5 |
Member
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 18
Karma: 12710
Join Date: Feb 2020
Device: Kindle Paperwhite
|
Hi Simon,
I hadn't see your recipe before, but I had also just created a custom one to handle the login as I am an Autosport subscriber too. The javascript and tinypass authentication has made things much more tricky. I have some python code that has managed to login in jupyter. I need to translate that into a recipe. I would recommend https://the-race.com/ now that half the autosport F1 writers have gone freelance. Currently it doesn't require a subscription, but that may change soon. Last edited by pacha2; 03-12-2020 at 08:28 AM. |
![]() |
![]() |
![]() |
#6 | ||
Wannabe Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 426
Karma: 2516674
Join Date: Apr 2011
Location: Geelong, Australia
Device: Kobo Libra 2, Kobo Aura 2, Sony PRS-T1, Sony PRS-350, Palm TX
|
Quote:
![]() Quote:
|
||
![]() |
![]() |
![]() |
#7 |
Member
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 18
Karma: 12710
Join Date: Feb 2020
Device: Kindle Paperwhite
|
Yeah the-race RSS is https://the-race.com/feed/
I've pasted and attached my very simple recipe, just adding the cover_url and masthead_url Spoiler:
|
![]() |
![]() |
![]() |
#8 | |
Wannabe Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 426
Karma: 2516674
Join Date: Apr 2011
Location: Geelong, Australia
Device: Kobo Libra 2, Kobo Aura 2, Sony PRS-T1, Sony PRS-350, Palm TX
|
Quote:
|
|
![]() |
![]() |
![]() |
#9 |
Member
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 18
Karma: 12710
Join Date: Feb 2020
Device: Kindle Paperwhite
|
I did some more digging and found there is also a pure F1 feed.
https://the-race.com/category/formula-1/feed/ |
![]() |
![]() |
![]() |
#10 |
Member
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 18
Karma: 12710
Join Date: Feb 2020
Device: Kindle Paperwhite
|
![]()
After much trial, error, pain and debugging I have the Autosport login working.
I have tested this for a couple of days and seems to work for me. Let me know how you get on. Spoiler:
|
![]() |
![]() |
![]() |
#11 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 45,356
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Do you have to hardcode the user agent? The recipe system automatically uses up-to-date user agents for desktop browsers. Also content-type should not be needed in add headers.
|
![]() |
![]() |
![]() |
#12 |
Member
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 18
Karma: 12710
Join Date: Feb 2020
Device: Kindle Paperwhite
|
Hi Kovid,
Yes, I think you are correct. I couldn't get Content-Type': 'application/json' to ever be used as the header as with the data, so must have just left that in. When does the user agent get chosen? Can I easily verify what user agent is being used? |
![]() |
![]() |
![]() |
#13 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 45,356
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
If you wish to set custom headers for a request, you can simply create a mechanize.Request object and pass it a headers dictionary. Then pass the request object to browser.open() instead of the URL.
The user agent is set when the browser object is created in get_browser(). You can see it by simply printing out browser.addheaders |
![]() |
![]() |
![]() |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Recipe works when mocked up as Python file, fails when converted to Recipe | ode | Recipes | 7 | 09-04-2011 04:57 AM |
Need Help with Recipe | UtahJames | Recipes | 1 | 04-12-2011 09:50 AM |
New recipe | kiklop74 | Recipes | 0 | 10-05-2010 04:41 PM |
Recipe Help | lrain5 | Calibre | 3 | 05-09-2010 10:42 PM |
Recipe Help Please | estral | Calibre | 1 | 06-11-2009 02:35 PM |