[thiago.eec]

Hi everyone.

I have a plugin called Skoob Sync. It uses webscraping to sync the user info. Now, the site started using Cloudfare when you try to login, giving me a 'HTTP Error 403: Forbidden'. The metadata plugin (Skoob Books) is not affected as it does not need to login.

I know this is not a calibre question, but since many of the plugin developers use webscraping, I thought that I might get some help here. Is there any way to bypass the Cloudfare check?

This is the relevant code for login:

Spoiler:

            self.opener = six.moves.urllib.request.build_opener(six.moves.urllib.request.HTTPCookieProcessor(self.cj))

            # Install our opener (note that this changes the global opener to the one
            # we just made, but you can also just call opener.open() if you want)
            six.moves.urllib.request.install_opener(self.opener)

            # Authentication page
            authentication_url = 'https://api.skoob.com.br/login'

            # Credentials
            payload = {
                'data[Usuario][email]': self.prefs['user'],
                'data[Usuario][senha]': self.key_password,
            }

            # Use urllib to encode the payload
            data = six.moves.urllib.parse.urlencode(payload).encode()

            # Build our Request object (supplying 'data' makes it a POST)
            login_req = six.moves.urllib.request.Request(authentication_url, data, headers=random_ua())

            login = six.moves.urllib.request.urlopen(login_req)

Spoiler:

# Get a random user agent from calibre. This is used on Skoob access.
def random_ua():
    try:
        from calibre import random_user_agent
        try:
            hdr = {'User-Agent': random_user_agent(allow_ie=False)}
            return hdr
        except TypeError:
            hdr = {'User-Agent': random_user_agent()}
            return hdr
    except ImportError:
        hdr = {'User-Agent': 'Mozilla/5.0 (Windows NT .1; Win64; x64)'}
        return hdr