Adding requests to calibre?

[A-Lagopus]

Hi,

I'm trying to convert cal2ky3 plugin to python3, because it's awesome and the author hasn't been active for a while.

This plugin uses http (http.client in python3) library to upload books to KyBook's web interface, and there's a nasty str and byte-mixing that worked in python2 but doesn't in python3.
I'm pretty sure it's this problem: link
But simply dumping everything into a byte string gives me Internal Server Error, probably due to different encodings but I'm not sure.

Every time I google something about http.client, someone suggets using requests instead, but calibre doesn't have this module installed. Installing it on my python environment doesn't help. I have read the documentation for calibre plugins but couldn't find anything on this topic.

Question - is there a way to add requests to calibre to circumvent the whole thing? Can it be imported from a plugin? I tried for a couple of evenings but I'm stuck because plugins are apparently .zipped when loaded into calibre.

Thanks for any help!

[kovidgoyal]

You can add requests to the plugin zip file and import it, see for example: https://www.mobileread.com/forums/sh...d.php?t=336389

[A-Lagopus]

Wow, thank you so much for replying!

Unfortunately, I'm still struggling to import responses...

If I put requests and urrlib into my folder and put these inside the module:

from calibre_plugins.kybook3_sync.urllib3 import urllib3
from calibre_plugins.kybook3_sync.requests import requests

Then I get this traceback:

  File "calibre/customize/zipplugin.py", line 192, in exec_module
  File "calibre_plugins.kybook3_sync.urllib3.util.connection", line 5, in <module>
    from urllib3.exceptions import LocationParseError
ModuleNotFoundError: No module named 'urllib3'

I wonder if I'm not approaching it from the right angle...

Essentially this code doesn't work with calibre:

def _encode_multipart_formdata(self, fields, files):
        limit = '-----------------------------'
        num = str(int((datetime.now() - datetime(1970, 1, 1)).total_seconds()))
        limit = limit + num
        crlf = '\r\n'
        lines = []
        for (key, value) in fields:
            lines.append('--' + limit)
            lines.append('Content-Disposition: form-data; name="%s"' % key)
            lines.append('')
            if isinstance(value,bytes):
                lines.append(value.decode())
            else:
                lines.append(value)
        for (key, filename, value) in files:
            lines.append('--' + limit)
            lines.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename))
            lines.append('Content-Type: %s' % self._get_content_type(filename))
            lines.append('')
            if isinstance(value,bytes):
                lines.append(value.decode())
            else:
                lines.append(value)
        lines.append('--' + limit + '--')
        lines.append('')
        body = crlf.join(lines)
        content_type = 'multipart/form-data; boundary=%s' % limit
        return content_type, body

in python3 it gives an error

line = ''.join(line.split())
TypeError: sequence item 0: expected str instance, bytes found

I wonder what's the most efficient way to resolve this is... So far I tried
1) Fixing the routine by changing all '' with b'' - didn't work due to me not knowing what's the encoding of a file I'm getting
2) Importing requests - a hassle, possible cross-platform issues

What would you suggest as the 'neatest' and most calibre-like way of uploading a file to a web server that uses simple plain-text authorization?

[kovidgoyal]

Use mechanize, it comes with calibre. https://mechanize.readthedocs.io/en/latest/

[A-Lagopus]

Hi Kovid!

Thanks again for your patience with my total lack of skills - I have some experience with python, but zero knowledge of web protocols and interactions.

I'll have to impose on your kindness once more - I've been trying to implement file upload with mechanized and I'm thoroughly stuck.

For context, plugin code with my changes is here:
https://github.com/a-lagopus/KyBook3Sync/tree/develop

The problematic piece of code is cal2ky3.py:787

KyBook3 exposes a web interface, with a "File Upload..." button coded like this in html:

<div class="btn-toolbar">
        <button type="button" class="btn btn-primary fileinput-button" id="upload-file">
          <span class="glyphicon glyphicon-upload"></span> Upload Files…
          <input id="fileupload" type="file" name="files[]" multiple="">
        </button>
        <button type="button" class="btn btn-success" id="create-folder">
          <span class="glyphicon glyphicon-folder-close"></span> Create Folder…
        </button>
        <button type="button" class="btn btn-default" id="reload">
          <span class="glyphicon glyphicon-refresh"></span> Refresh
        </button>
      </div>

I set up a mock mechanize app to try and upload something to it, but can't get it to work.
Any attempts at using a "find control" method to try and use add_file on it and upload the file causes an exception

no control matching id 'fileupload'

Tried 'fileupload', 'files[]', type=mechanize.FileControl - no luck.

import time
import urllib
from base64 import b64encode
from mechanize import Browser

br = Browser()
br.add_password("http://192.168.0.25:8080", "guest", "vNXD0d")
br.set_handle_robots(False)
response = br.open("http://192.168.0.25:8080/")

for form in br.forms():
    try:
        control = form.find_control(id="fileupload")
        print(control)
    except:
        print('Nope')

printing the forms gives me this:

<GET http://192.168.0.25:8080/ application/x-www-form-urlencoded
  <TextControl(<None>=)>>

Any help with this would be tremendously appreciated. I'm really willing to learn, but so far the learning curve proves a bit too steep for me...

[kovidgoyal]

Is that inside an actual <form> tag? i.e. is it part of a form? If not, then it will be in the so called "global form". That is the form you will need to use in mechanize, I cant recall offhand how to access it, but a bit of googling should tell you that.

[A-Lagopus]

It was inside of a global form indeed.
I couldn't do the upload though, after a couple days worth of troubleshooting.

However I finally managed to encode the raw http response properly by saving the request sent in py2 to a file and carefully checking py3 one against it.
I'll save mechanize for some other time

Thanks again for your help with this!
I'll take discussion of the plugin itself to an appropriate branch, once I tested it more, but for now it seems to work with py3, so yay.