01-27-2021, 05:09 PM | #1 |
Member
Posts: 10
Karma: 10
Join Date: Jan 2021
Device: iPad
|
Adding requests to calibre?
Hi,
I'm trying to convert cal2ky3 plugin to python3, because it's awesome and the author hasn't been active for a while. This plugin uses http (http.client in python3) library to upload books to KyBook's web interface, and there's a nasty str and byte-mixing that worked in python2 but doesn't in python3. I'm pretty sure it's this problem: link But simply dumping everything into a byte string gives me Internal Server Error, probably due to different encodings but I'm not sure. Every time I google something about http.client, someone suggets using requests instead, but calibre doesn't have this module installed. Installing it on my python environment doesn't help. I have read the documentation for calibre plugins but couldn't find anything on this topic. Question - is there a way to add requests to calibre to circumvent the whole thing? Can it be imported from a plugin? I tried for a couple of evenings but I'm stuck because plugins are apparently .zipped when loaded into calibre. Thanks for any help! |
01-27-2021, 09:33 PM | #2 |
creator of calibre
Posts: 43,960
Karma: 22669822
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You can add requests to the plugin zip file and import it, see for example: https://www.mobileread.com/forums/sh...d.php?t=336389
|
01-29-2021, 05:21 PM | #3 |
Member
Posts: 10
Karma: 10
Join Date: Jan 2021
Device: iPad
|
Wow, thank you so much for replying!
Unfortunately, I'm still struggling to import responses... If I put requests and urrlib into my folder and put these inside the module: Code:
from calibre_plugins.kybook3_sync.urllib3 import urllib3 from calibre_plugins.kybook3_sync.requests import requests Code:
File "calibre/customize/zipplugin.py", line 192, in exec_module File "calibre_plugins.kybook3_sync.urllib3.util.connection", line 5, in <module> from urllib3.exceptions import LocationParseError ModuleNotFoundError: No module named 'urllib3' Essentially this code doesn't work with calibre: Code:
def _encode_multipart_formdata(self, fields, files): limit = '-----------------------------' num = str(int((datetime.now() - datetime(1970, 1, 1)).total_seconds())) limit = limit + num crlf = '\r\n' lines = [] for (key, value) in fields: lines.append('--' + limit) lines.append('Content-Disposition: form-data; name="%s"' % key) lines.append('') if isinstance(value,bytes): lines.append(value.decode()) else: lines.append(value) for (key, filename, value) in files: lines.append('--' + limit) lines.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename)) lines.append('Content-Type: %s' % self._get_content_type(filename)) lines.append('') if isinstance(value,bytes): lines.append(value.decode()) else: lines.append(value) lines.append('--' + limit + '--') lines.append('') body = crlf.join(lines) content_type = 'multipart/form-data; boundary=%s' % limit return content_type, body Code:
line = ''.join(line.split()) TypeError: sequence item 0: expected str instance, bytes found 1) Fixing the routine by changing all '' with b'' - didn't work due to me not knowing what's the encoding of a file I'm getting 2) Importing requests - a hassle, possible cross-platform issues What would you suggest as the 'neatest' and most calibre-like way of uploading a file to a web server that uses simple plain-text authorization? |
01-29-2021, 09:32 PM | #4 |
creator of calibre
Posts: 43,960
Karma: 22669822
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Use mechanize, it comes with calibre. https://mechanize.readthedocs.io/en/latest/
|
01-31-2021, 04:52 PM | #5 |
Member
Posts: 10
Karma: 10
Join Date: Jan 2021
Device: iPad
|
Hi Kovid!
Thanks again for your patience with my total lack of skills - I have some experience with python, but zero knowledge of web protocols and interactions. I'll have to impose on your kindness once more - I've been trying to implement file upload with mechanized and I'm thoroughly stuck. For context, plugin code with my changes is here: https://github.com/a-lagopus/KyBook3Sync/tree/develop The problematic piece of code is cal2ky3.py:787 KyBook3 exposes a web interface, with a "File Upload..." button coded like this in html: Code:
<div class="btn-toolbar"> <button type="button" class="btn btn-primary fileinput-button" id="upload-file"> <span class="glyphicon glyphicon-upload"></span> Upload Files… <input id="fileupload" type="file" name="files[]" multiple=""> </button> <button type="button" class="btn btn-success" id="create-folder"> <span class="glyphicon glyphicon-folder-close"></span> Create Folder… </button> <button type="button" class="btn btn-default" id="reload"> <span class="glyphicon glyphicon-refresh"></span> Refresh </button> </div> Any attempts at using a "find control" method to try and use add_file on it and upload the file causes an exception Code:
no control matching id 'fileupload' Code:
import time import urllib from base64 import b64encode from mechanize import Browser br = Browser() br.add_password("http://192.168.0.25:8080", "guest", "vNXD0d") br.set_handle_robots(False) response = br.open("http://192.168.0.25:8080/") for form in br.forms(): try: control = form.find_control(id="fileupload") print(control) except: print('Nope') Code:
<GET http://192.168.0.25:8080/ application/x-www-form-urlencoded <TextControl(<None>=)>> |
01-31-2021, 09:50 PM | #6 |
creator of calibre
Posts: 43,960
Karma: 22669822
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Is that inside an actual <form> tag? i.e. is it part of a form? If not, then it will be in the so called "global form". That is the form you will need to use in mechanize, I cant recall offhand how to access it, but a bit of googling should tell you that.
|
02-06-2021, 01:40 PM | #7 |
Member
Posts: 10
Karma: 10
Join Date: Jan 2021
Device: iPad
|
It was inside of a global form indeed.
I couldn't do the upload though, after a couple days worth of troubleshooting. However I finally managed to encode the raw http response properly by saving the request sent in py2 to a file and carefully checking py3 one against it. I'll save mechanize for some other time Thanks again for your help with this! I'll take discussion of the plugin itself to an appropriate branch, once I tested it more, but for now it seems to work with py3, so yay. |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Using Calibre server and calibre.db doesn't seem to be adding new titles | CraftyClown | Server | 10 | 01-04-2020 02:29 PM |
Feature Requests for Adding Books | toomuchreading | Library Management | 5 | 03-17-2015 03:15 PM |
Calibre & Covers: Adding books through Calibre vs dragging and dropping? | VirgoGirl | Calibre | 12 | 06-08-2014 05:34 AM |
calibre-server questions/requests | Ghallo | Library Management | 0 | 02-19-2011 06:02 PM |
Calibre newbie -- bugs, comments, and feature requests :) | partnerinflight | Calibre | 6 | 04-19-2010 11:01 PM |