Conversion to Python 3

[JSWolf]

There are a number of plugins where thh person who created them is no longer active and in the case of Modify ePub, Rev. Bob has said he doesn't know Python 3.

So is anyone willing step up and help do some conversion so these plugins can still be used come Calibre 4.99/5?

I don't know Python so I'm not able to help other then maybe to test.

[chaley]

Starting in Feb I have some time to look at this issue. No guarantees: I also don't know P3 (but I trust I can learn what I need to know), have no interest in some plugins, and some others might be too opaque. Of course I must first deal with my own ...

I suggest that people propose here moribund-but-used plugins and see if anyone signs up.

[BetterRed]

See List of PIs to be ported to Python 3

BR

[JSWolf]

Originally Posted by BetterRed

See List of PIs to be ported to Python 3

BR

One very much used plugin that's not yet been ported to Python 3 that's not in the list is the DeDRM plugin. Would someone like to take this on?

[jhowell]

Originally Posted by JSWolf

One very much used plugin that's not yet been ported to Python 3 that's not in the list is the DeDRM plugin. Would someone like to take this on?

That is probably best discussed on Apprentice Harper's GitHub. Any USA-based developers may want to be cautious about openly violating the DMCA's prohibition on developing DRM removal software.

[JSWolf]

Originally Posted by jhowell

That is probably best discussed on Apprentice Harper's GitHub. Any USA-based developers may want to be cautious about openly violating the DMCA's prohibition on developing DRM removal software.

I did bring it up in the comments section of the blog and so far nobody has taken up the job. I'm hoping by mentioning it here that someone will take up the job. And just because the job is taken up by someone here doesn't mean that whoever does it has to say he/she was the one who did it.

Besides, DRM violates DMCA.

[DiapDealer]

Originally Posted by JSWolf

Besides, DRM violates DMCA.

Stop being obtuse. It does no such thing. And it's completely irrelevant to your request, to boot.

[jhowell]

Originally Posted by JSWolf

I did bring it up in the comments section of the blog and so far nobody has taken up the job.

If you look at the GitHub pull requests you will see that someone did some Python 3 work in November, but nothing ever happened with it. Which brings up another issue, Apprentice Harper appears to have abandoned that project.

Originally Posted by JSWolf

Besides, DRM violates DMCA.

Huh?

[davidfor]

Originally Posted by chaley

Starting in Feb I have some time to look at this issue. No guarantees: I also don't know P3 (but I trust I can learn what I need to know), have no interest in some plugins, and some others might be too opaque. Of course I must first deal with my own ...

I suggest that people propose here moribund-but-used plugins and see if anyone signs up.

I think it will be the same as what happened for Qt5. We'll fix our own plugins, then fix the ones we use. I have been maintaining a few of kiwidude's, so will do those. And @JimmXinu has been doing the same. But, I'm not in any hurry. Kovid said it would be the middle of the year before the release. I'll do them when I feel like it. Or am doing something else in the plugins.

I'll admit to not knowing Python 3 as well. I have read a couple of brief things, but, I'm following the BFI* method. Start in debug mode, look at the errors for a plugin, fix them, try again. Repeat until it loads. Then test the function and repeat. Then move onto the next plugin. But, apply the changes I did before.

There are a few things I know. For example, all the metadata source plugins I am replacing:

from urllib.parse import quote, unquote
from urllib import quote, unquote

with:

try:
    from urllib.parse import quote, unquote
except ImportError:
    from urllib import quote, unquote
try:
    from queue import Empty, Queue
except ImportError:
    from Queue import Empty, Queue
import six
from six import text_type as unicode

That should keep backwards compatibility. With some removals and additions as needed.

Then, any "prints" have to be changed to wrap parentheses around the argument. So, instead of:

print "The code shouldn't have reached here"

it is:

print("The code shouldn't have reached here")

Another one I found was:

except Exception, e:

which should be:

except Exception as e:

Other changes are:

• use "items" instead of "iteritems" - Works in 2.7 but, could be slower. Luckily, I'm only using it on small numbers of items.
• "if whatever is None" rather than "if whatever" - I've been gradually converting this anyway as I prefer it.

If anyone can see a problem with these suggestions, or has a good list of other things to look for, I'd be happy to see it.


(*Brute force and ignorance)

[kovidgoyal]

@davidfor Run 2to3 it will print out a diff of all the problems it finds which is a convenient way to see what to change. After that grep for bytes, encode and decode that should cover most things.

[DaltonST]

I've ported 14 of my 15 GUI plugins such that they are simultaneously compatible with Calibre 4.6 and 4.99.3. During that tedious, lengthy and sometimes painful journey, I kept a list of frequent solutions to problems I encountered. I hope this helps folks.

# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

from polyglot.builtins import as_bytes, as_unicode, codepoint_to_chr, is_py3, iteritems, map, only_unicode_recursive, range, unicode_type

from polyglot.io import PolyglotStringIO  

if is_py3:
    from queue import Queue
else:
    from Queue import Queue

for k,v in iteritems(my_dict):


from calibre.utils.serialize import json_dumps, json_loads  


if is_py3:
      from html.parser import HTMLParser as myhtmlparser    
else:
    import HTMLParser 
    myhtmlparser = HTMLParser.HTMLParser

                 
if is_py3:
    from urllib.request import urlopen
    from urllib.parse import urlencode 
else:
    from urllib import urlencode
    from urllib2 import urlopen


if is_py3:
    my_list = as_unicode(my_list)
else:
    my_list = as_bytes(my_list)
my_list = ast.literal_eval(my_list)

if not isinstance(text,unicode_type):        #unicode is a Py2 type; use Polyglot's unicode_type instead. 
     

You will see a lot of these errors:
TypeError: a bytes-like object is required, not 'str'
TypeError: can only concatenate str (not "bytes") to str 


Py3 redefined what a bytestring is internally:
The bytestring 'A' is this in Python 2:  A
the bytestring 'A' is this in Python 3:   (a 3 digit integer)

Python2 was very forgiving and helpful in converting bytestrings to and from unicode when, for example, concatenating them. It just did it silently, and was unknowingly taken for granted.

Python3 is not at all forgiving and not at all helpful in converting anything to anything. Hence, errors such as: 'TypeError: can only concatenate str (not "bytes") to str' will happen if you try to concatenate text with, for example, a bytes version of an integer. Ditto for doing a str.replace(x,y); both x and y must be strings, not bytestrings, since a str in Py3 is an instance of 'str', not 'bytes'.



DaltonST

[JimmXinu]

Originally Posted by davidfor

I think it will be the same as what happened for Qt5. We'll fix our own plugins, then fix the ones we use. I have been maintaining a few of kiwidude's, so will do those. And @JimmXinu has been doing the same. But, I'm not in any hurry. Kovid said it would be the middle of the year before the release. I'll do them when I feel like it. Or am doing something else in the plugins.

Yeah, I leapt into it when the Kovid posted the py3 beta versions, but I felt a lot less urgency after that 'middle of the year' time frame came out.

Plus I've seen very little response to posting py3 plugin test versions.

That said, I do plan to release FanFicFare including the py3 changes in the next week or so. That's going to be my first canary.

[Terisa de morgan]

Thank you very much to davidfor, kovigoyal and DaltonST for their comments about the conversion. I have some work to do myself and it will go a good way to begin.

[Terisa de morgan]

Originally Posted by JimmXinu

Plus I've seen very little response to posting py3 plugin test versions.

I think that, mainly, as there is no change in calibre or the plugin, no urgency to test. Some days before the change, if it is announced, the rush will begin.

[davidfor]

Originally Posted by Terisa de morgan

I think that, mainly, as there is no change in calibre or the plugin, no urgency to test. Some days before the change, if it is announced, the rush will begin.

There is no way that that would happen. I'm sure no-one here stayed up all night to finish their school assignments. Or to study for an exam. I'm sure we all carefully plan things out so that it's all completed well ahead of time so we can relax and do other things. I'm sure that no-one here has ever explained to a Project Manager that when they said "the end of the week" they meant "Friday at 81 o'clock".


And the above was written with my tongue firmly in my cheek