Beta testers needed

[kovidgoyal]

@davidfor: Thanks. Hopefully there aren't a lot of plugins that use their own private beautifulsoup modules.

[eschwartz]

Originally Posted by beedaddy

Stupid question: And the betas are Python 3? I downloaded calibre-3.40.1-x86_64.txz from the beta directory, and "debug device detection" tells me

The code has seen a lot of work to improve python 3 compatibility:

• several bits of vendored code have been devendored, so calibre now uses an external feedparser, markdown, html2text, and beautifulsoup4 -- this means calibre is ported from the old beautifulsoup3 to the new beautifulsoup4, which should be nice too.
• all the C extensions now compile cleanly on python3 (and seem to work)
• a "polyglot" module has been written to provide various bits of functionality that are changed on python2/python3, and is now being extensively used
• all code is able to import properly
• It's now possible to fully bootstrap, build, and even install, and is sufficient to *try* starting it...
• you can actually run the testsuite without it erroring out during the "search for tests" stage

... but there is still a ton more work to do. At the moment, the GUI doesn't start at all and there are over a hundred failures in the testsuite alone... but it's a start, and it means we can now easily test calibre.

If anyone has the spare time and knowledge, please feel free to help with the port.

In order to experiment with python3, you will need to compile it from source. You will need to be running linux, and you will need to install all the usual build-from-source dependencies, as listed here: https://github.com/kovidgoyal/build-...s/sources.json
Except for each PyPI dependency, you will want to install using python3 instead of python2.

Then, simply run the bootstrap process listed at https://github.com/kovidgoyal/calibr...TALL.rst#build
Except instead of using python2, use python3, and furthermore, you must set the environment variable CALIBRE_PY3_PORT.

So,

CALIBRE_PY3_PORT=1 python3 setup.py bootstrap

Then you can hack the code, try to run setup.py test, etc.

For Arch Linux users, the "calibre-git" package in the AUR has a good description of the build process, and a list of dependencies -- simply swap out "python-" for each "python2-*" dependency.

[Terisa de morgan]

Originally Posted by kovidgoyal

No it is still python 2 and no plugins should be affected.

This is the error for the Overdrive plugin:

Spoiler:

Signing in to BPL: : AttributeError ('function' object has no attribute 'HTML_ENTITIES')
Traceback (most recent call last):
File "calibre_plugins.overdrive_link.jobs", line 968, in sign_in
File "calibre_plugins.overdrive_link.overdrive", line 332, in sign_in
AttributeError: 'function' object has no attribute 'HTML_ENTITIES'

It doesn't happen with the standard calibre.

[eschwartz]

Originally Posted by Terisa de morgan

This is the error for the Overdrive plugin:

Spoiler:

Signing in to BPL: : AttributeError ('function' object has no attribute 'HTML_ENTITIES')
Traceback (most recent call last):
File "calibre_plugins.overdrive_link.jobs", line 968, in sign_in
File "calibre_plugins.overdrive_link.overdrive", line 332, in sign_in
AttributeError: 'function' object has no attribute 'HTML_ENTITIES'

It doesn't happen with the standard calibre.

https://www.crummy.com/software/Beau.../doc/#entities

This no longer exists in bs4.

[Terisa de morgan]

Yes, I've seen it, and the plugin works removing that part. And that means that more plugins can be affected, even if they are using python 2.

[kovidgoyal]

Yes plugins that happen to sue beautifulsoup in ways tat are incompatible with bs4 will be affected. They can should update their code to be bs4 compatible.

[Terisa de morgan]

There is a problem with the CountPages plugin, and it looks like it's related with the jobs. It doesn't calculate the number of pages (Adobe method) and the calibre debug log is:

Spoiler:

-------------------------------
Logfile for book ID 14487 (Murder at the Marina)
Method of counting _page_count_mode=Estimate _download_sources=[]
results= {}
14487
do_statistics_for_book: C:\Users\moret\AppData\Local\Temp\calibre_trdzoz\0 ynht3_count_pages\14487.epub 2 Estimate [] [u'PageCount'] 1500 False
Traceback (most recent call last):
File "calibre_plugins.count_pages.jobs", line 155, in do_statistics_for_book
File "calibre_plugins.count_pages.statistics", line 53, in get_page_count
File "calibre_plugins.count_pages.statistics", line 92, in _open_epub_file
File "site-packages\calibre\ebooks\oeb\iterator\book.py", line 149, in __enter__
File "site-packages\calibre\ebooks\oeb\iterator\book.py", line 88, in run_extract_book
File "site-packages\calibre\utils\ipc\simple_worker.py", line 230, in fork_job
File "site-packages\calibre\utils\ipc\simple_worker.py", line 123, in communicate
WorkerError: Worker failed

[davidfor]

Originally Posted by Terisa de morgan

There is a problem with the CountPages plugin, and it looks like it's related with the jobs. It doesn't calculate the number of pages (Adobe method) and the calibre debug log is:

Spoiler:

-------------------------------
Logfile for book ID 14487 (Murder at the Marina)
Method of counting _page_count_mode=Estimate _download_sources=[]
results= {}
14487
do_statistics_for_book: C:\Users\moret\AppData\Local\Temp\calibre_trdzoz\0 ynht3_count_pages\14487.epub 2 Estimate [] [u'PageCount'] 1500 False
Traceback (most recent call last):
File "calibre_plugins.count_pages.jobs", line 155, in do_statistics_for_book
File "calibre_plugins.count_pages.statistics", line 53, in get_page_count
File "calibre_plugins.count_pages.statistics", line 92, in _open_epub_file
File "site-packages\calibre\ebooks\oeb\iterator\book.py", line 149, in __enter__
File "site-packages\calibre\ebooks\oeb\iterator\book.py", line 88, in run_extract_book
File "site-packages\calibre\utils\ipc\simple_worker.py", line 230, in fork_job
File "site-packages\calibre\utils\ipc\simple_worker.py", line 123, in communicate
WorkerError: Worker failed

I ran Count Pages a couple of times last night with no issues. That was 3.40.2 on a Win10 box. Just tried beta 3.40.3 on Ubuntu with no problems either.

The error above is when the book is being unpacked. I don't think the plugin is doing anything strange there.

[Terisa de morgan]

Originally Posted by davidfor

I ran Count Pages a couple of times last night with no issues. That was 3.40.2 on a Win10 box. Just tried beta 3.40.3 on Ubuntu with no problems either.

The error above is when the book is being unpacked. I don't think the plugin is doing anything strange there.

Not the plugin but calibre itself. I'll have to debug it a bit more, because it happens with all my ebooks, it's not something random. And it's 3.40.3 on Windows 10.

[davidfor]

Originally Posted by Terisa de morgan

Not the plugin but calibre itself. I'll have to debug it a bit more, because it happens with all my ebooks, it's not something random. And it's 3.40.3 on Windows 10.

I have tried at home with 3.40.3 on Windows 10 and there were no problems.

[Terisa de morgan]

Originally Posted by davidfor

I have tried at home with 3.40.3 on Windows 10 and there were no problems.

I'll check again then.

[PandathePanda]

I got this when I started the beta (calibre-3.40.3-x86_64):

[ilovejedd]

Comments field is behaving weirdly for me in the beta. The <br> tags in comments are getting duplicated on all book records.

The following is from the exact same library. Note, I don't actually have a new line character [\n] in the Calibre library comments. Just added it here for readability.

Calibre Portable 3.40.1 (Windows 7 x86 and x86-64)

<p class="description">Published: 2016-10-31 Updated: 2019-02-03 Chapters: 71/? Words: 171444
<br>
<br><b>Summary</b>
<br>"But you, Lord Snow, you'll be fighting their battles forever." Ser Alliser Thorne
<br>
<br>Every time he died his last in that life he awoke again in another at the exact moment of Ghost's birth.</p>

Calibre Portable 3.40.3 (Windows 7 x86 and x86-64)

<p class="description">Published: 2016-10-31 Updated: 2019-02-03 Chapters: 71/? Words: 171444
<br>
<br>
<br>
<br><b>Summary</b>
<br>
<br>"But you, Lord Snow, you'll be fighting their battles forever." Ser Alliser Thorne
<br>
<br>
<br>
<br>Every time he died his last in that life he awoke again in another at the exact moment of Ghost's birth.</p>

[kovidgoyal]

@PandaThePanda, that's weird but I will add some code to ensure it does not happen. Can you run calibre in debug mode and post the log, that popup indicates that downloading the latest version is failing for you, I'd like to know why.

[kovidgoyal]

@ilovejedd: I tried pasting the provided HTML into the comments field for a book (source view of edit metadata dialog) then clicked OK. Then tried opening the edit metadata dialog again, there were no extra <br>s. Can you provide some info on how you are getting them?