[GUI Plugin] KyBook3 Sync - replaced

[karl1c]

The next Update will only be available to those who have donated £1 or more.

If you're using this plugin, please leave a comment to say whether it works for you. Thanks.

Does anyone with some Python experience want to help with developing this plugin? If so, I'd be happy to make it available. PM me.

Now ONLY syncs those file formats in the comma-separated list in settings.

This plugin provides master-slave synchronisation of books and metadata from Calibre to KyBook3 (http://kybook-reader.com).

KyBook3 provides support for almost every book and audiobook format and can read metadata from them. However, many files have incorrect metadata and/or cover. Even when KyBook3 downloads metadata and covers from Calibre's Content Server some metadata and cover errors occur. This plugin is an attempt to overcome these shortcomings.

Main Features:

• Upload selected books to KyBook3 (skips files with the same MD5 already in KyBook3)
• Sync title, authors, publishers, subjects, series, publication date, language, comments, ratings, (some) identifiers, & cover to KyBook3 with the following mapping:
  Calibre Tags -> KyBook3 Subjects
  Calibre Comments -> KyBook3 Annotations
  
• Backup of KyBook3's metadata

Special Notes:

• The plugin assumes that Calibre's metadata is more correct than KyBook3's
• Neither books nor metadata in Calibre are changed.
• No book files in KyBook3 will be changed or overwritten.
• Metadata in KyBook3 is overwritten by that in Calibre
• You should probably ensure that Calibre's Tags are single words with an initial capital
• Calibre's Comments can (optionally) have HTML stripped
• There is currently no way to sync from KyBook3 to Calibre (although this is planned)
• Sync takes place over Wi-Fi, so may take a long time. I suggest you sync in small batches. Alternatively, you can add all the books you want in KyBook3 from iTunes and then run KyBook3 Sync to sync just the metadata

Testimonials:

• "... syncing calibre with an actual reader app is a dream come true, thank you."

Installation Steps:

1. Download the attached zip file and install the plugin/restart Calibre/add to context menu or toolbar as described in the Introduction to plugins thread
2. Click on the KyBook3 Sync icon to see KyBook3's menu
3. Click on "Customize this plugin" in KyBook3's menu and set the Link, Username, & Password used by KyBook3's Content Server
4. You should now be able to select some books in Calibre and sync them to KyBook3

Debugging:
If you have problems with the plugin:

1. Be patient, syncing over WiFi can be slow and if your first book is a 500MB PDF, progress will be "stuck" at 1% for a long time!
2. In Calibre click Preferences | Restart in debug mode
3. After the restart, try running the plugin again
4. When it fails, or it appears to get stuck, close Calibre
5. A text file will appear with debug info, copy the contents and paste into a post here between spoiler tags (the icon with an eye and a red x) or save the text file to Dropbox and paste a link to it here.

Paypal Donations:

• If you find this plugin useful please feel free to show your appreciation. I have spent many unpaid hours in its development and support so any encouragement for me to continue is appreciated!
  

Version History:

Spoiler:

Version 1.0.11 - 18 April 2019
Corrected error when a book had no language assigned

Version 1.0.10 - 10 April 2019
Added syncing of series number (thanks to nimapourkarimi)
Corrected error when no files of the required formats were found
Removed setting ebookid fields to NULL in authors & sequences tables

Version 1.0.9 - 08 April 2019
Corrected error where /$User/covers was not created correctly from Windows (thanks to ilovejedd)

Version 1.0.8 - 08 April 2019
Now ONLY syncs formats specified in settings

Version 1.0.7 - 07 April 2019
Added syncing of rating
Added syncing of some identifiers (isbn, amazon, asin, oclc)
Corrected error in presentation of SQL in debug output

Version 1.0.6 - 05 April 2019
Added automatic logging when running as a plugin
Improved robustness of file uploads

Version 1.0.5 - 04 April 2019
Added progress bar for uploading DB file
Tidied up failure when Content Server on KyBook3 has not been started

Version 1.0.4 - 03 April 2019
Corrected creation of zip file (thanks to ilovejedd)
Added dynamic setting of Calibre's library path (thanks to ilovejedd)
Corrected error where some subjects appeared more than once in KyBook3

Version 1.0.3 - 02 April 2019
Added support for Windows paths
Corrected error in syncing series (thanks to nimapourkarimi)

Version 1.0.2 - 02 April 2019
Added support for Calibre's GUI Debug mode
Corrected filling lookups with NULL values

Version 1.0.1 - 01 April 2019
Changed port number and password for client/listener.

Version 1.0.0 - 01 April 2019
Initial release of KyBook3 Sync plugin

Future Development:
I plan to add the following features:

• syncing of identifiers (isbn, issn, etc.) [in progress]
• use Calibre's uuid to avoid calculating md5 and speed up processing
• adding to Calibre books found in KyBook3, but not already in Calibre
• syncing metadata back from KyBook3 to Calibre
• syncing of collections (does anyone actually use collections?)

[nimapourkarimi]

Thanks for this plugin, I can successfully sync some books but randomly encounter this error:
calibre, version 3.40.1 (darwin, embedded-python: True)
Failed to sync with KyBook3: Failed: Sync 3 books. Did you start KyBook3's Content Server?

Starting job: Sync 3 books
Sync complete, with 0 failures
Job: "Sync 3 books" failed with error:
Traceback (most recent call last):
File "site-packages/calibre/gui2/threaded_jobs.py", line 84, in start_work
File "calibre_plugins.kybook3_sync.jobs", line 109, in sync_threaded
EOFError

Called with args: ([30455, 30454, 30453], <calibre.db.cache.Cache object at 0x1192c18d0>) {u'log': <calibre.utils.logging.GUILog object at 0x127615b50>, u'abort': <threading._Event object at 0x127615ad0>, u'notifications': <Queue.Queue instance at 0x1220dccf8>}


...and I'm sure Kybook3's content server is on.

[nimapourkarimi]

I found out that this problem occurs with some of my PDF books, some of PDF files and all of my epub files sync correctly.

Edit: Nope, it happens with epubs too.

[karl1c]

@nimapourkarimi: Well that wasn't a good start was it? :-(

I've uploaded an amended version. Give that a go and see whether it is any better.

[nimapourkarimi]

Well no, it does that again with certain books.

[karl1c]

1. What OS are you using?
2. Are the same books failing each time?
3. Is there any commonality in the books that fail? (Very large, specific format, etc.)

[nimapourkarimi]

1. macOS Mojave 10.14.4 running calibre 3.40.1
2. Yes
3. No, I can sync a 250mb PDF file but a certain 2mb pdf file won't sync, that applies to epubs too, the file copies to kybook successfully but the process fails while syncing metadata to kybook.

[karl1c]

I've uploaded another version, give that a go.

[nimapourkarimi]

There is no error anymore but it gets stuck at metadata sync the moment it reaches those certain books and it remains in that position. I gave it some time but nothing happened and had to kill the job eventually.

[ilovejedd]

Tried the plugin. Job progress is just stuck at 1%.

Tested KyBook3 content server link is accessible via Chrome.

1. What OS are you using?
   Windows 10 Pro 64-bit Build 17134
   
2. Are the same books failing each time?
   all books fail to sync
   
3. Is there any commonality in the books that fail? (Very large, specific format, etc.)
   all books fail to sync (epub, 15KB - 1MB)

P.S.
Is there a way to disable the mini top menu bar and clock in KyBook3?

[karl1c]

@ilovejedd I suspect this was a pathing issue. Unfortunately, I don't have Windows to test it.

I've uploaded a new version that I hope will work. If not, and you're happy to help with some debugging, PM we and I'll try and help you get it working.

[ilovejedd]

Had a look at the source code and am I correct in guessing that you hard coded the "Calibre Library" folder name and that the plugin assumes it's located in the home directory?

I'm not familiar with python nor programming Calibre plugins but I believe there must be a different method of getting the library path(s).


Not sure what the relevant pieces of code are but I think this is probably a good example.

Job Spy/visualize_metadata_dialog.py

import os,sys,apsw
from PyQt5.Qt import (Qt, QDialog, QGridLayout, QVBoxLayout, QHBoxLayout, QFont,
                                        QWidget, QSize, QPushButton, QComboBox, QGroupBox, QSplitter )  # for visualizing
from PyQt5.Qt import QFileDialog,QObject,QApplication  # for exporting a .csv

from calibre import isbytestring
from calibre.constants import filesystem_encoding, DEBUG
from calibre.gui2 import gprefs, question_dialog, info_dialog

from calibre_plugins.job_spy.qpainter_charts import Chart, PieChart, ViewerLegend, ViewerPieChart, DataTable
from calibre_plugins.job_spy.config import prefs

#-----------------------------------------------------------------------------------------
    def apsw_connect_to_library(self):

        my_db = self.gui.library_view.model().db

        path = my_db.library_path
        if isbytestring(path):
            path = path.decode(filesystem_encoding)
        path = path.replace(os.sep, '/')
        path = os.path.join(path, 'metadata.db')
        path = path.replace(os.sep, '/')

        if isbytestring(path):
            path = path.decode(filesystem_encoding)

        if path.endswith("/"):
            path = path[0:-1]

        try:
            my_db =apsw.Connection(path)
            is_valid = True
        except Exception as e:
            if DEBUG: print("path to metadata.db is: ", path)
            if DEBUG: print("error: ", str(e))
            is_valid = False
            return None,None,is_valid

        my_cursor = my_db.cursor()

        mysql = "PRAGMA main.busy_timeout = 15000;"      #PRAGMA busy_timeout = milliseconds;
        my_cursor.execute(mysql)

        return my_db,my_cursor,is_valid

P.S.
Forgot to mention earlier that I had to repackage the zip in order to load the plugin (got rid of the top level KyBook3 Sync directory). Calibre was complaining about __init__.py not being in the correct location. I'd be happy to help debug but please note I use Calibre Portable and I dunno how to run calibre-debug with that since the paths are all different.

[ilovejedd]

Thanks! Tested it and it works great if an entry only has one format. Seems like it only sends thumbnails, though.

Plugin hangs at Metadata Sync when there are multiple formats after the ebooks have been sent to the device. I'll PM you debug logs tomorrow.

[karl1c]

I've added the syncing of ratings and some identifiers (notably isbn, oclc, and asin). If your identifiers turn up in KyBook3 as 'id' or not at all, let me know and I'll amend the code to allow for them.

[theengine]

Hi,
Thanks very much for this plugin. I've been testing it since the first version.
I've noticed one issue.
I currently use the Calibre content server to transfer a book to Kybook3. When I do this it appears in the download section of the Kybook3 library. When I shut down my iPhone and restart it Kybook3 remembers my place in the book.
However, when I use the plugin to transfer the book to my iPhone, it is just in the Library and when I shutdown and restart my iPhone Kybook3 has forgotten my place in the book.
Is it possible to fix this issue?

Thanks