[Conversion Plugin] KFX Input

[DNSB]

Originally Posted by Ma'am-I-Am

I'll definitely take a look at that.
I sometimes download scanned-to-PDF books at archive.org. They are usually huge and take a relatively long time to load. Is there a fix for those things, i.e reduce file size and load quicker?

Not really any way. All the archive.org .PDF files I've looked at have been scanned images of the pages plus a very poorly OCRed text layer to allow searching.

The images make the files large but their quality is not the greatest so when I tried running various OCR packages on the images, the results needed way too much work to be useful.

[jhowell]

Version 2.0.0 - 01 Jun 2023

Allow conversion of Kindle Scribe notebooks. See the plugin documentation for instructions. (This is a work in progress. Please report any incorrect rendering or errors encountered in the KFX Input plugin thread.)

Handle the PDF backed fixed layout variant of KFX format.

Improve handling of margins expressed as a percentage in cases where the width of the element is unknown.

Remove support for calibre versions prior to 5.0. (Older versions of this plugin that run under calibre 2.x through 4.x are available in this post.)

[tomsem]

Looks good!

I tried one of my notebooks and Sticky Notes for one of my books.

- Apple Books expects a cover image and result is it truncates the first page of the notebook.
- calibre ebook-viewer (and ebook-edit) are applying a theme (dark white etc) and they probably should not be. So if they are set to dark theme you cannot see the black strokes. I think these just don't handle FXL rendering correctly?

Only suggestion so far is to fill in some placeholder metadata for title/author (e.g. "{filename} - {creation date}" , "KFX input plugin v{version}", and maybe include a simple cover for it with these.

Would make it easier, for example, to bring into calibre if someone wants to manage them there.

[jhowell]

Originally Posted by tomsem

Apple Books expects a cover image and result is it truncates the first page of the notebook.

I will add an option to the plugin to create a dummy cover page for notebooks. (Without that the calibre commands to replace an existing cover will not work properly.)

Originally Posted by tomsem

calibre ebook-viewer (and ebook-edit) are applying a theme (dark white etc) and they probably should not be. So if they are set to dark theme you cannot see the black strokes. I think these just don't handle FXL rendering correctly?

I will add a white background to notebook pages. That should fix this problem.

Calibre does not handle fixed layout EPUBs well, however the EPUBs created from notebooks are simple enough that they should work in calibre.

Originally Posted by tomsem

Only suggestion so far is to fill in some placeholder metadata for title/author (e.g. "{filename} - {creation date}" , "KFX input plugin v{version}", and maybe include a simple cover for it with these.

Unfortunately the file name is not readily available at the point where the title and author are set. For now I will change it to use the title "Notebook" followed by the 23-character internal book ID.

You can always change the title, author, and cover of the EPUB to whatever you want using the calibre CLI:

ebook-meta notebookname.epub --authors="New Author" --title="New Title" --cover=newcover.jpg

[tomsem]

Originally Posted by jhowell

I will add an option to the plugin to create a dummy cover page for notebooks. (Without that the calibre commands to replace an existing cover will not work properly.)



I will add a white background to notebook pages. That should fix this problem.

Calibre does not handle fixed layout EPUBs well, however the EPUBs created from notebooks are simple enough that they should work in calibre.



Unfortunately the file name is not readily available at the point where the title and author are set. For now I will change it to use the title "Notebook" followed by the 23-character internal book ID.

You can always change the title, author, and cover of the EPUB to whatever you want using the calibre CLI:

ebook-meta notebookname.epub --authors="New Author" --title="New Title" --cover=newcover.jpg

Great, thank you!

The placeholder title should work, but would be inclined to add timestamp as well so if imported to calibre there won't be duplicate warnings. (e.g. someone wants to store versions of a notebook in calibre)

Time allowing I'm planning to work on a script to help manage the steps, and orchestrate KFX Input and calibre CLI calls. Gathering requirements, use cases.

Eventually perhaps this becomes some Scribe Notebook manager plugin, if there's sufficient interest.

[jhowell]

Originally Posted by tomsem

The placeholder title should work, but would be inclined to add timestamp as well so if imported to calibre there won't be duplicate warnings. (e.g. someone wants to store versions of a notebook in calibre).

I will add the date and time to the placeholder title.

[George96]

Dear all!
I have an issue using the KFX-Input Plugin. "Issue" means it simply doesn't work.

I think I followed the (brilliant and detailled - thx for that!) instructions in this threat, but it simply doesn't work. :-( It always ends in a conversion error....

Where do I enter the serial-number of the device? I didn't find that field to enter in Calibre...

Once I added that, it might be working. Let's see...

Thanks for your help in advance!
Regards George.

[jhowell]

Originally Posted by George96

I have an issue using the KFX-Input Plugin. "Issue" means it simply doesn't work.

I think I followed the (brilliant and detailled - thx for that!) instructions in this threat, but it simply doesn't work. :-( It always ends in a conversion error....

What specific error message are you receiving from the plugin?

Originally Posted by George96

Where do I enter the serial-number of the device? I didn't find that field to enter in Calibre...

Once I added that, it might be working. Let's see...

The KFX Input plugin can only perform conversion once DRM removal has been accomplished. The serial number is need by the DeDRM plugin in order to remove DRM from books that are copied from a Kindle device. Detailed assistance with DRM removal is not allowed here on MobileRead. There is a FAQ and you can ask for help with that on GitHub.

You should also read the "DRM Concerns" section in the first post of this thread. If your Kindle is running a firmware version greater than 5.12.1 (which is true of most Kindles) then the DeDRM plugin will not be able to remove KFX DRM. If that is the case for you then see this thread for alternative methods of obtaining Kindle books in a format compatible with DeDRM.

[jhowell]

Version 2.1.0 - 18 Jun 2023

Miscellaneous improvements in the conversion of Scribe notebooks.

Handle Scribe notebooks containing multiple page templates. (Only one will be active.)

Add a white background to Scribe notebook pages. This should allow them to render better in dark mode.

Change the title of EPUBs created from Scribe notebooks from "Unknown" to "Notebook" followed by the internal book ID and current date. The calibre ebook-meta command can be used to change the title or author if something else is desired.

Add an option to the plugin command line interface to create a generic EPUB cover page showing the title of the book if the book does not already have a cover.

[tomsem]

Finally had a chance to try 2.1.0. Getting this error when trying to convert nbk (renamed to 'test.kpf)':

(base) ➜  test /Applications/calibre.app/Contents/MacOS/calibre-debug -r "KFX Input" -- test.kpf
Building icons.rcc
Software versions: KFX Input 2.1.0, calibre 6.22*, macOS-13.4.1-x86_64-i386-64bit
KFX Input plugin help is available at https://www.mobileread.com/forums/showthread.php?t=291290

Processing test.kpf
Processing container: test.kpf
Traceback (most recent call last):
  File "runpy.py", line 196, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "site.py", line 41, in <module>
  File "site.py", line 37, in main
  File "calibre/debug.py", line 282, in main
  File "calibre_plugins.kfx_input.__init__", line 194, in cli_main
  File "calibre_plugins.kfx_input.kfxlib.yj_book", line 211, in decode_book
  File "calibre_plugins.kfx_input.kfxlib.kpf_container", line 51, in deserialize
  File "calibre_plugins.kfx_input.kfxlib.utilities", line 586, in as_ZipFile
  File "zipfile.py", line 1258, in __init__
  File "zipfile.py", line 1325, in _RealGetContents
zipfile.BadZipFile: File is not a zip file
(base) ➜  test

No doubt I'm doing something wrong.

[jhowell]

Originally Posted by tomsem

Finally had a chance to try 2.1.0. Getting this error when trying to convert nbk (renamed to 'test.kpf)':

No doubt I'm doing something wrong.

You need to pass the name of the notebook folder to the KFX Input plugin. See the section "Converting Scribe notebooks" for details.

(KPF is a variant of KFX format in a ZIP file. That is different from the format used by Scribe notebooks which is why it fails.)

[tomsem]

Originally Posted by jhowell

You need to pass the name of the notebook folder to the KFX Input plugin. See the section "Converting Scribe notebooks" for details.

(KPF is a variant of KFX format in a ZIP file. That is different from the format used by Scribe notebooks which is why it fails.)

Perfect, thank you!

FYI I am seeing issues with a number of notebooks: some show warnings/errors and generate .ePub, others error out before generating one. Will supply details after I collect examples.

[tomsem]

Failures:

• Key Error
• Insufficient data

Success, with errors/warnings ('cosmetic'):

• Book has short symbol format without document_data max_id (virtually every !!EBOK!! & !!PDOC!! nbk) - I think most of these are 'empty' e.g. no pen annotations
• Missing sections
• repeats previous location

Unfortunately I no longer have the PDOC corresponding to the missing_sections nbk (it was orphaned somehow when I deleted the KFX-PR document). I'll see if I can find another example.

[jhowell]

Originally Posted by tomsem

Failures: ...

Thanks for the error reports! I will take a look at them today and will let you know if I need any additional information to help solve them.

I apologize for relying on you so much to do testing and provide samples. I have been considering picking up a Scribe for reading and experimentation but have been reluctant because of the cost and the fact that I have no interest in using it for note taking.

The low end Essentials Bundle is $285 right now for Prime members. That is tempting me.

[jhowell]

Originally Posted by tomsem

Key Error

That relates to a feature (local_delta_fragments) used in Scribe notebooks. I made some assumptions about how to handle that based on the samples I had seen, but your new example includes a case that I didn't account for. This will be fixed in the next plugin release, at least for all of the cases I have encountered so far.

Originally Posted by tomsem

Insufficient data

That notebook uses a brush type (value 0 internally) that I have not encountered before. Strokes for that are missing some of the data that is present for other brush types causing the plugin to fail.

I have seen "brush pen" mentioned as a possible brush type in the Scribe software. Is that what you used to draw in this notebook?

Originally Posted by tomsem

Book has short symbol format without document_data max_id (virtually every !!EBOK!! & !!PDOC!! nbk) - I think most of these are 'empty' e.g. no pen annotations

Those errors result from the notebook being empty. That will be fixed in the next plugin release.

Originally Posted by tomsem

Missing sections

Unfortunately I no longer have the PDOC corresponding to the missing_sections nbk (it was orphaned somehow when I deleted the KFX-PR document). I'll see if I can find another example.

Your "repeats previous location" sample also has this error so I do not need a separate example of it.

It is possible that error results from deleted sticky note annotations not being completely removed from the notebook. One way to test this would be to add a sticky note to a book that has not been previously annotated and then delete the note. I suspect that the resulting annotation notebook will exhibit this problem.

If so then I will change this from an error to a more specific warning in the plugin. If that is not the actual cause then more research will be needed.

Originally Posted by tomsem

repeats previous location

This appears to be a relatively new behavior, since sequential points of a stroke never had the same location in any of the Scribe notebook I examined previously. I will handle this case in the next release of the plugin.