[Metadata Writer Plugin] KePub Metadata Writer

[davidfor]

The code in library/server is the current content server code. The code in srv is the rewrite that Kovid is doing. Unfortunately, he seems to have dropped the changes I made for kepubs. I'll have a look at putting it back in and see if he is OK with it.

The list of formats to update the metadata is a bit harder. The problem is these need the metadata writer plugin for the format. But, as this is not included in calibre for kepubs, there would need to be extra checking to make an update wasn't attempted when the writer wasn't there. From a Kobo users point of view, adding the metadata writer and reader plugins to calibre would be best. But, I understand Kovid's desire not to include single site formats in the base calibre.

[jgoguen]

If there were some way for plugins to hook into the content server and say "support my format too" it would be enough. I think right now I just need to add 'kepub' to a couple lists at initialization time?

[davidfor]

I wouldn't have thought that was needed. Calibre knows what metadata writer plugins it has and what formats they support. If that is built for the server at startup, it should be able to handle formats correctly. But, maybe the server is not loading all the plugins.

In any case, I'm not sure what the state of the new server is. I can start it and reach it from a browser, but I get an empty file as a response. It's probably a configuration problem on my part, but I can't see it right now.

[darryl]

Here are the very simple changes I made. I do not plan to submit them myself. Obviously the changes to the file in srv are not necessary given this is Kovid's re-write. With the changes to the file in library/server there is no problem now with creating a content server metadata plugboard for the kepub format. When downloading kepub books from the content server they do download as kepub epub and the metadata plugboard is applied, at least so far as my setup to modify the title is concerned. It does not, however, actually create a collection or add the book to the collection. Am I correct in assuming that this is a function purely of the driver? It does work over USB using the Kobo Touch driver, but not with the content server.

Given that adding support for new formats seems to be relatively trivial, is it actually worthwhile to support all formats with metadata writers? Support can easily be added on request of users or developers who use the particular format. Support for epub, mobi, azw3 and now kepub seems to cover all of the most common formats, with the exception, I suppose, of pdf, which is not really a great format for ereaders anyway.

Spoiler:

Subject: [PATCH] Content Server support for kepub format metadata plugboards

---
src/calibre/library/server/content.py | 4 ++--
src/calibre/srv/content.py | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/calibre/library/server/content.py b/src/calibre/library/server/content.py
index ef7a7a7..5295cdf 100644
--- a/src/calibre/library/server/content.py
+++ b/src/calibre/library/server/content.py
@@ -21,7 +21,7 @@ from calibre.ebooks.metadata.opf2 import metadata_to_opf
from calibre.utils.config import tweaks

plugboard_content_server_value = 'content_server'
-plugboard_content_server_formats = ['epub', 'mobi', 'azw3']
+plugboard_content_server_formats = ['epub', 'mobi', 'azw3', 'kepub']

class CSSortKeyGenerator(SortKeyGenerator):

@@ -216,7 +216,7 @@ class ContentServer(object):
fm = self.db.format_metadata(id, format, allow_cache=False)
if not fm:
raise cherrypy.HTTPError(404, 'book: %d does not have format: %s'%(id, format))
- update_metadata = format in {'MOBI', 'EPUB', 'AZW3'}
+ update_metadata = format in {'MOBI', 'EPUB', 'AZW3', 'KEPUB'}
mi = newmi = self.db.get_metadata(
id, index_is_id=True, cover_as_data=True, get_cover=update_metadata)

diff --git a/src/calibre/srv/content.py b/src/calibre/srv/content.py
index 4f3b13d..e2e7fa9 100644
--- a/src/calibre/srv/content.py
+++ b/src/calibre/srv/content.py
@@ -28,7 +28,7 @@ from calibre.utils.magick.draw import thumbnail, Image
from calibre.utils.shared_file import share_open

plugboard_content_server_value = 'content_server'
-plugboard_content_server_formats = ['epub', 'mobi', 'azw3']
+plugboard_content_server_formats = ['epub', 'mobi', 'azw3', 'kepub']
update_metadata_in_fmts = frozenset(plugboard_content_server_formats)
lock = Lock()

--
2.4.4

[davidfor]

Originally Posted by darryl

Here are the very simple changes I made. I do not plan to submit them myself. Obviously the changes to the file in srv are not necessary given this is Kovid's re-write.

Are you sure of that? I can see equivalent code in the new server code that is limited to updating just epub, mobi and azw3.

With the changes to the file in library/server there is no problem now with creating a content server metadata plugboard for the kepub format. When downloading kepub books from the content server they do download as kepub epub and the metadata plugboard is applied, at least so far as my setup to modify the title is concerned. It does not, however, actually create a collection or add the book to the collection. Am I correct in assuming that this is a function purely of the driver? It does work over USB using the Kobo Touch driver, but not with the content server.

Yes, the collections are created by the driver. It is doing this by directly updating the database on the device. The content server can't do this as the browser on the device is requesting a file, saving it and then triggering the book processing for it. There is no way for the content server to tell the device anything about collections.

Given that adding support for new formats seems to be relatively trivial, is it actually worthwhile to support all formats with metadata writers? Support can easily be added on request of users or developers who use the particular format. Support for epub, mobi, azw3 and now kepub seems to cover all of the most common formats, with the exception, I suppose, of pdf, which is not really a great format for ereaders anyway.

Yes, but it means that someone has to make the request. And if someone adds a new format and creates the metadata writer plugin, they have to get the server code updated to handle it. If it is a brand new format that sweeps the world in a couple of days, then updating the server is OK. But, if it is a specialized format that very few people use, it wouldn't make sense.

[darryl]

Thanks for your comments @davidfor. I was fairly sure I was correct about the driver handling the collections, for pretty much the reasons you confirmed. However, I am neither competent or confident in python. I can usually roughly follow the logic of the program but that is about it. After your post mentioning Kovid's re-write I did suspect that it may be in the code under srv, but without a better understanding than I have could not be sure.

And the changes to the file under srv? It will need to be changed eventually to reflect not only the changes I suggested but to add back your own code, at least if kepub is to be supported in the new server. My thinking was that it does not need to be changed now given that the new server still seems to be in the early stages of a rewrite. Given this, I was wondering whether Kovid would welcome changes to it at this stage. Also, it does not seem to incorporate your kepub.epub code at the moment so adding kepub to the formats in one line would probably not be all that useful.

In principle, it would be nice to support all formats with a metadata writer.

[jgoguen]

I would prefer to keep KePub out of the main calibre code, only because it's a single-site format that not even all users of the single site even like. It's also subject to change at Kobo's whims. Rather, I would prefer to ask Kovid to update (or accept a patch for) https://github.com/kovidgoyal/calibr...ontent.py#L219 to allow a plugin to register itself as providing the necessary format support.

[kovidgoyal]

The new server is nowhere near ready just now. As for allowing plugins to register themselves, I dont want to do that, because not all plugins are suitable for use in a server context. Some of them can be quite slow/leak memory/crash(pdf) which is why the list of formats is artificially restricted, as opposed to being the list of formats supported by all metadata writer plugins.

There is also the question of plugboards, IIRC the list of formats that can be used with plugboards in the server is also hardcoded.

I'm not sure what the best way to proceed here is. I suppose I could add kepub to the hardcoded list and have the code disard it if not kepub writer plugin is available.

[jackie_w]

Originally Posted by jgoguen

I would prefer to keep KePub out of the main calibre code, only because it's a single-site format that not even all users of the single site even like. It's also subject to change at Kobo's whims.

Just for the sake of philosophical discussion that description could equally well describe Amazon's azw3/kf8, couldn't it?

[jgoguen]

Well, yes Amazon has changed the extension when they've changed the format (.azw, .awz3). Kobo hasn't changed their format yet, so we don't know what they would do. But it would be trivial to say "open the ZIP file, look for this file with these specific contents; if present use the new format parser" and keep the same .kepub.epub double extension. Either way, we could play that cat-and-mouse game. Now that you make me think about it a bit, it feels like a code maturity thing. Kovid made the decision to incorporate AZW and AZW3, but it's my impression that the code around that is much more mature and stable than the KePub code is.

Who knows, perhaps someday if I can produce KePub files nearly as well as Kobo can perhaps Kovid will want to incorporate it all into calibre core despite it being site-specific and not near-universally preferred. If he does I'll happily make the patches to bring everything in, but I'd think that would also need a bunch of work with davidfor to merge the code in cleanly; if KePub is coming into calibre core it shouldn't be as yet another driver. A huge chunk of my problem is that I can't justify purchasing a book from Kobo (they're so expensive, and nothing I actually want is eligible for coupons) but I need a recently released book from them to make sure I'm generating KePub files that work like their current releases.

How about this: if someone has a recently released (say, released during the month of June) well-formatted English-language KePub file they've stripped the DRM from and they're willing to email me privately, I'll put in another effort to get the plugin code generating better KePub files. A lot of the recent updates have been made based on books that are a few years old by now, and I think Kobo has made some internal improvements to how they generate KePub files since then. If Kovid wants to bring KePub into core calibre and David doesn't object to me merging the drivers, I'll also dedicate a good chunk of my copious spare time to the patch to integrate it all to calibre.

[PeterT]

I'll examine my wife's collection of romance books and see if any of them are recent publications...

[jgoguen]

Trying to motivate me to finish quickly I see

[PeterT]

Not really; just I know she buys a LOT from Kobo, and she is away right now, so access to her PC is very easy! I'll go spelunking tomorrow when I'm more with it.

[davidfor]

There should be some free books that fit the bill. Finding them might be the problem. Looking through my books, I can find a few that might have been added to Kobo this year. The most recent I can find quickly is https://store.kobobooks.com/en-US/ebook/backtrek. The files in the kepub are dated 30/3/2015. I will have another look when I have more time.

I did notice one little problem with the conversion recently. The added spans surround the sentence, but the space between the sentences is between the spans. It looks like:

end of sentence.</span> <span ..>Start of next sentence.

I think the only problem would be if someone managed to start or end a selecting on that space.

[davidfor]

Originally Posted by kovidgoyal

The new server is nowhere near ready just now.

OK, I've been wondering what the status was. But, unless it is going to be a few months, I don't want to do anything in the current server.

As for allowing plugins to register themselves, I dont want to do that, because not all plugins are suitable for use in a server context. Some of them can be quite slow/leak memory/crash(pdf) which is why the list of formats is artificially restricted, as opposed to being the list of formats supported by all metadata writer plugins.

There is also the question of plugboards, IIRC the list of formats that can be used with plugboards in the server is also hardcoded.

I'm not sure what the best way to proceed here is. I suppose I could add kepub to the hardcoded list and have the code disard it if not kepub writer plugin is available.

That would be safe. The kepub metadata writer just subclasses EPUBMetadataWriter to add the extension and then adds a property to the cover image in the OPF.

Hang on, I hadn't looked at the code in the version in this thread. It doesn't do the cover settings, so it is just adding the extension. jgoguen, is there are reason you dropped setting the cover property?