Kepub format questions

[davidfor]

Originally Posted by geek1011

Kepubify aims to be fast, correct (as in being as close as possible to normal kepubs),

The latter was also the intention of the KoboTouchExtended driver. If there is something it doesn't do, people should report it.

And comparing speed between to two is also a problem. During the send-to-device, calibre is making sure there are making sure the metadata and the covers are up to date. Plus there is the metadata updating on the device.

scriptable, simple, easy-to-use, non-bloated,

The latter is a bit rude to say as the driver isn't bloated. Whether calibre itself is bloated depends on your viewpoint. It's big, but it does a lot. Calibre is also scriptable.

and provides fixes for many common kepub bugs. It also doesn't need to manage your whole library to work properly.

KTE is slower, but better integrated with Calibre (though kepubify will be getting plugin sometime this year). In addition, it supports the whole range of conversion options Calibre has. Some people also find it easier to use, as everything can be done from a GUI.

You can also use a mix of the two if you prefer. You can use kepubify to convert your books to kepub, but KTE to manage syncing them with your kobo.

If you are converting books to kepubs and keeping them in the calibre library, you don't need the extended driver. The base KoboTouch driver handles sending kepubs. And the extended driver doesn't add much when doing this. But you should add the kepub metadata reader and writer plugins.

Also, kepubify comes with a standalone seriesmeta tool to update series metadata on all your books even if you don't use calibre to manage them.

I'm amused by the fact that this post is effectively telling people they don't need calibre, but, the seriesmeta tool relies on calibre specific metadata for the series info. But, it won't work ePub3 books. ePub3 has a definition for series info in the metadata and calibre uses that.

@forster01: I think it comes down to how you are managing your ebooks. If you use calibre, then use it to do the send-to-device. Whether that is via the KoboTouchExtended driver or a conversion to kepub, it doesn't matter much. If you are using other tools to manage your books, then kepubify makes more sense.

[forster01]

for a 7 years i've been reading epubs only. now i'm reading my first kepubifyed kepub and i'm loving it... great job geek1011!!!

[geek1011]

Originally Posted by davidfor

The latter was also the intention of the KoboTouchExtended driver. If there is something it doesn't do, people should report it.

It isn't so much what it doesn't do, but what it does do. It's more Calibre's fault than anything though (the extra stuff added). Also, the span logic in kepubify is slightly different, and seems to match up better.

And comparing speed between to two is also a problem. During the send-to-device, calibre is making sure there are making sure the metadata and the covers are up to date. Plus there is the metadata updating on the device.

I'm comparing just the conversion time between a plain epub and a kepub with the most minimal settings I can do. I've discussed this quite a few times before.

The latter is a bit rude to say as the driver isn't bloated. Whether calibre itself is bloated depends on your viewpoint. It's big, but it does a lot.

Yes, I guess. My reasoning was that it does a lot more than needed for converting epub to kepub. One of my biggest pet peeves is the HTML and CSS mods calibre does, along with the additional calibre contributor[role=bkp] and calibre_user_meta_map (or something like that) stuff it adds to the OPF.

Calibre is also scriptable.

Last time I checked, ebook-convert didn't work right from the command line with kepubs. Also, it doesn't play well when using it with many other small tools (it cannot use an arbitrary location as the kobo path, it cannot take css from the output of another tool).

If you are converting books to kepubs and keeping them in the calibre library, you don't need the extended driver. The base KoboTouch driver handles sending kepubs. And the extended driver doesn't add much when doing this. But you should add the kepub metadata reader and writer plugins.

I'm aware of that; I've read the source code a few times before. I'm just simplifying it for the benefit of the average reader.

I'm amused by the fact that this post is effectively telling people they don't need calibre, but, the seriesmeta tool relies on calibre specific metadata for the series info. But, it won't work ePub3 books. ePub3 has a definition for series info in the metadata and calibre uses that.

I've never once seen the epub3 series metadata in the wild (but I've seen calibre series metadata, as it's almost a standard). Also, although the seriesmeta tool relies on calibre metadata, it doesn't require calibre itself. Note that I don't have anything personal against calibre, but I dislike some of it's design decisions (but I understand that some are necessary in a program that large), and it isn't a good fit for my needs.

@forster01: I think it comes down to how you are managing your ebooks. If you use calibre, then use it to do the send-to-device. Whether that is via the KoboTouchExtended driver or a conversion to kepub, it doesn't matter much. If you are using other tools to manage your books, then kepubify makes more sense.

That sums it up quite nicely. That is more or what less what I meant.

Also, as I said, you can also use a combination of both (see https://www.mobileread.com/forums/sh...d.php?t=313137).

[davidfor]

Originally Posted by geek1011

It isn't so much what it doesn't do, but what it does do. It's more Calibre's fault than anything though (the extra stuff added). Also, the span logic in kepubify is slightly different, and seems to match up better.

Yes, I guess. My reasoning was that it does a lot more than needed for converting epub to kepub. One of my biggest pet peeves is the HTML and CSS mods calibre does, along with the additional calibre contributor[role=bkp] and calibre_user_meta_map (or something like that) stuff it adds to the OPF.

Most of that sounds more like a conversion than the driver. The driver should only touch the CSS except to add things that are in kepub and the hyphenation options. The changes to the HTML should only be the spans, and a wrapping div or two. The formatting of the code might change a little, but I don't think it runs a pretty print or anything over it.

Adding the custom column metadata is done anything that updates the metadata in the book. It's not pretty if you open the OPF, but it is well within the standards.

Last time I checked, ebook-convert didn't work right from the command line with kepubs. Also, it doesn't play well when using it with many other small tools (it cannot use an arbitrary location as the kobo path, it cannot take css from the output of another tool).

I'll admit to not doing much scripting using calibre, so I haven't hit that. But, I'll have a look at the conversion to see what's up. But, this doesn't use the driver, it is using the input and output plugins.

The file path for the input and output can be specified, so I'm not sure what you mean there. And an extra CSS file can be supplied as a parameter.

But, a conversion is not what you want to do if you want complete control over the result.

I'm aware of that; I've read the source code a few times before. I'm just simplifying it for the benefit of the average reader.

The hassle with that is there is confusion over the necessity of the KoboTouchExtended driver. Some people seem to think it is necessary, install it and then complain about what it is doing.

I've never once seen the epub3 series metadata in the wild (but I've seen calibre series metadata, as it's almost a standard).

I haven't looked for the epub3 series metadata, but I'll have to look at the ones I have from Kobo and see if they are using it. And yes, the calibre series metadata is the closest thing to a standard for epub2. I tried to convince Kobo to read it some time ago, but they weren't interested. With the epub3 support, it might be time to try again.

[geek1011]

Originally Posted by davidfor

Most of that sounds more like a conversion than the driver. The driver should only touch the CSS except to add things that are in kepub and the hyphenation options. The changes to the HTML should only be the spans, and a wrapping div or two. The formatting of the code might change a little, but I don't think it runs a pretty print or anything over it.

It doesn't. But Calibre conversion and the Kepub output go together. You cannot use the Kepub output plugin without Calibre, so you cannot avoid the extra Calibre cruft.

Adding the custom column metadata is done anything that updates the metadata in the book. It's not pretty if you open the OPF, but it is well within the standards.

Well, the meta tag is deprecated in Epub 3.0.1 and removed in 3.1. Also, I manually go and clean every book I own and like to keep it that way.

I'll admit to not doing much scripting using calibre, so I haven't hit that. But, I'll have a look at the conversion to see what's up. But, this doesn't use the driver, it is using the input and output plugins.

Yes, I know the difference between the driver and the i/o plugins, as I have said before. IIRC, it was an issue with the GUI imports.

The file path for the input and output can be specified, so I'm not sure what you mean there. And an extra CSS file can be supplied as a parameter.

It can't do batch conversion while preserving file structure and only modifying changed files.

But, a conversion is not what you want to do if you want complete control over the result.

Which is why Calibre doesn't fit my needs.

The hassle with that is there is confusion over the necessity of the KoboTouchExtended driver. Some people seem to think it is necessary, install it and then complain about what it is doing.

Yes, I agree with that, but I know the difference myself if that's that you're implying.

I haven't looked for the epub3 series metadata, but I'll have to look at the ones I have from Kobo and see if they are using it. And yes, the calibre series metadata is the closest thing to a standard for epub2. I tried to convince Kobo to read it some time ago, but they weren't interested. With the epub3 support, it might be time to try again.

None of my Kobo books use it. As for the epub3 metadata, I don't think Kobo will do it (it's even less likely), as the current metadata extraction is not complex enough to handle the epub3 style without some changes. I think I might be able to patch it to read Calibre metadata though, if I can find some unused space I can fit the code in along with another variable.

[davidfor]

Originally Posted by geek1011

It doesn't. But Calibre conversion and the Kepub output go together. You cannot use the Kepub output plugin without Calibre, so you cannot avoid the extra Calibre cruft.

Sorry, your comment was following on from discussion of the driver. I assumed you meant the changes came from that.

Well, the meta tag is deprecated in Epub 3.0.1 and removed in 3.1.

Looks like it is replaced with opf:meta. Or at least that is what is in the epub3 I just created.

Also, I manually go and clean every book I own and like to keep it that way.

I clean the books and make sure the metadata is correct. But, I don't care about the layout of the OPF or NCX. About the only thing I do in the OPF is to set the language to "en-GB" so the spelling checker shows less errors if the book appears to use UK English. And that's basically if there is more "colour" than "color".

It can't do batch conversion while preserving file structure and only modifying changed files.

Ok, that's a difference in viewpoint. To me, deciding which file and where tends to be the responsibility of the controlling script.

Yes, I agree with that, but I know the difference myself if that's that you're implying.

It was for other readers. I'm a bit tired of the, "But I thought I had to install it" response after someone complains about books turning up as kepubs.

None of my Kobo books use it. As for the epub3 metadata, I don't think Kobo will do it (it's even less likely), as the current metadata extraction is not complex enough to handle the epub3 style without some changes. I think I might be able to patch it to read Calibre metadata though, if I can find some unused space I can fit the code in along with another variable.

I'm sure the firmware doesn't read the epub3 metadata at the moment. But, if I can find it in a book I got from Kobo, I can justify a bit of a push to read it. And hopefully fix some of the other things at the same time.

[geek1011]

Originally Posted by davidfor

Sorry, your comment was following on from discussion of the driver. I assumed you meant the changes came from that.

Ok.

Looks like it is replaced with opf:meta. Or at least that is what is in the epub3 I just created.

Yes, I think that's what it is now, but I haven't seen it either. I think it needs more visibility and clarity in the spec, as you're the only other person I've talked to who even recognized it.

I clean the books and make sure the metadata is correct. But, I don't care about the layout of the OPF or NCX. About the only thing I do in the OPF is to set the language to "en-GB" so the spelling checker shows less errors if the book appears to use UK English. And that's basically if there is more "colour" than "color".

I clean everything: the html (who in their right mind uses divs for paragraphs, unclosed style tags, self-closing title tags, and a nbsp instead of normal spaces), CSS (you don't need to put ! important on everything and assign ids each overriding the font to each sentence), old, ncx (the navigation docs the publishers create are atrocious! out of ordered, duplicated indexes!). All epubs I own are consistent and pass all the validators.

Ok, that's a difference in viewpoint. To me, deciding which file and where tends to be the responsibility of the controlling script.

Yes. To me, the right balance is having tools support simple recursive conversion of unconverted files, and doing the rest in the script.

It was for other readers. I'm a bit tired of the, "But I thought I had to install it" response after someone complains about books turning up as kepubs.

That annoys me too . And the question: "how do I install kepubify"! Everyone (well, everyone who isn't reading this) on Windows is so used to installers and wizards for everything, they don't appreciate portable apps! It drives me nuts.

I'm sure the firmware doesn't read the epub3 metadata at the moment. But, if I can find it in a book I got from Kobo, I can justify a bit of a push to read it. And hopefully fix some of the other things at the same time.

Yes. I'll see if I can find it too, and if I do, I'll also add support to seriesmeta as well.

EDIT: P.S. And regarding the conversion speeds: when I did some testing a year ago, an average ~300pp novel takes kepubify .4s to convert vs 13s for the Calibre plugin. Large manuals like The C++ Programming Language by Bjarne Stroustrup takes kepubify 6s to convert vs 14-20 min for the Calibre plugin (it gets slow at either the css stuff or the span stuff, and it stalls 10% of the time, that might be a bug). Also, kepubify handles malformed epubs (broken opf, invalid html) better than Calibre most of the time.

[danielr]

https://pgaskin.net/kepubify/
Convert Epub to kepub with a touch.

[geek1011]

Originally Posted by danielr

https://pgaskin.net/kepubify/
Convert Epub to kepub with a touch.

That's me! Did you read the previous posts?

[jackie_w]

@geek1011,

It's probably not my place to be saying this, but I'm going to anyway ...

I agree with davidfor's quote below:

Originally Posted by davidfor

@forster01: I think it comes down to how you are managing your ebooks. If you use calibre, then use it to do the send-to-device. Whether that is via the KoboTouchExtended driver or a conversion to kepub, it doesn't matter much. If you are using other tools to manage your books, then kepubify makes more sense.

I've been thinking about this for a while. Kepubify is a great option for those who prefer to avoid calibre but I really wonder whether it's a good idea to add kepubify as a calibre plugin. Isn't it likely to add unnecessary confusion to which Kobo/kepub/calibre features happen in kepubify and which happen in the KoboTouch/KoboTouchExtended drivers, or even some things being done twice. Isn't it likely to result in extra support work for both sides?

[geek1011]

Originally Posted by jackie_w

I've been thinking about this for a while. Kepubify is a great option for those who prefer to avoid calibre but I really wonder whether it's a good idea to add kepubify as a calibre plugin. Isn't it likely to add unnecessary confusion to which Kobo/kepub/calibre features happen in kepubify and which happen in the KoboTouch/KoboTouchExtended drivers, or even some things being done twice. Isn't it likely to result in extra support work for both sides?

I have been wondering this as well. I'm on the fence on if I'm making it a UI plugin or a conversion plugin, but I'm thinking of making it a UI plugin (a button) for the following reasons:
1. No possibility of confusion. One is a kepubify button, the other is Calibre's conversion interface.
2. Easier for me to implement. No need to deal with the other Calibre conversion steps.
3. No conflicts with the existing plugin.
4. No automated conversion to cause confusion.
5. Easier to understand which the user is using.
6. Kepubify would retain control over it's conversion process.
7. No users trying to use kepubify through ebook-convert instead of the actual command line interface.

If you have any objections or suggestions/criticism with my reasoning, I'd be happy to hear it. I'm not making the plugin for myself (I don't use Calibre); I'm making it due to the many requests I have recieved.

Note that even if I don't make it a plugin, I can still add integration by allowing a user to drag a Calibre library over kepubify (which I'm working on right now, see the kepubify v3 issue on GitHub). Kepubify would detect the library and convert with a .kepub extension instead and will also convert to the correct folder.

[PeterT]

Originally Posted by geek1011

Note that even if I don't make it a plugin, I can still add integration by allowing a user to drag a Calibre library over kepubify (which I'm working on right now, see the kepubify v3 issue on GitHub). Kepubify would detect the library and convert with a .kepub extension instead and will also convert to the correct folder.

So you'd end up adding files into the calibre library that would not be represented in the calibre database, and would not be usable by calibre in any manner; users could not select the kepub files and transfer them thru calibre to their device.

Sent from my Nexus 7 using Tapatalk

[geek1011]

Originally Posted by PeterT

So you'd end up adding files into the calibre library that would not be represented in the calibre database, and would not be usable by calibre in any manner; users could not select the kepub files and transfer them thru calibre to their device.

Sent from my Nexus 7 using Tapatalk

No, I'd be adding them to the db.

[jackie_w]

Originally Posted by geek1011

If you have any objections or suggestions/criticism with my reasoning, I'd be happy to hear it. I'm not making the plugin for myself (I don't use Calibre); I'm making it due to the many requests I have recieved.

First let me say that I have concerns not objections. I'm not the target market for a kepubify calibre plugin so all I have is opinions - which you're free to ignore. After all, I'm not the one who will will be fielding support questions in these forums

The following is all personal opinion. I fully admit to being somewhat of a pessimist by nature, but then again, as someone once said "a pessimist is what an optimist calls a realist"

My first question would be why are kepubify users asking for it to be included in calibre? What are the features which aren't already available in KoboTouch (KT) and KoboTouchExtended (KTE)? The answer to this question is fundamental to what ought to happen next.

I don't know all the features in kepubify. Do you replicate some/all the add-ons in KTE?, some of the ones in the base KT? If so, this would be my biggest area of concern - similar features - but done differently (possibly). If/when a user posts here to say that one of these duplicate features isn't working as they expect they are not going to know whether it's KT/KTE or kepubify plugin causing the problem or possibly a conflict between the two. The latter could quickly degenerate into conflict between the two sets of developers. There are no winners if that happens. For that reason, personally, I'd avoid all duplicate features, e.g. leave metadata handling to KT.

As an aside, if it was up to me I'd remove some of the existing KTE add-ons because I don't think they belong there. But it isn't up to me, so I just don't use those and leave others in peace to do as they wish. It's fairly common for users to want all their favourite features gathered together in one place. Many developers have difficulty playing 'Bad Cop' so they prefer to say 'yes' when deep-down they know they should say 'no' Feature-creep and -duplication ensues, to the benefit of no-one in the long run.

If your kepubify plugin was just going to be an epub-to-kepub (and possibly azw3-to-kepub) format-shifter - with the aim of adding a physical kepub to the calibre library without going through the full epub-to-kepub conversion process (as per Kepub Output plugin) then I wouldn't have any particular concerns about that. It's extra add-ons which duplicate the work of KT/KTE or other published plugins which concern me.

Originally Posted by geek1011

Note that even if I don't make it a plugin, I can still add integration by allowing a user to drag a Calibre library over kepubify (which I'm working on right now, see the kepubify v3 issue on GitHub). Kepubify would detect the library and convert with a .kepub extension instead and will also convert to the correct folder.

Again, I'm not the target market for this but as it's still basically a standalone feature I can't immediately see any particular cause for concern.

[geek1011]

Originally Posted by jackie_w

First let me say that I have concerns not objections. I'm not the target market for a kepubify calibre plugin so all I have is opinions - which you're free to ignore. After all, I'm not the one who will will be fielding support questions in these forums

Opinions are what creates discussion.

The following is all personal opinion. I fully admit to being somewhat of a pessimist by nature, but then again, as someone once said "a pessimist is what an optimist calls a realist"

I'm usually like that too.

My first question would be why are kepubify users asking for it to be included in calibre? What are the features which aren't already available in KoboTouch (KT) and KoboTouchExtended (KTE)? The answer to this question is fundamental to what ought to happen next.

Mainly speed, stability for large books, and some of the built-in CSS mods (such as the full screen fixes until recently, the CSS inliner to deal with the buggy FW and some buggy books, etc).

I don't know all the features in kepubify. Do you replicate some/all the add-ons in KTE?, some of the ones in the base KT? If so, this would be my biggest area of concern - similar features - but done differently (possibly). If/when a user posts here to say that one of these duplicate features isn't working as they expect they are not going to know whether it's KT/KTE or kepubify plugin causing the problem or possibly a conflict between the two. The latter could quickly degenerate into conflict between the two sets of developers. There are no winners if that happens. For that reason, personally, I'd avoid all duplicate features, e.g. leave metadata handling to KT.

The metadata handling all will be in seriesmeta (an add-on tool for kepubify), which is a standalone tool which will not be included in the Calibre plugin. It can be used along with Calibre without issues though.

As an aside, if it was up to me I'd remove some of the existing KTE add-ons because I don't think they belong there. But it isn't up to me, so I just don't use those and leave others in peace to do as they wish. It's fairly common for users to want all their favourite features gathered together in one place. Many developers have difficulty playing 'Bad Cop' so they prefer to say 'yes' when deep-down they know they should say 'no' Feature-creep and -duplication ensues, to the benefit of no-one in the long run.

I have a well-defined scope for kepubify. Kepubify (the converter part only) will only:
- Convert epubs to kepubs
- Apply fixes to book-breaking (i.e. book is not readable) CSS and HTML layout which are in multiple books by a publisher
- Remove extra MS/ADE/iBooks/Calibre cruft
- Apply CSS modifications to fix or tweak bugs (see above) or Kepub features (like hyphenation)
- Manage basic recursive converting to an arbitrary output directory (for ease of use)

Seriesmeta will do anything to do with metadata management on Kobo devices as long as it does not interfere with Calibre or the Kobo store.

If your kepubify plugin was just going to be an epub-to-kepub (and possibly azw3-to-kepub) format-shifter - with the aim of adding a physical kepub to the calibre library without going through the full epub-to-kepub conversion process (as per Kepub Output plugin) then I wouldn't have any particular concerns about that. It's extra add-ons which duplicate the work of KT/KTE or other published plugins which concern me.

You are correct in not having any concern.

Thanks for the feedback! This will probably clear up other people's questions as well.