Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 02-11-2013, 12:58 PM   #1
kc7zzv
Member
kc7zzv began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Jan 2009
Device: Handspring Palm
Looking for direction for making a kepub plugin

A kepub is a format developed by Kobo based off of epub. It's (almost) an epub with a few files added, the main "html" doc broken into smaller chunks, and extra tags added.

Are there any examples of "format" plugins (pdf,mobi,epub,cbz...) that are mostly based off of another plugin? I'm trying to figure out how I should be doing this, without going the simple route and stupidly copying and pasting a lot of code.

I'm reasonably experienced at programming, by my python experience is so lacking, I want to at least be imitating a good example.
kc7zzv is offline   Reply With Quote
Old 02-11-2013, 07:46 PM   #2
user_none
Sigil & calibre developer
user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.
 
user_none's Avatar
 
Posts: 2,488
Karma: 1063785
Join Date: Jan 2009
Location: Florida, USA
Device: Nook STR
Input or output support?

The base flow is: Input format -> intermediate (epub) format -> Output format. An input plugin takes the input format and converts it to html/CSS. The output format takes html/CSS and turns it into the output format.

TXT is a bit complex (lots of options) but it's a good one to look at for general flow. Mainly because it is a non-html format on both ends.

You'll probably end up subclassing a lot of the epub code since you really just need to translate the non-epub tags to something similar for input. You may or may not need to write an output plugin depending on how difficult it is to find patterns in an epub to turn into the kepub.
user_none is offline   Reply With Quote
Old 02-11-2013, 07:56 PM   #3
kc7zzv
Member
kc7zzv began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Jan 2009
Device: Handspring Palm
Quote:
Originally Posted by user_none View Post
Input or output support?

The base flow is: Input format -> intermediate (epub) format -> Output format. An input plugin takes the input format and converts it to html/CSS. The output format takes html/CSS and turns it into the output format.

TXT is a bit complex (lots of options) but it's a good one to look at for general flow. Mainly because it is a non-html format on both ends.

You'll probably end up subclassing a lot of the epub code since you really just need to translate the non-epub tags to something similar for input. You may or may not need to write an output plugin depending on how difficult it is to find patterns in an epub to turn into the kepub.
I want input and output support, but since a kepub is an epub, I'm hoping it'll be almost free. The only non-conforming part that I know of is extra attributes in the span tags, which I assume the parser will happily ignore.


When you say the intermediate format is epub, do you mean it's basically an exploded (unzipped) epub?
kc7zzv is offline   Reply With Quote
Old 02-11-2013, 08:01 PM   #4
user_none
Sigil & calibre developer
user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.
 
user_none's Avatar
 
Posts: 2,488
Karma: 1063785
Join Date: Jan 2009
Location: Florida, USA
Device: Nook STR
Quote:
Originally Posted by kc7zzv View Post
I want input and output support, but since a kepub is an epub, I'm hoping it'll be almost free.
It should be nearly free.

Quote:
Originally Posted by kc7zzv View Post
The only non-conforming part that I know of is extra attributes in the span tags, which I assume the parser will happily ignore.
I can't remember off the top of my head but I think it will stip the invalid tags.


Quote:
Originally Posted by kc7zzv View Post
When you say the intermediate format is epub, do you mean it's basically an exploded (unzipped) epub?
Pretty much.
user_none is offline   Reply With Quote
Old 02-11-2013, 10:25 PM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
http://manual.calibre-ebook.com/conv...l#introduction

http://manual.calibre-ebook.com/deve...ml#code-layout

For input all you will need to do is unzip and possibly convert any kepub specific features into their epub equivalents and return the path to the opf file.

For output, you will need to do the following (in outline)

1) Transform the markup to conform to kepub (add the span tags used for annotations, etc.)

1.5) Do various bits of epub specific processing like covers and so on, as is done in the epub output plugin.

2) Output to disk using the oeb writer as is done in the epub output plugin

3) zip up the files, again as is done in the epub output plugin

You can subclass the epub output plugin and re-use various bits of code from it. If you need some refactoring to make the code more re-usable, let me know.
kovidgoyal is offline   Reply With Quote
Old 02-12-2013, 05:25 AM   #6
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by kc7zzv View Post
A kepub is a format developed by Kobo based off of epub. It's (almost) an epub with a few files added, the main "html" doc broken into smaller chunks, and extra tags added.
Quote:
Originally Posted by kc7zzv View Post
I want input and output support, but since a kepub is an epub, I'm hoping it'll be almost free. The only non-conforming part that I know of is extra attributes in the span tags, which I assume the parser will happily ignore.
This is a little picky, but the "broken into smaller chunks" is wrong. The division of text into files in kepub is exactly the same as for epub. I have compared the kepub and epub versions of books I have bought from Kobo, and in general they are the same. None of the extra code is illegal. There are extra spans with a Kobo specific class and a sequential id.

If you want to do this, go ahead, but I don't think it is very useful. While any non-DRM protected kepub can be treated as an ePub, the extras aren't useful anywhere outside a Kobo device. Kepubs can be read with the Kobo iOS or Android apps, but I think they can only read those download from Kobo by the app.

If you want to use an epub as a kepub on the devices, then all you need to do is rename it to have ".kepub.epub" as the extension. That gives most of the "advantages" of a kepub.

There is also no point in adding DRM protected kepubs to calibre. You can't read them and you can't remove the DRM. And I don't think you can put them back onto the device and read them.

Finally, there is already a driver being developed to send epubs to the Kobo devices as kepubs. This is at https://github.com/jgoguen/calibre-kobo-driver. This does the updates to the epubs when the book is sent to the device. I will be merging this into the current driver when I am happy that it works well and I have time. Last time I checked, it would handle at least 90% of the epubs out there. The ones it doesn't would probably work after a calibre conversion. This has been discussed in the Kobo forum.
davidfor is offline   Reply With Quote
Old 02-13-2013, 12:50 AM   #7
kc7zzv
Member
kc7zzv began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Jan 2009
Device: Handspring Palm
Quote:
Originally Posted by davidfor View Post
This is a little picky, but the "broken into smaller chunks" is wrong. The division of text into files in kepub is exactly the same as for epub. I have compared the kepub and epub versions of books I have bought from Kobo, and in general they are the same. None of the extra code is illegal. There are extra spans with a Kobo specific class and a sequential id.
You're right. I remembered wrong. It was a custom attribute in one of the meta files, in regard to the cover.

Quote:
Originally Posted by davidfor View Post
If you want to do this, go ahead, but I don't think it is very useful. While any non-DRM protected kepub can be treated as an ePub, the extras aren't useful anywhere outside a Kobo device. Kepubs can be read with the Kobo iOS or Android apps, but I think they can only read those download from Kobo by the app.

If you want to use an epub as a kepub on the devices, then all you need to do is rename it to have ".kepub.epub" as the extension. That gives most of the "advantages" of a kepub.
As for splitting, I was oversimplifying. Basically, Calibre splits stories at a size that gives HORRENDOUS performance on my Kobo. I assume this value is a sane default on most devices, and I know it's a sane default on my old Kindle, but on the Kobo, it sucks. I assume this is optimized for 95% of readers, so this isn't a bug in Calibre. I want to see what I can do here, and because the Kobo needs more breaks, I want to see if I can make the splitting smarter.


Also, I want to change the files into something that can take full advantage of the "kepub reader" that Kobos have. Mostly, adding the ids, adding a simple checkbox for the css needed to help the Kobo remember the correct page. No, that's not a typo. Formatting can make the Kobo not remember the page at all. So, you might think that those features aren't worth the effort, but I do.

> There is also no point in adding DRM protected kepubs to calibre.

Agreed.

> Finally, there is already a driver being developed to send epubs to the Kobo devices as kepubs. This is at https://github.com/jgoguen/calibre-kobo-driver. This does the updates to the epubs when the book is sent to the device. I will be merging this into the current driver when I am happy that it works well and I have time. Last time I checked, it would handle at least 90% of the epubs out there. The ones it doesn't would probably work after a calibre conversion. This has been discussed in the Kobo forum.

My goal is to make this work with that plugin. I want the books per-converted and in my library, where it's easier to tweak them, and to make transferring to the Kobo faster after the first conversion, at the cost of some space.


So, to make a long story short, yes, you can use epubs on a Kobo, but if you want to make them work very well, you have to apply a lot of tweaks, including changing the file name. And, those changes makes the epub worst in most other readers. To me, that's justification for treating it as a separate format, considering that it already has a separate extension.


Edit: Yes, I understand that even if I do this, it will still need to be used with a custom driver.
kc7zzv is offline   Reply With Quote
Old 02-13-2013, 04:14 AM   #8
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by kc7zzv View Post
As for splitting, I was oversimplifying. Basically, Calibre splits stories at a size that gives HORRENDOUS performance on my Kobo.
You can change the split size in the epub output preference, but...

Quote:
Originally Posted by kc7zzv View Post
I want to see if I can make the splitting smarter.
...it doesn't split smart.

Quote:
Originally Posted by kc7zzv View Post
So, to make a long story short, yes, you can use epubs on a Kobo, but if you want to make them work very well, you have to apply a lot of tweaks, including changing the file name.
I don't own a Kobo but can't you configure the filename as needed via the metadata plugboard?

Whatever you and davidfor come up with I'm sure will benefit the Kobo community.
DoctorOhh is offline   Reply With Quote
Old 02-13-2013, 08:12 AM   #9
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by kc7zzv View Post
You're right. I remembered wrong. It was a custom attribute in one of the meta files, in regard to the cover.
That isn't correct. When a kepub is synced from the Kobo servers, the cover image is supplied separately. And the database entries aren't added in the same way as when a book is sideload. But, I assume you are talking about about the "cover-image" property. From looking at non-DRM kepubs, this is not used in them. My understanding is this is a property is from the epub3 spec. Adding it to a kepub, which is really a epub2, is a cheat I discovered a few weeks ago. I assume it works because Kobo is using the same code to handle epub3 and their kepubs.
Quote:

As for splitting, I was oversimplifying. Basically, Calibre splits stories at a size that gives HORRENDOUS performance on my Kobo. I assume this value is a sane default on most devices, and I know it's a sane default on my old Kindle, but on the Kobo, it sucks. I assume this is optimized for 95% of readers, so this isn't a bug in Calibre. I want to see what I can do here, and because the Kobo needs more breaks, I want to see if I can make the splitting smarter.
I don't know why you are seeing that performance. I recently took a 30 chapter epub that was built with one chapter per file and combined them down to three files. The sizes where about 350KB, 250KB and 700KB each. This worked perfectly on the device with no noticeable performance from the split version.

But if the sizes are the problem, you have your conversion options set wrong. You should use the "Structure Detection" page of the conversion options to split the files into chapter based files in the epub. This is what you are saying is needed to be done to create a kepubs, so you will be relying on those options anyway.

And to be clear, a kepub does not have to be split up into a file per chapter. They follow the exact same rules for this as an epub. The designer of the book decides how to do it and Kobo just adds the extra spans and ids. I haven't seen many kepubs that don't split the chapters, but I do have some. And an important thing is that every kepub I have looked at, has exactly the same file structure as the epub I also got from the Kobo shop.
Quote:
Also, I want to change the files into something that can take full advantage of the "kepub reader" that Kobos have. Mostly, adding the ids, adding a simple checkbox for the css needed to help the Kobo remember the correct page. No, that's not a typo. Formatting can make the Kobo not remember the page at all. So, you might think that those features aren't worth the effort, but I do.
Quote:
To be honest, I don't like the kepub reader. I don't like the space taken at the top of the screen by the title all the time. I don't like the per-chapter page numbering. The only thing that tempts me to read the kepub version of books I have bought is the reading stats. They are nice, but I am not hooked on them.
> There is also no point in adding DRM protected kepubs to calibre.

Agreed.

> Finally, there is already a driver being developed to send epubs to the Kobo devices as kepubs. This is at https://github.com/jgoguen/calibre-kobo-driver. This does the updates to the epubs when the book is sent to the device. I will be merging this into the current driver when I am happy that it works well and I have time. Last time I checked, it would handle at least 90% of the epubs out there. The ones it doesn't would probably work after a calibre conversion. This has been discussed in the Kobo forum.

My goal is to make this work with that plugin. I want the books per-converted and in my library, where it's easier to tweak them, and to make transferring to the Kobo faster after the first conversion, at the cost of some space.
If kepub handling is added to calibre, that driver or updates to the current driver are not needed. The only thing that is needed is to add kepub to the list of formats that the driver handles. And to make sure the extension is handled correctly. If you are looking at a version of Joel's driver that does database updates, they will definitely not be in the merged driver. And looking at the code, Joel has pulled them out as there are two many issues with them.

Performance is a valid consideration. But, I suspect most will still be done when sent to the device the first time. And how many will be sent multiple times? The main times would be when you change the book for some reason. In that case, editing the original source makes more sense than the converted kepub.
Quote:
So, to make a long story short, yes, you can use epubs on a Kobo, but if you want to make them work very well, you have to apply a lot of tweaks, including changing the file name. And, those changes makes the epub worst in most other readers. To me, that's justification for treating it as a separate format, considering that it already has a separate extension.
What changes are you talking about? I tweak my epubs, but that is because I prefer them to look like paper books. In fact, if I have a paper version, I use that as the guide. But, I would be stunned if I am doing anything that would make it look worse on other readers. About the only thing I do that I think is specific to the Kobo reader is to push chapter headings low enough on the screen that they aren't hidden by the top menu when it is displayed. If you have have some examples, I would love to hear what they are.

The reference to file name changes here has me confused. Are you talking about changes to get the epub treated as a kepub? Or when you want to just use it as an epub? Because of the rest of the paragraph, I assume the latter. But what changes do you need to make? The default calibre save template and file name handling works perfectly. The only issue I can think of is related to unusual characters in the title of a book. As the save template used these in the file name, that caused problem. The fix was actually to correct the book title.

I know I am being discouraging about this. I have thought about this on and off since I started working on the driver. I have never seen enough advantage to it. When Joel's driver turned up, I thought that was a better way to handle this. kepubs are only useful in one place, so keep them there.

Having said all the above, if you go ahead and do this, I'll watch with interest. And I'll help however I can. That is especially so for the testing. If this is done, I want it working properly. Also, if any changes are needed in the driver, I'll make them.
davidfor is offline   Reply With Quote
Old 02-13-2013, 12:18 PM   #10
kc7zzv
Member
kc7zzv began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Jan 2009
Device: Handspring Palm
Quote:
Originally Posted by davidfor View Post
That isn't correct. When a kepub is synced from the Kobo servers, the cover image is supplied separately. And the database entries aren't added in the same way as when a book is sideload. But, I assume you are talking about about the "cover-image" property. From looking at non-DRM kepubs, this is not used in them. My understanding is this is a property is from the epub3 spec. Adding it to a kepub, which is really a epub2, is a cheat I discovered a few weeks ago. I assume it works because Kobo is using the same code to handle epub3 and their kepubs.
Could you PLEASE stop belaboring this point. I don't CARE. Either it doesn't follow the spec, in which case I don't care, because it's "close enough" for import to treat as an epub or it does follow the spec, in which case it's (by definition) close enough to treat as an epub for import purposes.

> I don't know why you are seeing that performance. I recently took a 30 chapter epub that was built with one chapter per file and combined them down to three files. The sizes where about 350KB, 250KB and 700KB each. This worked perfectly on the device with no noticeable performance from the split version.

Well, I don't either. I assume it has something to do with the size, and I expect to find out during my initial exploration.

> But if the sizes are the problem, you have your conversion options set wrong. You should use the "Structure Detection" page of the conversion options to split the files into chapter based files in the epub. This is what you are saying is needed to be done to create a kepubs, so you will be relying on those options anyway.

> to be clear, a kepub does not have to be split up into a file per chapter. They follow the exact same rules for this as an epub. The designer of the book decides how to do it and Kobo just adds the extra spans and ids. I haven't seen many kepubs that don't split the chapters, but I do have some. And an important thing is that every kepub I have looked at, has exactly the same file structure as the epub I also got from the Kobo shop.

As I see it, this is entirely beside the point. Split size (in my very limited testing) can heavily impact performance, in which case, the split size on kepubs should be optimized for the kobo. It is possible that it doesn't have an impact on performance, in which case I expect to find that out too.

Just give up on convincing me that this won't speed things up. I've read enough information supporting this to believe it. Either it will, and I'll find that out in my exploration, before I write much code, or it won't, and I'll find that out during my exploration, before I write much code.


> What changes are you talking about? I tweak my epubs, but that is because I prefer them to look like paper books. In fact, if I have a paper version, I use that as the guide. But, I would be stunned if I am doing anything that would make it look worse on other readers. About the only thing I do that I think is specific to the Kobo reader is to push chapter headings low enough on the screen that they aren't hidden by the top menu when it is displayed. If you have have some examples, I would love to hear what they are.

a) Chopping the epub into tiny pieces puts in page-breaks in ugly places. It it doesn't help performance, then this isn't something I want on other readers.
b) Adding the kobo ids, spans, and javascript make the file bigger, and so does splitting the file into more pieces.



Finally, something you didn't mention is that some people don't want their Kobo reading their epubs in the "kepub reader". (Yes, calling it the kepub reader is a slight oversimplification. Please don't waste our time correcting me) By treating both formats as first class formats, users can easily choose which they want in the same way they choose between other formats, like epub vs lit.


Quote:
Originally Posted by DoctorOhh View Post
You can change the split size in the epub output preference, but...



...it doesn't split smart.
Yes. I want to try to split on chapters.



Quote:
Originally Posted by DoctorOhh View Post
I don't own a Kobo but can't you configure the filename as needed via the metadata plugboard?

I once had a computer case... It didn't have enough circulation when I put a beefy video card in it. It worked really well after drilling several holes in it, but it didn't work out of the box for me.

Basically, I want to make Calibre work really well with the Kobo, out of the box, not just for me after configuring settings, templates, the plugboard, etc...

Last edited by kc7zzv; 02-13-2013 at 12:21 PM.
kc7zzv is offline   Reply With Quote
Old 02-13-2013, 12:36 PM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You will find there is no optimum split size. That's because rendering complexity doesn't actually depend on split size, it depends on markup complexity. Split size is just an easily understandable crutch the original developers at Adobe used as a way to get *most* epubs to work well with their rendering engine.

calibre does try to split at logical points when those are available. You will discover that for books where the splitting does not happen at logical points, the books have no machine identifiable logical points to split at.

Finally, the extra markup in kepubs is almost certainly there to allow for annotations and bookmarking and likely serves no other purpose. Apparently the developers at Kobo, like the developers at Amazon with KF8 (which uses a very similar scheme) chose not to implement the EPUB 3 CFI spec.
kovidgoyal is offline   Reply With Quote
Old 02-13-2013, 10:22 PM   #12
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by kc7zzv View Post
Could you PLEASE stop belaboring this point. I don't CARE. Either it doesn't follow the spec, in which case I don't care, because it's "close enough" for import to treat as an epub or it does follow the spec, in which case it's (by definition) close enough to treat as an epub for import purposes.
My apologies for pointing out an error in your post and explaining it. But, as it was the first time I referred to this in this thread, I think "Belabouring this point", is a bit much.

Now if you had accused me of being pedantic in my explanation, then I will plead guilty. I was trying to make sure you understood what was going on. The only issue with the "close enough" is that Kobo might decide this is a bug (reading a epub 3 attribute in an epub 2 file) and fix it.

But, to belabour a point before I continue. I think that having calibre capable of putting properly formatted kepubs on the Kobo devices is a good idea. But, I am not convinced that having kepub as a format inside calibre is very useful. And I think don't think it is more useful than the driver method.
Quote:
> I don't know why you are seeing that performance. I recently took a 30 chapter epub that was built with one chapter per file and combined them down to three files. The sizes where about 350KB, 250KB and 700KB each. This worked perfectly on the device with no noticeable performance from the split version.

Well, I don't either. I assume it has something to do with the size, and I expect to find out during my initial exploration.
Huh? "Well, I don't either" what? A big part of your premise is that size matters. If you don't see a difference in performance, than why are you worrying? But below you have done some testing and see a heavy impact.
Quote:
> But if the sizes are the problem, you have your conversion options set wrong. You should use the "Structure Detection" page of the conversion options to split the files into chapter based files in the epub. This is what you are saying is needed to be done to create a kepubs, so you will be relying on those options anyway.

> to be clear, a kepub does not have to be split up into a file per chapter. They follow the exact same rules for this as an epub. The designer of the book decides how to do it and Kobo just adds the extra spans and ids. I haven't seen many kepubs that don't split the chapters, but I do have some. And an important thing is that every kepub I have looked at, has exactly the same file structure as the epub I also got from the Kobo shop.

As I see it, this is entirely beside the point. Split size (in my very limited testing) can heavily impact performance, in which case, the split size on kepubs should be optimized for the kobo. It is possible that it doesn't have an impact on performance, in which case I expect to find that out too.

Just give up on convincing me that this won't speed things up. I've read enough information supporting this to believe it. Either it will, and I'll find that out in my exploration, before I write much code, or it won't, and I'll find that out during my exploration, before I write much code.
I am not trying to convince you that splitting a epub or kepub into smaller chunks won't be faster when turning the page. I am fairly sure that it will be. But there is a minimum size that makes it slower as going from one file in the epub/kepub to another is slower.

From what I see for performance and usability, the best split is at chapter marks [pedantic me: where it says "Chapter x" or "Part x" or just "x"]. It is rare to find a book that has chapters large enough to want to split them into smaller files. But, it is common to find epubs that have not been split at chapters.

As you think that size of file is so important, do the tests now. Play with the split settings in the calibre conversion to to set-up file size. Then use Joel's driver to put properly formatted kepubs onto the device. But also put the epub version on to see what the difference is with the device treating the same book as an epub an a kepub (yes the kepub has extra spans in it which makes it not the same, but without the spans, it isn't correctly formatted for the kepub reader). And please tell me the results. I am very interested to know what they are. I'll admit that I expect the size range for well performing epubs and kepubs will be very wide. I know that the first thing I will do after seeing the results is examining my library to see how they fit and what changes I need to make.

Quote:
> What changes are you talking about? I tweak my epubs, but that is because I prefer them to look like paper books. In fact, if I have a paper version, I use that as the guide. But, I would be stunned if I am doing anything that would make it look worse on other readers. About the only thing I do that I think is specific to the Kobo reader is to push chapter headings low enough on the screen that they aren't hidden by the top menu when it is displayed. If you have have some examples, I would love to hear what they are.

a) Chopping the epub into tiny pieces puts in page-breaks in ugly places. It it doesn't help performance, then this isn't something I want on other readers.
Why one earth would you do that? It is not needed on any of the Kobo devices to split the text into "tiny pieces". Or do we disagree on the how big "tiny" is? And if you read my posts, I am a big advocate of splitting at chapter breaks. That will not put page-breaks in ugly places.
Quote:
b) Adding the kobo ids, spans, and javascript make the file bigger, and so does splitting the file into more pieces.
Huh? Of course all that is the case. Why would you do any of that if the size is a big concern to you. The spans and ids have nothing to do with other readers (that was the point I was asking about) as they are only added to kepubs. Same with the kepub specific javascript. But I don't understand how this affects other readers as kepubs don't go onto them.
Quote:
Finally, something you didn't mention is that some people don't want their Kobo reading their epubs in the "kepub reader". (Yes, calling it the kepub reader is a slight oversimplification. Please don't waste our time correcting me) By treating both formats as first class formats, users can easily choose which they want in the same way they choose between other formats, like epub vs lit.
Firstly, I won't correct you because I think "kepub reader" is a perfectly good description. I do use "kepub renderer" in some places, but that is because it fits the discussion.

But, I'm not sure if I understand the first sentence. I do know that a lot of people don't want their epubs read in the kepub reader (I'm one of them). And I do know that a lot of people want to read their epubs in the kepub reader. If you go over to the Kobo forum, you will see I have discussed it there plenty of times. In fact, I started a thread to tell people how to do this (and had someone promptly point to a much earlier thread that I had forgotten about).
Quote:
Yes. I want to try to split on chapters.
This time I am belabouring a point. Calibre conversion does this already and does it very well. You don't need to do anything to get it. And you don't want to not offer it as an option. Some people don't want this and will want the single file. Even if the performance is not as good.
Quote:
Basically, I want to make Calibre work really well with the Kobo, out of the box, not just for me after configuring settings, templates, the plugboard, etc...
Sorry, but calibre does work very well with the Kobo device out of the box with no configuration changes. Any changes people make are because they want to fine tune it. The issues you are talking about are related to the structure of the books being read or maybe some firmware bugs.

I'll repeat my ending statement from the last post:

I am being discouraging about this because I don't think it is that useful. And I am also playing Devil's Advocate a bit to make sure you understand what you are planning to do. But, if you decide to do this, I will be happy to help.
davidfor is offline   Reply With Quote
Old 02-13-2013, 10:39 PM   #13
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by kovidgoyal View Post
Finally, the extra markup in kepubs is almost certainly there to allow for annotations and bookmarking and likely serves no other purpose. Apparently the developers at Kobo, like the developers at Amazon with KF8 (which uses a very similar scheme) chose not to implement the EPUB 3 CFI spec.
That is correct. I have looked at the stored bookmarks on the device for kepubs to check this. And to compare with what they are doing for epubs and other formats.

I just had a quick look at the EPUB 3 CFI spec. Is the bookmark in the calibre viewer based on this? From the examples, it looks similar, but that might be a coincidence. The reason for the interest is that I have been meaning to look see if I could translate bookmarks between the Kobo to the viewer.
davidfor is offline   Reply With Quote
Old 02-13-2013, 11:17 PM   #14
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Yes the viewer uses CFI (more precisely, it uses CFI for the location inside a particular HTML file, but the file itself is pointed to by other means).
kovidgoyal is offline   Reply With Quote
Old 02-14-2013, 06:05 AM   #15
jgoguen
Generally Awesome Person
jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.jgoguen ought to be getting tired of karma fortunes by now.
 
Posts: 1,061
Karma: 2178845
Join Date: Jan 2013
Location: /dev/kmem
Device: Kobo Clara HD, Kindle Oasis
kc7zzv

Why would you write a plugin at a point where you can't choose to disable the kepub features when sending a book to the device? And it sounds like you're going to be reimplementing a good chunk of some of calibre's conversion pipeline. I already need to decide if I want to process CSS on my own, and I'm leaning toward "no way, let calibre handle it". Code should be re-used whenever possible, but only copied when it's essential.

It strikes me (as a purely personal opinion, perhaps wrong because text fails to convey intonation and emotion) that you're being a bit rude to davidfor. He's trying to help you make your plugin better. It is a good thing to play Devil's Advocate to make sure issues with whatever is currently known get brought up and thought about. It also is an excellent way to make a good developer consider whether they are approaching the problem in the right way. It doesn't mean you won't get help, just that you need to think. Go look at the .kepub.epub thread in the Kobo reader forum, davidfor disagreed strongly with the database code, but he gave me some really good information on how to proceed with making it better anyway. He also disagrees with me having the kepub processing enabled by default, but I agree with his reasoning on keeping it off by default; the only reason I don't is because it's the entire point of my plugin to do that processing. Maybe you feel that you're getting criticism instead? Good developers take polite criticism and use it to improve the code. It's not always easy, but it's for the best. You were even given a link that gives you a good chunk of the code you're going to need.

TL;DR - I'm not convinced (and I may not be alone) that a file format plugin is the way to approach this, but if you want to do it anyway you'll find lots of help if you want it.
jgoguen is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Touch .kepub.epub davidfor Kobo Reader 233 01-12-2014 08:15 AM
Kepub - how does it work? Lynx-lynx Kobo Reader 5 01-31-2013 06:09 AM
Plugin calibre epub to kepub Tumeconnais Kobo Developer's Corner 0 01-16-2013 04:26 PM
Plugin for making a summary list about your books equilan1986 Plugins 2 01-08-2013 07:21 AM
A little help and direction... stevejay Deals and Resources (No Self-Promotion or Affiliate Links) 12 02-19-2009 05:24 PM


All times are GMT -4. The time now is 05:46 AM.


MobileRead.com is a privately owned, operated and funded community.