02-11-2013, 12:58 PM | #1 |
Member
Posts: 12
Karma: 10
Join Date: Jan 2009
Device: Handspring Palm
|
Looking for direction for making a kepub plugin
A kepub is a format developed by Kobo based off of epub. It's (almost) an epub with a few files added, the main "html" doc broken into smaller chunks, and extra tags added.
Are there any examples of "format" plugins (pdf,mobi,epub,cbz...) that are mostly based off of another plugin? I'm trying to figure out how I should be doing this, without going the simple route and stupidly copying and pasting a lot of code. I'm reasonably experienced at programming, by my python experience is so lacking, I want to at least be imitating a good example. |
02-11-2013, 07:46 PM | #2 |
Sigil & calibre developer
Posts: 2,488
Karma: 1063785
Join Date: Jan 2009
Location: Florida, USA
Device: Nook STR
|
Input or output support?
The base flow is: Input format -> intermediate (epub) format -> Output format. An input plugin takes the input format and converts it to html/CSS. The output format takes html/CSS and turns it into the output format. TXT is a bit complex (lots of options) but it's a good one to look at for general flow. Mainly because it is a non-html format on both ends. You'll probably end up subclassing a lot of the epub code since you really just need to translate the non-epub tags to something similar for input. You may or may not need to write an output plugin depending on how difficult it is to find patterns in an epub to turn into the kepub. |
02-11-2013, 07:56 PM | #3 | |
Member
Posts: 12
Karma: 10
Join Date: Jan 2009
Device: Handspring Palm
|
Quote:
When you say the intermediate format is epub, do you mean it's basically an exploded (unzipped) epub? |
|
02-11-2013, 08:01 PM | #4 | ||
Sigil & calibre developer
Posts: 2,488
Karma: 1063785
Join Date: Jan 2009
Location: Florida, USA
Device: Nook STR
|
Quote:
Quote:
Pretty much. |
||
02-11-2013, 10:25 PM | #5 |
creator of calibre
Posts: 43,858
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
http://manual.calibre-ebook.com/conv...l#introduction
http://manual.calibre-ebook.com/deve...ml#code-layout For input all you will need to do is unzip and possibly convert any kepub specific features into their epub equivalents and return the path to the opf file. For output, you will need to do the following (in outline) 1) Transform the markup to conform to kepub (add the span tags used for annotations, etc.) 1.5) Do various bits of epub specific processing like covers and so on, as is done in the epub output plugin. 2) Output to disk using the oeb writer as is done in the epub output plugin 3) zip up the files, again as is done in the epub output plugin You can subclass the epub output plugin and re-use various bits of code from it. If you need some refactoring to make the code more re-usable, let me know. |
02-12-2013, 05:25 AM | #6 | ||
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
Quote:
If you want to do this, go ahead, but I don't think it is very useful. While any non-DRM protected kepub can be treated as an ePub, the extras aren't useful anywhere outside a Kobo device. Kepubs can be read with the Kobo iOS or Android apps, but I think they can only read those download from Kobo by the app. If you want to use an epub as a kepub on the devices, then all you need to do is rename it to have ".kepub.epub" as the extension. That gives most of the "advantages" of a kepub. There is also no point in adding DRM protected kepubs to calibre. You can't read them and you can't remove the DRM. And I don't think you can put them back onto the device and read them. Finally, there is already a driver being developed to send epubs to the Kobo devices as kepubs. This is at https://github.com/jgoguen/calibre-kobo-driver. This does the updates to the epubs when the book is sent to the device. I will be merging this into the current driver when I am happy that it works well and I have time. Last time I checked, it would handle at least 90% of the epubs out there. The ones it doesn't would probably work after a calibre conversion. This has been discussed in the Kobo forum. |
||
02-13-2013, 12:50 AM | #7 | ||
Member
Posts: 12
Karma: 10
Join Date: Jan 2009
Device: Handspring Palm
|
Quote:
Quote:
Also, I want to change the files into something that can take full advantage of the "kepub reader" that Kobos have. Mostly, adding the ids, adding a simple checkbox for the css needed to help the Kobo remember the correct page. No, that's not a typo. Formatting can make the Kobo not remember the page at all. So, you might think that those features aren't worth the effort, but I do. > There is also no point in adding DRM protected kepubs to calibre. Agreed. > Finally, there is already a driver being developed to send epubs to the Kobo devices as kepubs. This is at https://github.com/jgoguen/calibre-kobo-driver. This does the updates to the epubs when the book is sent to the device. I will be merging this into the current driver when I am happy that it works well and I have time. Last time I checked, it would handle at least 90% of the epubs out there. The ones it doesn't would probably work after a calibre conversion. This has been discussed in the Kobo forum. My goal is to make this work with that plugin. I want the books per-converted and in my library, where it's easier to tweak them, and to make transferring to the Kobo faster after the first conversion, at the cost of some space. So, to make a long story short, yes, you can use epubs on a Kobo, but if you want to make them work very well, you have to apply a lot of tweaks, including changing the file name. And, those changes makes the epub worst in most other readers. To me, that's justification for treating it as a separate format, considering that it already has a separate extension. Edit: Yes, I understand that even if I do this, it will still need to be used with a custom driver. |
||
02-13-2013, 04:14 AM | #8 | ||
US Navy, Retired
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
...it doesn't split smart. Quote:
Whatever you and davidfor come up with I'm sure will benefit the Kobo community. |
||
02-13-2013, 08:12 AM | #9 | |||||
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
Quote:
But if the sizes are the problem, you have your conversion options set wrong. You should use the "Structure Detection" page of the conversion options to split the files into chapter based files in the epub. This is what you are saying is needed to be done to create a kepubs, so you will be relying on those options anyway. And to be clear, a kepub does not have to be split up into a file per chapter. They follow the exact same rules for this as an epub. The designer of the book decides how to do it and Kobo just adds the extra spans and ids. I haven't seen many kepubs that don't split the chapters, but I do have some. And an important thing is that every kepub I have looked at, has exactly the same file structure as the epub I also got from the Kobo shop. Quote:
Performance is a valid consideration. But, I suspect most will still be done when sent to the device the first time. And how many will be sent multiple times? The main times would be when you change the book for some reason. In that case, editing the original source makes more sense than the converted kepub. Quote:
The reference to file name changes here has me confused. Are you talking about changes to get the epub treated as a kepub? Or when you want to just use it as an epub? Because of the rest of the paragraph, I assume the latter. But what changes do you need to make? The default calibre save template and file name handling works perfectly. The only issue I can think of is related to unusual characters in the title of a book. As the save template used these in the file name, that caused problem. The fix was actually to correct the book title. I know I am being discouraging about this. I have thought about this on and off since I started working on the driver. I have never seen enough advantage to it. When Joel's driver turned up, I thought that was a better way to handle this. kepubs are only useful in one place, so keep them there. Having said all the above, if you go ahead and do this, I'll watch with interest. And I'll help however I can. That is especially so for the testing. If this is done, I want it working properly. Also, if any changes are needed in the driver, I'll make them. |
|||||
02-13-2013, 12:18 PM | #10 | |||
Member
Posts: 12
Karma: 10
Join Date: Jan 2009
Device: Handspring Palm
|
Quote:
> I don't know why you are seeing that performance. I recently took a 30 chapter epub that was built with one chapter per file and combined them down to three files. The sizes where about 350KB, 250KB and 700KB each. This worked perfectly on the device with no noticeable performance from the split version. Well, I don't either. I assume it has something to do with the size, and I expect to find out during my initial exploration. > But if the sizes are the problem, you have your conversion options set wrong. You should use the "Structure Detection" page of the conversion options to split the files into chapter based files in the epub. This is what you are saying is needed to be done to create a kepubs, so you will be relying on those options anyway. > to be clear, a kepub does not have to be split up into a file per chapter. They follow the exact same rules for this as an epub. The designer of the book decides how to do it and Kobo just adds the extra spans and ids. I haven't seen many kepubs that don't split the chapters, but I do have some. And an important thing is that every kepub I have looked at, has exactly the same file structure as the epub I also got from the Kobo shop. As I see it, this is entirely beside the point. Split size (in my very limited testing) can heavily impact performance, in which case, the split size on kepubs should be optimized for the kobo. It is possible that it doesn't have an impact on performance, in which case I expect to find that out too. Just give up on convincing me that this won't speed things up. I've read enough information supporting this to believe it. Either it will, and I'll find that out in my exploration, before I write much code, or it won't, and I'll find that out during my exploration, before I write much code. > What changes are you talking about? I tweak my epubs, but that is because I prefer them to look like paper books. In fact, if I have a paper version, I use that as the guide. But, I would be stunned if I am doing anything that would make it look worse on other readers. About the only thing I do that I think is specific to the Kobo reader is to push chapter headings low enough on the screen that they aren't hidden by the top menu when it is displayed. If you have have some examples, I would love to hear what they are. a) Chopping the epub into tiny pieces puts in page-breaks in ugly places. It it doesn't help performance, then this isn't something I want on other readers. b) Adding the kobo ids, spans, and javascript make the file bigger, and so does splitting the file into more pieces. Finally, something you didn't mention is that some people don't want their Kobo reading their epubs in the "kepub reader". (Yes, calling it the kepub reader is a slight oversimplification. Please don't waste our time correcting me) By treating both formats as first class formats, users can easily choose which they want in the same way they choose between other formats, like epub vs lit. Quote:
Quote:
I once had a computer case... It didn't have enough circulation when I put a beefy video card in it. It worked really well after drilling several holes in it, but it didn't work out of the box for me. Basically, I want to make Calibre work really well with the Kobo, out of the box, not just for me after configuring settings, templates, the plugboard, etc... Last edited by kc7zzv; 02-13-2013 at 12:21 PM. |
|||
02-13-2013, 12:36 PM | #11 |
creator of calibre
Posts: 43,858
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You will find there is no optimum split size. That's because rendering complexity doesn't actually depend on split size, it depends on markup complexity. Split size is just an easily understandable crutch the original developers at Adobe used as a way to get *most* epubs to work well with their rendering engine.
calibre does try to split at logical points when those are available. You will discover that for books where the splitting does not happen at logical points, the books have no machine identifiable logical points to split at. Finally, the extra markup in kepubs is almost certainly there to allow for annotations and bookmarking and likely serves no other purpose. Apparently the developers at Kobo, like the developers at Amazon with KF8 (which uses a very similar scheme) chose not to implement the EPUB 3 CFI spec. |
02-13-2013, 10:22 PM | #12 | ||||||||
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
Now if you had accused me of being pedantic in my explanation, then I will plead guilty. I was trying to make sure you understood what was going on. The only issue with the "close enough" is that Kobo might decide this is a bug (reading a epub 3 attribute in an epub 2 file) and fix it. But, to belabour a point before I continue. I think that having calibre capable of putting properly formatted kepubs on the Kobo devices is a good idea. But, I am not convinced that having kepub as a format inside calibre is very useful. And I think don't think it is more useful than the driver method. Quote:
Quote:
From what I see for performance and usability, the best split is at chapter marks [pedantic me: where it says "Chapter x" or "Part x" or just "x"]. It is rare to find a book that has chapters large enough to want to split them into smaller files. But, it is common to find epubs that have not been split at chapters. As you think that size of file is so important, do the tests now. Play with the split settings in the calibre conversion to to set-up file size. Then use Joel's driver to put properly formatted kepubs onto the device. But also put the epub version on to see what the difference is with the device treating the same book as an epub an a kepub (yes the kepub has extra spans in it which makes it not the same, but without the spans, it isn't correctly formatted for the kepub reader). And please tell me the results. I am very interested to know what they are. I'll admit that I expect the size range for well performing epubs and kepubs will be very wide. I know that the first thing I will do after seeing the results is examining my library to see how they fit and what changes I need to make. Quote:
Quote:
Quote:
But, I'm not sure if I understand the first sentence. I do know that a lot of people don't want their epubs read in the kepub reader (I'm one of them). And I do know that a lot of people want to read their epubs in the kepub reader. If you go over to the Kobo forum, you will see I have discussed it there plenty of times. In fact, I started a thread to tell people how to do this (and had someone promptly point to a much earlier thread that I had forgotten about). Quote:
Quote:
I'll repeat my ending statement from the last post: I am being discouraging about this because I don't think it is that useful. And I am also playing Devil's Advocate a bit to make sure you understand what you are planning to do. But, if you decide to do this, I will be happy to help. |
||||||||
02-13-2013, 10:39 PM | #13 | |
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
I just had a quick look at the EPUB 3 CFI spec. Is the bookmark in the calibre viewer based on this? From the examples, it looks similar, but that might be a coincidence. The reason for the interest is that I have been meaning to look see if I could translate bookmarks between the Kobo to the viewer. |
|
02-13-2013, 11:17 PM | #14 |
creator of calibre
Posts: 43,858
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Yes the viewer uses CFI (more precisely, it uses CFI for the location inside a particular HTML file, but the file itself is pointed to by other means).
|
02-14-2013, 06:05 AM | #15 |
Generally Awesome Person
Posts: 1,061
Karma: 2178845
Join Date: Jan 2013
Location: /dev/kmem
Device: Kobo Clara HD, Kindle Oasis
|
kc7zzv
Why would you write a plugin at a point where you can't choose to disable the kepub features when sending a book to the device? And it sounds like you're going to be reimplementing a good chunk of some of calibre's conversion pipeline. I already need to decide if I want to process CSS on my own, and I'm leaning toward "no way, let calibre handle it". Code should be re-used whenever possible, but only copied when it's essential.
It strikes me (as a purely personal opinion, perhaps wrong because text fails to convey intonation and emotion) that you're being a bit rude to davidfor. He's trying to help you make your plugin better. It is a good thing to play Devil's Advocate to make sure issues with whatever is currently known get brought up and thought about. It also is an excellent way to make a good developer consider whether they are approaching the problem in the right way. It doesn't mean you won't get help, just that you need to think. Go look at the .kepub.epub thread in the Kobo reader forum, davidfor disagreed strongly with the database code, but he gave me some really good information on how to proceed with making it better anyway. He also disagrees with me having the kepub processing enabled by default, but I agree with his reasoning on keeping it off by default; the only reason I don't is because it's the entire point of my plugin to do that processing. Maybe you feel that you're getting criticism instead? Good developers take polite criticism and use it to improve the code. It's not always easy, but it's for the best. You were even given a link that gives you a good chunk of the code you're going to need. TL;DR - I'm not convinced (and I may not be alone) that a file format plugin is the way to approach this, but if you want to do it anyway you'll find lots of help if you want it. |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Touch .kepub.epub | davidfor | Kobo Reader | 233 | 01-12-2014 08:15 AM |
Kepub - how does it work? | Lynx-lynx | Kobo Reader | 5 | 01-31-2013 06:09 AM |
Plugin calibre epub to kepub | Tumeconnais | Kobo Developer's Corner | 0 | 01-16-2013 04:26 PM |
Plugin for making a summary list about your books | equilan1986 | Plugins | 2 | 01-08-2013 07:21 AM |
A little help and direction... | stevejay | Deals and Resources (No Self-Promotion or Affiliate Links) | 12 | 02-19-2009 05:24 PM |