A need on new type of plugin or recipe

SauliusP. · 03-13-2013, 03:14 AM

There is a need of new type of plugin, which I woud call "InterconversionPlugin" :-)

Generic use case:
- Convert a book from one format to another.
- Plugin to make modifications on an intermediate OEB.

Sample use case:
- Convert book from EPUB to AZW3.
- Scan intermediate OEB, parse it and add soft-hyphenation marks (as Kindle does not support active hyphenation, but with new firmwares interprets correctly).

Requirements:
- Plugin should have it's configuration icon in conversion dialogue an its own configuration widget.
- There could be any count of these intermediate plugins.
- [Low Priority.] End user should be able to change the order of plugins to be applied (in case they might work on same area and have influence on final result).

It would be possible to add GUI plugin and do the same type of work or actually replicate all conversion dialogue and even to make it extensible, but it is probably not worth it.

kovidgoyal · 03-13-2013, 03:42 AM

I'm confused. Are you asking for plugins that can make changes *during* a conversion, that is, while the conversion pipeline is running?

If all you want to do is make changes to epub and azw3 files in OEB form, just create a user interface action plugin and use the polish.container module to get the OEB representation.

SauliusP. · 03-13-2013, 04:03 AM

Well, yes, I imagine possible intermediate changes in conversion pipeline — right in the middle between Input and Output.

If polish.container could "explode" AZW3 (and any other format) to OEB and then "implode" back with changes, that would be enough. Is it possible?

kovidgoyal · 03-13-2013, 04:11 AM

It can explode and implode azw3 and epub.

SauliusP. · 03-13-2013, 04:30 AM

Any example, please?

kovidgoyal · 03-13-2013, 04:34 AM

Just look at the source of container.py at the bottom is code to explode and implode any epub and azw3 file passed in on the command line.

SauliusP. · 03-13-2013, 04:48 AM

Allright, thank you. That seems to be a possibility. Will come back with questions here if any.

SauliusP. · 03-13-2013, 05:49 AM

Unfortunately, at first glance I don't understand how I could use anything from container.py. It seems to be specific to epub only. And epub itself is very close (if not the same) to OEB. I would like to make hyphenator as much format independent as possible. That's why I was thinking of intermediate conversion action.

In case it is not possible to use something generic to any format, I will stick probably with epub and azw3, maybe mobi too.

kovidgoyal · 03-13-2013, 05:54 AM

As I said before, container.py works with both epub and azw3.

SauliusP. · 03-16-2013, 09:46 AM

Need further help.

Hyphenator implemented and I have employed Container for both EPUB and AZW3.

However, rebuilding AZW3 strips entities from text. They remain in EPUB (which, I saw, simply rebuilds zip). I believe, it is some "problem" in re-conversion in AZW3 container's commit method. I have tried to leave hyphens instead of and then they are present. But when I add entities, they disappear.

Also, I have tried conversion from EPUB with these entities and AZW3 again does strip them. Digging through Calibre's code and settings I could not find, how to preserve these entities. Probably I am missing something simple, but...

Maybe those entities are stripped on purpose, as only new Kindle firmware supports them?

Please help!

kovidgoyal · 03-16-2013, 12:28 PM

It's possible that hyphenation entites are stripped from azw3 as the Kindle doesn't support them I dont recall. You'll have to look through the code of the azw3 plugin to find out. It may also be that the entities are converted into their unicode equivalent code points.

SauliusP. · 03-16-2013, 12:40 PM

Entity with name or with character code is definitely stripped out. Will try direct unicode character too. But somehow doubt any of those would work.

SauliusP. · 03-18-2013, 03:04 AM

Problem with AZW3 hyphenation solved. However, after rebuilding AZW3 book (with "tweak book" native feature too) Kindle does not show TOC and does not allow navigation through chapters with four-way controller (on Keyboard and 4 NT). Something's gone missing after rebuild.

kovidgoyal · 03-18-2013, 03:18 AM

If you're asking for my help you're going to have to provide more info than that. The azw3 rebuild process uses the same code as the azw3 output conversion process and the ToC most definitely works on azw3 files generated by the conversion process.

At a minimum an azw3 file whose toc works before tweaking but not after tweaking.

SauliusP. · 03-18-2013, 04:12 AM

Hi Kovid,

Please, help :-)
Kindly download books from here.

I have included two books. English original and processed through my hyphenation plugin. Lithuanian original, hyphenated and also original tweaked via included "tweak ebook" menu. "Originals" both are converted from EPUB and show ToC and chapters fine. Processed and tweaked do not.

Device is Kindle 4 non-touch with latest firmware.
Tried with version 0.9.22 of Calibre.

I know, that ToC and chapters work with AZW3, it is my primary used format. But have never tried tweaking before. I have also tried to explode original and modified books, OEB contains "toc.ncx" and no other obvious issues.

03-13-2013, 03:14 AM	#1
SauliusP. Plugin developer Posts: 108 Karma: 24394 Join Date: Feb 2012 Location: Lithuania Device: Kindle	A need on new type of plugin or recipe There is a need of new type of plugin, which I woud call "InterconversionPlugin" :-) Generic use case: - Convert a book from one format to another. - Plugin to make modifications on an intermediate OEB. Sample use case: - Convert book from EPUB to AZW3. - Scan intermediate OEB, parse it and add soft-hyphenation marks (as Kindle does not support active hyphenation, but with new firmwares interprets correctly). Requirements: - Plugin should have it's configuration icon in conversion dialogue an its own configuration widget. - There could be any count of these intermediate plugins. - [Low Priority.] End user should be able to change the order of plugins to be applied (in case they might work on same area and have influence on final result). It would be possible to add GUI plugin and do the same type of work or actually replicate all conversion dialogue and even to make it extensible, but it is probably not worth it.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
common news recipe for blogger type of feeds?	JayKindle	Recipes	0	10-08-2012 11:35 PM
Recipe using a plugin	wsuetholz	Recipes	1	12-28-2011 10:24 PM
Need advice for creating a slightly odd type of plugin	ldolse	Development	4	05-25-2011 11:31 AM
New Plugin Type Idea: Library Plugin	cgranade	Plugins	3	09-15-2010 12:11 PM
PDB to PMLZ File type plugin	pdurrant	Plugins	18	02-20-2010 02:19 PM

03-13-2013, 03:42 AM	#2
kovidgoyal creator of calibre Posts: 43,837 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I'm confused. Are you asking for plugins that can make changes during a conversion, that is, while the conversion pipeline is running? If all you want to do is make changes to epub and azw3 files in OEB form, just create a user interface action plugin and use the polish.container module to get the OEB representation.

03-13-2013, 04:03 AM	#3
SauliusP. Plugin developer Posts: 108 Karma: 24394 Join Date: Feb 2012 Location: Lithuania Device: Kindle	Well, yes, I imagine possible intermediate changes in conversion pipeline — right in the middle between Input and Output. If polish.container could "explode" AZW3 (and any other format) to OEB and then "implode" back with changes, that would be enough. Is it possible?

03-13-2013, 04:11 AM	#4
kovidgoyal creator of calibre Posts: 43,837 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It can explode and implode azw3 and epub.

03-13-2013, 04:30 AM	#5
SauliusP. Plugin developer Posts: 108 Karma: 24394 Join Date: Feb 2012 Location: Lithuania Device: Kindle	Any example, please?

03-13-2013, 04:34 AM	#6
kovidgoyal creator of calibre Posts: 43,837 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Just look at the source of container.py at the bottom is code to explode and implode any epub and azw3 file passed in on the command line.

03-13-2013, 04:48 AM	#7
SauliusP. Plugin developer Posts: 108 Karma: 24394 Join Date: Feb 2012 Location: Lithuania Device: Kindle	Allright, thank you. That seems to be a possibility. Will come back with questions here if any.

03-13-2013, 05:49 AM	#8
SauliusP. Plugin developer Posts: 108 Karma: 24394 Join Date: Feb 2012 Location: Lithuania Device: Kindle	Unfortunately, at first glance I don't understand how I could use anything from container.py. It seems to be specific to epub only. And epub itself is very close (if not the same) to OEB. I would like to make hyphenator as much format independent as possible. That's why I was thinking of intermediate conversion action. In case it is not possible to use something generic to any format, I will stick probably with epub and azw3, maybe mobi too.

03-13-2013, 05:54 AM	#9
kovidgoyal creator of calibre Posts: 43,837 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	As I said before, container.py works with both epub and azw3.

03-16-2013, 09:46 AM	#10
SauliusP. Plugin developer Posts: 108 Karma: 24394 Join Date: Feb 2012 Location: Lithuania Device: Kindle	Need further help. Hyphenator implemented and I have employed Container for both EPUB and AZW3. However, rebuilding AZW3 strips entities from text. They remain in EPUB (which, I saw, simply rebuilds zip). I believe, it is some "problem" in re-conversion in AZW3 container's commit method. I have tried to leave hyphens instead of and then they are present. But when I add entities, they disappear. Also, I have tried conversion from EPUB with these entities and AZW3 again does strip them. Digging through Calibre's code and settings I could not find, how to preserve these entities. Probably I am missing something simple, but... Maybe those entities are stripped on purpose, as only new Kindle firmware supports them? Please help!

03-16-2013, 12:28 PM	#11
kovidgoyal creator of calibre Posts: 43,837 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It's possible that hyphenation entites are stripped from azw3 as the Kindle doesn't support them I dont recall. You'll have to look through the code of the azw3 plugin to find out. It may also be that the entities are converted into their unicode equivalent code points.

Advert

Advert

03-16-2013, 12:40 PM	#12
SauliusP. Plugin developer Posts: 108 Karma: 24394 Join Date: Feb 2012 Location: Lithuania Device: Kindle	Entity with name or with character code is definitely stripped out. Will try direct unicode character too. But somehow doubt any of those would work.

03-18-2013, 03:04 AM	#13
SauliusP. Plugin developer Posts: 108 Karma: 24394 Join Date: Feb 2012 Location: Lithuania Device: Kindle	Problem with AZW3 hyphenation solved. However, after rebuilding AZW3 book (with "tweak book" native feature too) Kindle does not show TOC and does not allow navigation through chapters with four-way controller (on Keyboard and 4 NT). Something's gone missing after rebuild.

03-18-2013, 03:18 AM	#14
kovidgoyal creator of calibre Posts: 43,837 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	If you're asking for my help you're going to have to provide more info than that. The azw3 rebuild process uses the same code as the azw3 output conversion process and the ToC most definitely works on azw3 files generated by the conversion process. At a minimum an azw3 file whose toc works before tweaking but not after tweaking.

03-18-2013, 04:12 AM	#15
SauliusP. Plugin developer Posts: 108 Karma: 24394 Join Date: Feb 2012 Location: Lithuania Device: Kindle	Hi Kovid, Please, help :-) Kindly download books from here. I have included two books. English original and processed through my hyphenation plugin. Lithuanian original, hyphenated and also original tweaked via included "tweak ebook" menu. "Originals" both are converted from EPUB and show ToC and chapters fine. Processed and tweaked do not. Device is Kindle 4 non-touch with latest firmware. Tried with version 0.9.22 of Calibre. I know, that ToC and chapters work with AZW3, it is my primary used format. But have never tried tweaking before. I have also tried to explode original and modified books, OEB contains "toc.ncx" and no other obvious issues.