How to make Sigil live (in calibre)

[kovidgoyal]

I see lots of concern over a non-issue here. As I said before this tool will be a replacement (upgrade) for the Tweak Book functionality in calibre, it will not use code from the conversion module, but instead from the Polish Books module. That code is designed to make minimal changes to the markup/css, unlike conversions. Start up calibre and run Polish Books on a couple of epubs and see what happens.

And the tool will be launch-able as a standalone program without the need to use the library part of calibre, just like all the other tools in calibre. For example, ebook-convert.exe, ebook-viewer.exe, lrfviewer.exe, ebook-polish.exe, fetch-metadata.exe, ebook-meta.exe

And let me just say, it's obvious after all this time, from the fate of this thread and the one before it, that that there is no interest in maintaining the current Sigil codebase. Therefore, while I appreciate that some people want to hang on to Sigil or the concept of "separate" programs or whatever, the choice at this point is simple: Either we go with my offer, or the ability to conveniently tweak ebooks in EPUB/AZW3 using open tools, in a cross platform form manner, stagnates and eventually dies. While that's not a big deal to me personally, as I dont use Sigil anyway, I still think it is important, thus the offer.

[JSWolf]

I have a feature request...

Be able to take MS Reader OPF and reformat it it to be ePub compliant. That was one can load in an MS Reader file and easily convert it to ePub since it already is mostly there.

[Perkin]

Originally Posted by kovidgoyal

I see lots of concern over a non-issue here. As I said before this tool will be a replacement (upgrade) for the Tweak Book functionality in calibre, it will not use code from the conversion module, but instead from the Polish Books module. That code is designed to make minimal changes to the markup/css, unlike conversions. Start up calibre and run Polish Books on a couple of epubs and see what happens.

How will calibre get from the 'raw' html & resources to epub - it has to convert it - thus causing the styling and markup changes.
Sigil doesn't.

[Toxaris]

You misunderstand Kovid I think. You don't load the raw html in Calibre, but in the seperate program.

[kovidgoyal]

There is no raw html. This is an ebook editor, in other words you give it an ebook already in epub or azw3. And adding an import tool to process html files into epub with no changes is trivial calibre already does that, try adding an html file to calibre you will get a zip, with all linked files, images, styles and an autogenerated opf with no changes made anywhere (except for link rewriting). Just rename that zip to .epub and add in the boilerplate META-INF/content.xml and mimetype files and there you go -- an epub from html.

@JSWolf: In fact the above approach can be generalized to *all* ebook formats calibre can handle, including LIT.

[Perkin]

Originally Posted by kovidgoyal

There is no raw html. This is an ebook editor, in other words you give it an ebook already in epub or azw3. And adding an import tool to process html files into epub with no changes is trivial calibre already does that, try adding an html file to calibre you will get a zip, with all linked files, images, styles and an autogenerated opf with no changes made anywhere (except for link rewriting). Just rename that zip to .epub and add in the boilerplate META-INF/content.xml and mimetype files and there you go -- an epub from html.

Just tried and works.
created empty book
imported html - calibre zipped it and all resources
had to rezip (as *.epub) contents (with stored mimetype file)
the META-INF/container.xml file was already there
I then had to split and create toc in Sigil (as I would have done anyway)

So that seems to easily done and creates an epub as Sigil would have done (minus the sub-folder structure) without the usual calibre style/markup conversion alterations.

If this was streamlined and given as 'import as epub' when dropping the original html into calibre, that would be quite acceptable.

Then having the 'tweak-epub-sigil' to edit the epub would/should be as easy to use as sigil as it is now.

Edit: the content.opf includes and references a non-existent toc.ncx file

[Katsunami]

Originally Posted by Perkin

<snip: conversion and worries about messed up HTML and CSS>

Ah, *THAT* is the crux of the matter!

As Minsc in Baldur's Gate II would say:

"Hehe.... You... you're a smart one! I understand no-houw! Haha!"

No, that is not how it would work; not in my view at least. The editor would work the same way as Sigil does, creating an EPUB out of the HTML files, NOT converting from HTML to EPUB (with all the .calibre classes as a result).

Kovid beat me to the punch with his reply. It describes exactly how I see this editor come to be... and lo and behold, some, or maybe even most of the EPUB-creation can already be done, and the editor hasn't even been written yet.

How's that for fast development?

[quote]

Originally Posted by Perkin

Then having the 'tweak-epub-sigil' to edit the epub would/should be as easy to use as sigil as it is now.

Yes

Originally Posted by kovidgoyal

And let me just say, it's obvious after all this time, from the fate of this thread and the one before it, that that there is no interest in maintaining the current Sigil codebase. Therefore, while I appreciate that some people want to hang on to Sigil or the concept of "separate" programs or whatever, the choice at this point is simple: Either we go with my offer, or the ability to conveniently tweak ebooks in EPUB/AZW3 using open tools, in a cross platform form manner, stagnates and eventually dies. While that's not a big deal to me personally, as I dont use Sigil anyway, I still think it is important, thus the offer.

As one who doesn't do Calibre development (or any open-source development, yet), I wouldn't have dared to say it as bluntly as that, but it's basically true.

===

While I know C and C++ much better than Python at this point, I'm not going to program in C/C++ "for fun" if I need to get work done for which execution speed is not of paramount importance, but development speed and ease is.

If I want to write an editor, then I want to write editor functions, not spend half my time managing memory. (A chess engine would be different: speed is of greatest importance, and managing memory and bits and bytes is what it needs to do to get results, so C/C++ are perfect choices there.)

[Hitch]

Originally Posted by kovidgoyal

There is no raw html. This is an ebook editor, in other words you give it an ebook already in epub or azw3. And adding an import tool to process html files into epub with no changes is trivial calibre already does that, try adding an html file to calibre you will get a zip, with all linked files, images, styles and an autogenerated opf with no changes made anywhere (except for link rewriting). Just rename that zip to .epub and add in the boilerplate META-INF/content.xml and mimetype files and there you go -- an epub from html.

@JSWolf: In fact the above approach can be generalized to *all* ebook formats calibre can handle, including LIT.

And we're still waiting for an editor-building volunteer/donor to show up, is that where we're at?

Hitch

[JSWolf]

Originally Posted by Katsunami

There is.

I make epubs from scratch very seldom, but I fix about everything I buy in some way or another. Splitting and merging is not very important when fixing stuff, but it is when creating books from scratch.

Splitting/merging is very important. It's one of the things I do use Sigil for.

[JSWolf]

Originally Posted by Perkin

When fixing editing epubs, one of the most often uses of Sigil for me (apart from quick spelling mistake edits) is having the handy ability to split and merge, keeping track and updating all the file/link references without Sigil would be a nightmare.

When building my own epubs (personal use), I have the html source file and any images/css/fonts
Once I import a html file, and the resources linked in it, I usually run the 'Split at markers', then 'Generate ToC', Check the metadata, fix semantics, check the validity, then save the epub. Job done (usually).

If you do the same import in calibre, a lot of styling etc is flattened and you end up with loads of 'calibre###' styles - not as nice as the given style names if you then need to edit the epub at any point.

So even if in future Sigil is embedded in calibre I'd still want the existing Sigil (or it's abilities) so any simple html-epub conversions don't get calibre'd.

Very well said. I too don't want my styles converted. I want them with the names I've put in the CSS. I want Sigil in Calibre to work like stand-alone Sigil where when I put something in and it's correct, it stays how I've added it.

[JSWolf]

Originally Posted by Katsunami

Not ones that I bought, no. (And the classics from Feedbooks and B&N are also perfectly formatted and divided, except for some stuff that I change because it's not to my taste.) I've encountered many problems, but never bad splitting; not even once. With regard to epubs that weren't divided at all, I've never encountered them either. Not the ones I get from stores, that is.



Hm, ok. Have you seen this problem with a particular publisher? In that case, I may want to avoid them

When I clean up Discworld ePub, I do split/merge differently then the publisher does. Because there are no chapters, I split/merge such that each flow ends at a section break. The publisher doesn't do that in most cases.

[JSWolf]

Originally Posted by Hitch

Again, I'm not particularly trying to be negative, but Kovid's userbase has different goals than I do, myself. Many of the Calibre users are making books for themselves, or converting/cleaning PD books, or...well, other stuff. I'm not saying that what they're doing is "bad" or not as important, or any other spin; I'm simply saying, my prognostications as to the direction that a Sigil-add-in will take, for Calibre users, is likely to be different than the direction I'd go, assuming arguendo I had those type of resources. I mean, hell, I could be completely wrong, but....I'm pretty sure that somewhere around here, if I looked hard enough, I could find an archived post from me saying "sometime within 2 years, we'll be looking for a new maintainer..." and here we are. (NO reflection whatsoever on User_none, who has gone above and beyond the call of duty; Sigil is an amazing product today, compared to what it was when he took over. AH-MAZING.)

I prefer to do a lot of things by hand. Sigil allows me to do that or helps me do that. If I do convert something to ePub via Calibre, I do take it into Sigil and see what sort of cleaning up needs to be done based on what I prefer. I'm on your side that Sigil (inside of Calibre) would need to not do any sort of conversion that changes the code.

[JSWolf]

Originally Posted by kovidgoyal

I see lots of concern over a non-issue here. As I said before this tool will be a replacement (upgrade) for the Tweak Book functionality in calibre, it will not use code from the conversion module, but instead from the Polish Books module. That code is designed to make minimal changes to the markup/css, unlike conversions. Start up calibre and run Polish Books on a couple of epubs and see what happens.

I do use the font subsetting of Polish Books and I can say it works very well and does not do much changing. It does however change the CSS like this...

Original CSS

.p {
margin-top: 0;
margin-bottom: 0;
text-indent: 1.2em
}

New CSS (something like that)

.p {
    margin-top: 0;
    margin-bottom: 0;
   text-indent: 1.2em
   }

I do wish that would not happen and that the CSS would be left alone except for the styles that specifically need to be changed.

[kovidgoyal]

Originally Posted by Hitch

And we're still waiting for an editor-building volunteer/donor to show up, is that where we're at?

Hitch

Yes, although, I have already started building the shell that does the various tasks I spoke about earlier. It's already better than Sigil in one respect -- it has support for global undo/redo, so any operation you do can be undone, including file renames, global search and replace and so on. Not to mention that it works for both azw3 and epub2.

If no one volunteers to build the editor, I will probably just stick something very basic in myself and leave it at that. The new tool is a significant update to the existing Tweak Book feature in calibre, so it is worth doing from that perspective for me. The way it is shaping up, I may actually be persuaded to abandon vim for clean up tasks that dont involve lots of actual text editing. However, it is still early days, and with something this complex, there are no guarantees. I have been known to occasionally get bored and abandon a project part way through.

For people that want to check it out, setup a calibre development environment and run it with

calibre-debug src/calibre/gui2/tweak_book/main.py some_file.epub

Right now it only has a file browser, this is very early stages.

[st_albert]

Re: editor
At the risk of resurrecting a dead horse, I do recall that in the runup to Sigil v. 0.6.0, ckeditor was tried as a "previewer." As elibrarian indicated, it could also be used as an editor, though Sigil wasn't set up to use it that way. Eventually, ckeditor was dropped for 0.6.0 (IIRC) for reasons too deep for me.

But... perhaps if we're starting more or less from scratch, we could use ckeditor as a starting point? I speak of what I know not, so maybe there are fundamental problems I'm unaware of. (and "what's this 'we' s**t, white man?" anyway... )

BTW, as I remember it, the ckeditor interface in Sigil 0.5.x looked a lot like the present previewer. But I searched all my machines which have 0.7.2 and above installed, and can't find the ckeditor_config.js file so I suppose it (ckeditor) is not presently in use.