How to make Sigil live

[user_none]

Originally Posted by Katsunami

Could one...

Originally Posted by Katsunami

1. Use QT as the GUI framework using PySide or PyQT?

Yes. Sigil already uses Qt as the GUI.

Originally Posted by Katsunami

2. Use QScintilla (Scintilla's QT counterpart) as the editor?

Yes but I don't recommend it. I looked at using it but decided against it. It's cumbersome to work with. It also has poor regex support. You can hack it on but it's not very friendly. It doesn't support RTL languages. But mainly the API isn't nice. It would bring a lot of cool features to Sigil instantly but it may be more trouble than it's worth.

Originally Posted by Katsunami

3. Use Webkit as the browser?

Just like Sigil already does, yes, QWebKit via one of the above mentioned Python bindings.

Originally Posted by Katsunami

4. Port Sigil's EPUB capabilities into a library to use as a springboard to get the editor running as an EPUB-editor, instead of merely another SciTE clone?

Not really.

Originally Posted by Katsunami

5. Glue all of that stuff together using Python? (Cython, Swig, Ctype, etc...)

Again not really.

Originally Posted by Katsunami

Then we would have the very basics: a program that can open, save, edit and preview EPUB's, but it won't have any other stuff like TOC-editing, Metatag-editor and such, but those could be created one at a time.

You mean like BBEdit, Vim, probably others (Notepad++ I think)? So basically a text editor that doesn't do text editing as good as other things out there?

Originally Posted by Katsunami

How much would development time possibly be shortened?

Probably none. The advantage of Sigil are those things that you said could be added later. At that point you'd have Sigil Python which isn't as good a Sigil. That means no one will want to use it so no feed back, and no one interested in helping when Sigil is already better.

Originally Posted by Katsunami

For me, the greatest roadblock is the EPUB-stuff. I just don't know anything about it. I know how an EPUB looks like from the inside of course, but I don't know all the details that are necessary... that's why I use Sigil.

That's going to make it a lot harder...

Originally Posted by Katsunami

I for one *can* program in C and C++, but I won't do it in my spare time as well as for work (I mainly write stuff in C and sometimes C++ for microcontrollers.)

One of the things that demotivates me when I look at doing something at home using C is I have access to some very helpful libraries at work that make C development much easier. I don't want to come home and write a hashtable (for example) when I have a very powerful, full featured, and well tested one at work.

I'm not saying I don't like C or it's not useful. It is extremely useful and I enjoy working with it. I just don't like doing the same task over. Once for work then again at home. When it comes to raw performance it's hard to beat C.

Originally Posted by Katsunami

In the Netherlands, there's an entire generation of people coming out of school that haven't even touched C or C++. Many schools and universities have standardized on Java and/or C#.

The same trend is happening in the USA as well.

[mrmikel]

When it comes to raw performance it's hard to beat C.

The main reason I didn't use Sigil much in the beginning was that it was SOOOO slow. Now it is quite fast. It is not so fast, though, that I want it to work slower. When working with large files, it gets draggy, even with 4 cores and 6 gig of memory. Python is probably going to work more slowly.

Thank you for working on Sigil for such a long time after pounding on code all day at work. It is a fantastic tool and for me, a hobbyist, it does everything needed.

The only problem is fully discovering and using all its features.

[kovidgoyal]

@user_none: I fully understand the reasons you chose to not rewrite Sigil in python, I would probably have done the same if I were in your shoes. I was just saying, it's a pity

Regarding folding it into calibre, or alternatively, adding a GUI based ebook tweak tool to calibre, I dont imagine being able to directly use Sigil's code, it would need to be re-written. However, calibre has lots of ebook related code that can be re-used, for example, the ToC editor and the Polish Books code. This would make various bits of the re-implementation available for free and would have the bonus of supporting both epub and azw3. The idea would be to take inspiration and techniques from Sigil, not use the code directly.

I would be happy to help with such an effort, unfortunately, it's not something I am in a position to do alone, as my plate is pretty full with work that is a higher priority for calibre.

[Katsunami]

Originally Posted by user_none

Yes but I don't recommend it. I looked at using it but decided against it. It's cumbersome to work with. It also has poor regex support. You can hack it on but it's not very friendly. It doesn't support RTL languages. But mainly the API isn't nice. It would bring a lot of cool features to Sigil instantly but it may be more trouble than it's worth.

I see. It seems Scintilla is omnipresent. I have been unable to find another text editor component that even comes close to Scintilla's features. What component does Sigil use; or did you write it yourself?

Not really.

Is this something that is not possible, because the editor itself has to "know" EPUB?

Again not really.

Why would you recommend against this approach?

You mean like BBEdit, Vim, probably others (Notepad++ I think)? So basically a text editor that doesn't do text editing as good as other things out there?

No, what I meant was to write a text editor based on a well-known text editing component, but with the addition of being able to open and save EPUB files. Then, Sigil's features could be implemented one by one.

If it would be possible to put Sigil's EPUB capability into a library (so, to open an EPUB, hand out an object with everything in it, and also save an EPUB if the library receives such an object back again), then it may be possible to fork an existing editor that already has a plugin structure, tack the EPUB library beneath it (coupled to its Open and Save functions), and then write the other needed parts as plugins.

Even if that other editor is written in something like C++, it would save on maintaining the editor itself.

I'm just brainstorming here...

That's going to make it a lot harder...

I know. Where did you acquire the knowledge to program this? "Just read the EPUB spec" wouldn't cut it, I think. Of course I can implement specifications, but I have never done anything as large as EPUB from scratch and on my own.

One of the things that demotivates me when I look at doing something at home using C is I have access to some very helpful libraries at work that make C development much easier. I don't want to come home and write a hashtable (for example) when I have a very powerful, full featured, and well tested one at work.

Agreed. C++, and certainly C on their own are just languages, with relatively small libraries. To do anything meaningful without spending 50 years to do it requires extra libraries. Learning a new language is not the problem nowadays. That's a work a few days at most, and some experimenting. It's all the gazillion libraries and their workings.

And on speed and compactness... that's the reason why it's still used a lot for microcontrollers. When programming a chip that runs at 10-25 MHz and has 32-512 kB of RAM, C is still first choice most of the time.

[Katsunami]

Originally Posted by kovidgoyal

However, calibre has lots of ebook related code that can be re-used, for example, the ToC editor and the Polish Books code. This would make various bits of the re-implementation available for free and would have the bonus of supporting both epub and azw3. The idea would be to take inspiration and techniques from Sigil, not use the code directly.

Calibre itself is of course Metadata King, and together with Polish Books (I still read that like "books from Poland" half of the time ) it would replace Sigil's Metadata Editor. You have a ToC editor as well. These two parts of Sigil are the ones I personally use most.

How much work would you think it to be, if Calibre was extended with an EPUB/AZW3 file explorer and a code editor, based on some already existing component?

[user_none]

Originally Posted by Katsunami

I see. It seems Scintilla is omnipresent. I have been unable to find another text editor component that even comes close to Scintilla's features. What component does Sigil use; or did you write it yourself?

Scintilla is indeed widely used. That doesn't make it any easier to work with. It just means there isn't much else out there for reusbable editing widgets. Most such components are project specific.

The one Sigil uses is a subclass of QPlainTextEdit with a lot of functionality added. Look qt QtCreator. It uses the same technique for their editor. There really isn't any reason it can't be as powerful as Scintilla. It just isn't there yet.

Originally Posted by Katsunami

Is this something that is not possible, because the editor itself has to "know" EPUB?

The editor itself doesn't have to know about it but Sigil was never designed to be used as library. It does not have an API to broker editing operations. A lot of the operations rely on the GUI components. Some of the information about the EPUB is stored in GUI widgets and pulled out of it when needed. It's not all in one nice and easy to access place that can be decoupled from the GUI.

Originally Posted by Katsunami

Why would you recommend against this approach?

The approach is fine. I'm saying I wouldn't recommend trying. You'd be better off starting over and looking at how Sigil deals with certain cases then coding that then trying to turn Sigil into a library.

Originally Posted by Katsunami

No, what I meant was to write a text editor based on a well-known text editing component, but with the addition of being able to open and save EPUB files. Then, Sigil's features could be implemented one by one.

All of the editors I mentioned can open and save epubs. They lack the advanced editing components such as a metadata editor but the basics and the starting point you've mentioned they already do.

I just feel that the focus should be on differentiating a rewrite from other editors. Focus on what those editors that can open and save epubs don't offer. If your goal is create an editor that can't do with Sigil already does and what other editors already do then worry at that time start thinking about how to add the epub features you're taking the wrong approach.

Originally Posted by Katsunami

If it would be possible to put Sigil's EPUB capability into a library (so, to open an EPUB, hand out an object with everything in it, and also save an EPUB if the library receives such an object back again), then it may be possible to fork an existing editor that already has a plugin structure, tack the EPUB library beneath it (coupled to its Open and Save functions), and then write the other needed parts as plugins.

If it were easy to make Sigil into a library then this approach would work. But as I said Sigil was never designed to be library and you can't decouple most operations from the GUI.

Originally Posted by Katsunami

I know. Where did you acquire the knowledge to program this? "Just read the EPUB spec" wouldn't cut it, I think. Of course I can implement specifications, but I have never done anything as large as EPUB from scratch and on my own.

Honestly, reading the spec. It's all there. That said reading the spec doesn't mean you need to read all of it. Read about how an epub is structured. Implement opening. Look at the parts that deal with content but only html. Implement it. Read the part about images. Implement. Repeat.

Once you have an over view of the format focus on the parts you actually need to care about when you need to worry about them. This does mean you'll end up having to recode when you read something later on that you didn't account for but you'll save in the long run by not spending time designing around features you'll never implement.

Sigil doesn't touch and completely ignores parts of the epub spec. Other than a basic over view of those parts I don't know implementation details.

Finally, understanding the epub format has a lot to do with calibre. I've implemented conversion for a number formats. Calibre's conversion pipe line is in -> OEB (basically the guts of an epub) -> out. That has helped a lot.

Originally Posted by Katsunami

Agreed. C++, and certainly C on their own are just languages, with relatively small libraries. To do anything meaningful without spending 50 years to do it requires extra libraries. Learning a new language is not the problem nowadays. That's a work a few days at most, and some experimenting. It's all the gazillion libraries and their workings.

This is one of the advantages of languages like Python. They come with a standard library. Typically if the standard library supports something that's what you learn and use. There is little need to have to choose between multiple libraries that do the same thing.

The Python community even takes this even further. XML libraries like lxml actually use the same API as the XML API that comes with Python. So transitioning between the two is pretty much seamless.

Originally Posted by Katsunami

How much work would you think it to be, if Calibre was extended with an EPUB/AZW3 file explorer and a code editor, based on some already existing component?

Shouldn't be long. Calibre's book object will have a list of files. Show the list. When an item is selected load it into an editor. When the editor loses focus replace the file's contents with the data from the editor.

As for the editor you'll need a good editor. I was looking at this and it would work with minimal tweaking. It uses the subclassed QPlainTextEdit concept I spoke about.

Look at this, this, and this blog post I wrote for examples of extending and working with QPlainTextEdit. These concepts are the basis of what Sigil, QtCreator and Spyderlib do to provide their editors.

Hooking in the regex stuff would be the hardest part but python has very strong regex (search and replace) support. You'd essentially get the data from the editor, run the search, get the offsets and highlight that area of the editor.

[Katsunami]

Originally Posted by user_none

Some of the information about the EPUB is stored in GUI widgets and pulled out of it when needed. It's not all in one nice and easy to access place that can be decoupled from the GUI.

I hope the original designer of Sigil had a very good reason to do it like that. Personally, I would never have intertwined Sgil's GUI and business logic.

(Even if it wasn't for the fact that, when I was in university, it would almost certainly lower your mark by two points or so if you did something like that without a *very* good reason...)

OK... in that case, turning Sigil into an EPUB library is out. It would indeed be much better to just look at the code, and rewrite it in Python.

All of the editors I mentioned can open and save epubs. They lack the advanced editing components such as a metadata editor but the basics and the starting point you've mentioned they already do.

OK, thanks. I did not know that.

<spec>

Meh. No magic bullet there. It's the same as I implement specs for microcontrollers, only a lot bigger (and probably more ambiguous too, with things like "may have the option to... but does not require to...").

Finally, understanding the epub format has a lot to do with calibre. I've implemented conversion for a number formats. Calibre's conversion pipe line is in -> OEB (basically the guts of an epub) -> out. That has helped a lot.

I can imagine.

<recommendations>

Thanks, I will certainly look at that. I've set up a basic Python development environment to test some stuff. I just picked 3.3.2 (latest) for some eperimentation.

What version of Python does Calibre use at this moment? There is some (very small) stuff I may want to add to one of Kiwidude's plugins. I think it's a good exercise to get going with Python and take a first look at how Calibre actually works, but of course then I must use the same Python version as Calibre does.

[DiapDealer]

Originally Posted by user_none

Hooking in the regex stuff would be the hardest part but python has very strong regex (search and replace) support. You'd essentially get the data from the editor, run the search, get the offsets and highlight that area of the editor.

Python's standard re module leaves a lot to be desired with regard to non-ASCII text. Matthew Barnett's regex module (available from PyPi) does a much better job (and will hopefully replace re eventually) and is very PCRE-ish. But even then--as much as I know traditional Python fans might not like hearing it--I probably wouldn't start a new python, cross-platform epub editor project (that will be dealing extensively with unicode) with Python 2.x.x. I'd begin with Python 3.3 right from the get-go.

From the Python 3.3 release-notes:

Changes introduced by PEP 393 are the following:

Python now always supports the full range of Unicode codepoints, including non-BMP ones (i.e. from U+0000 to U+10FFFF). The distinction between narrow and wide builds no longer exists and Python now behaves like a wide build, even under Windows.
With the death of narrow builds, the problems specific to narrow builds have also been fixed, for example:
len() now always returns 1 for non-BMP characters, so len('\U0010FFFF') == 1;
surrogate pairs are not recombined in string literals, so '\uDBFF\uDFFF' != '\U0010FFFF';
indexing or slicing non-BMP characters returns the expected value, so '\U0010FFFF'[0] now returns '\U0010FFFF' and not '\uDBFF';
all other functions in the standard library now correctly handle non-BMP codepoints.
The value of sys.maxunicode is now always 1114111 (0x10FFFF in hexadecimal). The PyUnicode_GetMax() function still returns either 0xFFFF or 0x10FFFF for backward compatibility, and it should not be used with the new Unicode API (see issue 13054).
The ./configure flag --with-wide-unicode has been removed.

Concerning Barnett's regex module:

The regex library therefore much more closely follows the (current) recommendations of UTS#18: Unicode Regular Expressions in how it approaches things. It meets or exceeds the UTS#18 Level 1 requirements in most if not all regards, something you normally have to use the ICU regex library or Perl itself for — or if you are especially courageous, the new Java 7 update to its regexes, as that also conforms to the Level One requirements from UTS#18.

Beyond meeting those Level One requirements, which are all absolutely essential for basic Unicode support but which are not met by Python’s current re library, the regex library also meets the Level Two requirements for RL2.5 Named Characters (\N{...})), RL2.2 Extended Grapheme Clusters (\X), and the new RL2.7 on Full Properties from revision 14 of UTS#18.

Matthew’s regex module also does Unicode casefolding so that case insensitive matches work reliably on Unicode, which re does not.

But that's just me.

[user_none]

Originally Posted by Katsunami

I hope the original designer of Sigil had a very good reason to do it like that. Personally, I would never have intertwined Sgil's GUI and business logic.

The GUI is the business logic. It was designed as an application not as a GUI with a library. This is how most GUI applications are designed. Tight coupling between the GUI and business logic.

Don't forget the GUI need to show the data elements. The GUI needs to hold those elements. You can have them stored in a library object but then you need to duplicate them for the GUI to display them because the GUI needs them in a format understandable by the GUI. So you end up with two options.

1) You have two sets of the same data that need to be kept in sync. Make a change to the GUI and make sure you change the lib version. Make an API change using the lib then you have to ensure the GUI gets updated. In this case how do you inform the GUI the elements changes? You need to signal to the GUI that that happened. Now you have a whole lot more logic to deal with.

2) Put the data into the GUI format (model and view based not necessarily owned by the GUI). Then you only have one copy to deal with and you don't need extra logic for syncing. You could have a library that uses the GUI model for the data but you're in the same boat because it's coupled to the GUI framework and not really a library (in the sense of a decoupled reusable component) at that point.

This is a basic trade off in GUI programming of dealing with time and effort to get something done. What it really comes down to is why spend the extra time and effort on something that you're design isn't supposed to ever need to do.

Originally Posted by Katsunami

(Even if it wasn't for the fact that, when I was in university, it would almost certainly lower your mark by two points or so if you did something like that without a *very* good reason...)

Have you seen how the C# classes at university are taught? GUI builders where the business logic goes into the GUI event handlers.

Originally Posted by Katsunami

What version of Python does Calibre use at this moment? There is some (very small) stuff I may want to add to one of Kiwidude's plugins. I think it's a good exercise to get going with Python and take a first look at how Calibre actually works, but of course then I must use the same Python version as Calibre does.

Calibre uses and ships with 2.7. This is mainly because of the effort involved in porting all of calibre to Python 3. New code however has all Python 3 compatibility features enabled so it's pretty close to what you'd write for Python 3.

Originally Posted by DiapDealer

Python's standard re module leaves a lot to be desired with regard to non-ASCII text

Good to know. I don't deal with non-ASCII myself so it's never been an issue for me.

That said Sigil doesn't use Qt's regex for user level searches anyway. I wrote a custom bindings for QString to libPCRE and all the replacement code in Sigil because I found QRegExp lacking. Qt5's new PCRE based replacement is still lacking the replace options mine supports. So, that doesn't necessarily negate Python. A different regex library like the one you mentioned could be used in place of Pythons standard re.

[kovidgoyal]

The hardest part of writing a sigil replacement inside calibre would be the editor component, especially if you want both a code editor and a WYSIWYG editor.

The rest: editing a list of files, editing metadata, editing ToC, renaming/moving around files would be trivial.

So to move this conversation along, here's a concrete offer. I offer to implement the *rest* part of it -- list of files, edit metadata, edit tocs, renaming/moving files, editing the spine, various automated actions from the Polish Books component -- if someone will volunteer to tackle the editor component.

I can set up a basic GUI shell that does most of that in a few weeks, and then let whoever volunteers work on the editor component. But, I will only spend the time to do that if I get a firm commitment from someone to handle the editor part. That committment will need to include ongoing support for fixing bugs in the editor component.

[Toxaris]

That is a very generous offer Kovid! It makes me want to try out Python again to be honest. I tried it, but lacked a target.
However, I first need to finish the programs I am currently working on first and I would do not have the time to do big contributions, but for small ones I would be fine.

[jbacelar]

Ah Kovid! you are great.

And... an idea.
Who needs a WYSIWYG editor?
With the code editor and the preview window, I think that's enough.

[user_none]

Originally Posted by jbacelar

Who needs a WYSIWYG editor?
With the code editor and the preview window, I think that's enough.

I would agree with this statement. The WYSIWYG part is going to greatly complicate development.

[Katsunami]

Originally Posted by user_none

1) You have two sets of the same data that need to be kept in sync. Make a change to the GUI and make sure you change the lib version. Make an API change using the lib then you have to ensure the GUI gets updated. In this case how do you inform the GUI the elements changes? You need to signal to the GUI that that happened. Now you have a whole lot more logic to deal with.

<snip>

Have you seen how the C# classes at university are taught? GUI builders where the business logic goes into the GUI event handlers.

I've been taught option 1. It was strictly forbidden to put business logic into the GUI classes or event handlers. If that is now the default modus operandi, then universities have done a 180 degree turn.

Calibre uses and ships with 2.7. This is mainly because of the effort involved in porting all of calibre to Python 3. New code however has all Python 3 compatibility features enabled so it's pretty close to what you'd write for Python 3.

Thanks, in that case I'll revert to Python 2.7 for now. Maybe I can add a small wanted function to the Count Pages plugin; for someone who knows Calibre and Python already, it'd probably be like 15 minutes of work

[theducks]

Originally Posted by jbacelar

Ah Kovid! you are great.

And... an idea.
Who needs a WYSIWYG editor?
With the code editor and the preview window, I think that's enough.

I only agree 75% with this.

Two of the GREAT benefits of Sigil:
1) Text location Synchronization from the GUI (BV) to nearby in CV
2) Simple BV edits Split, Join (sorta works in 7.x. Code cleanup needed afterwards.) and simple character edits (words that may be spelled correct, but are the wrong word in context. Auto-incorrect ). Anything complex, I switch to CV where I mostly do work(need to see what part of the CSS to fix).