Sigil's Future

[crutledge]

I guess that I'm not the only Chicken Little. I'm afraid that Sigil, as it stands at this moment, is as far as it will go.

I will continue to use it and doing the cleanup on the output.

RIP

[Valloric]

Originally Posted by crutledge

I may be "whistling by the graveyard" but I have become concerned about the future of Sigil. I would really hate to see it go the way of Book Designer.

It would be difficult for Sigil to go the way of BD. BD was not open source, and no one could continue BD development after the original developer left.

Sigil is OSS, and that guarantees at least the option of someone else taking up the reins (and someone else will). Nothing of the sort was possible with BD. I cannot overstate the importance of this distinction.

Originally Posted by crutledge

Without a central clearing house for enhancements, additions, and changes, versions of Sigil could proliferate bringing chaos and failure.

The end is not quite nigh, so let's ease up a bit on the doom-and-gloom.

Originally Posted by kiwidude

I think people are right to be concerned about Sigil's future IMHO. There are a lot of good things about it which is why it has been a welcome tool, but it has some really serious issues. For a start, it is and has always been very buggy. I don't mean minor annoyance bugs of which there are many, I mean the most dangerous kind of bugs that lose book content or ignore changes. These made me lose any confidence in the software.

"Buggy"... it depends on your POV. "Very buggy"... I think that's going a bit too far, but that's just IMHO. I'm well aware that other people's experiences with Sigil have been different from my own. Personally, I pretty much never encounter bugs in Sigil. This is selection bias, I know: when I do find a bug in Sigil, I fix it immediately. So my workflow is extremely bug-free. I realize regular users don't have this option.

I'm not going to deny that there haven't been data-loosing bugs. There have been. Still, any major bugs in release versions are usually fixed fairly quickly. Major bugs in the betas and RC's are, well, in the betas and RC's. It's not an official release for a reason.

Major bugs can't magically disappear over night. Someone has to notice them, report them and fix them. Step one is easy. Step three is harder, but it seems that step two is the hardest one. If I've said it once, I've said it a million times: I can't fix a bug that hasn't been reported. And the report has to be good, i.e. reproducible. Otherwise I'm shooting ducks in the dark.

Originally Posted by kiwidude

I haven't used 0.4 yet, which Valloric himself has admitted is "buggy as hell" on these forums.

0.4 is at RC1. It's called pre-release software for a reason.

Originally Posted by kiwidude

Of course we hope he can do something about the most serious ones before he leaves, but that is a legacy I don't want to take on. Unfortunately many of these bugs are of the extremely intermittent difficult to repeat kind, which are horrible to try to fix without intimate knowledge of the code.

The key data-loss bug in Sigil pre-0.4 is what people have called "the pink screen of death". You know, the one where you get a big pink square at top of your Book View display that tells you shit has happened. This and other problems related to the same root cause have been fixed in 0.4 with a dialog that informs users of well-formed errors in the Code View when you make a mistake. So a whole slew of data-loss bugs has been fixed. Frankly, I don't know of any major data-loss bug left (AFAIK).

0.4 has brought its own set of bugs, but those will be ironed out before the first "real" release.

Originally Posted by kiwidude

My other main issue is the choice of C++ as a language for an oss application. It reduces your pool of potential developers as less are enthused to work on such an app in their free time.

This. This a billion times over. I can't tell you how many people I've met face-to-face that have said they'd love to work on Sigil, but it's written in C++ which they either don't know well enough or don't care to learn.

If I were starting Sigil today, I wouldn't do it in C++ for this and only this reason. C++ has justifiably been called an "experts-only" language. There's nothing more dangerous than someone who thinks they know C++ "well enough". It's not a language you can just "pick up". Well, it's a language you think you can pick up, but only after 10 years of daily use and professional betterment do you realize the depth of your ignorance. I've been using C++ for over 10 years, I've written many, many tens of thousands of lines of code in it (maybe even hundreds of thousands), I've read countless books on it and I still wouldn't call myself an expert in it. Far from it. Frankly, I don't think a C++ expert really exists (this is a sentiment that even C++ leaders have expressed on many occasions).

But we are where we are. Sigil is in C++ and there's no going around that.

Originally Posted by kiwidude

I didn't know Python before I started work on Calibre, but I had enough interest to learn it to do enough to contribute to that. It has a far lower barrier to entry for temporary contributors. I am not saying that I in any way like developing in python btw or that it should have been Vallorics choice, just that as a combination for Calibre it worked.

It goes without saying, Python is the easiest programming language to pick up that I know of (and I know a few). I've been teaching my 13-year-old brother programming using Python. He's smart, and learns fast. Python makes it effortless to teach CS concepts. There's a reason why MIT switched to Python for their introductory programming courses.

OTOH, Python comes with it's own bag of drawbacks, not the least of which is the poor run-time performance.

But let's not go into this discussion; it's the type of discussion that never ends.

Originally Posted by kiwidude

The other thorn for contribution is the challenge of cross platform. Apart from Qt just being filth to work with

Qt is absolutely awesome. It's like .NET for C++. On the other hand, QtWebKit, which is one module of many in the Qt Framework, is a smoldering piece of canine excrement.

Originally Posted by kiwidude

how many developers out there have the resources of a Mac as well as a Linux and windows boxes, plus the will to test everything three times?

This is exaggerated. Qt abstracts platform differences well enough that you very rarely have to venture off your dev platform to make sure that everything works on other platforms. As someone who has been working on a popular cross-platform application for the last two and a half years, I think I can say this with some authority.

Originally Posted by kiwidude

Personally I have given passing thought a few times to writing an alternative

Making an epub editor is harder than it looks. Trust me.

Originally Posted by kiwidude

as Sigil development had gone feature wise in a direction of interest to Valloric as is entirely his right, but that has continued to delay with his limited dev time a stable release of ncx mgmt and of course still no spell checking. For my own needs I could care less about flight check stuff etc, as I just want something to split and merge, regex and wysiwyg spell checking.

Every user has his own set of pet features. The problem is that everyone has a different set of pet features.

I have yet to grow a third arm. I've been concentrating really hard on making it happen, but the bitch just won't come out.

Originally Posted by crutledge

I guess that I'm not the only Chicken Little. I'm afraid that Sigil, as it stands at this moment, is as far as it will go.

RIP

I'll repeat what I said previously, "The end is not quite nigh, so let's ease up a bit on the doom-and-gloom. ".

[yekim54]

Originally Posted by Valloric

I'll repeat what I said previously, "The end is not quite nigh, so let's ease up a bit on the doom-and-gloom.".

Thanks for your commentary ... that made my day.

[crutledge]

Originally Posted by Valloric

I'll repeat what I said previously, "The end is not quite nigh, so let's ease up a bit on the doom-and-gloom. ".

Praise the Lord and pass the de-bugger.

I feel better already.

[rakulos]

When I think of C++ code It allways reminds me of the following;

//When I wrote this, only God and I understood what I was doing
//Now, God only knows

(I think it's a Karl Weierstrass quote)

I only tried C++ for a few months and came away feeling stupid!

[kiwidude]

Originally Posted by Valloric

"Buggy"... it depends on your POV. "Very buggy"... I think that's going a bit too far, but that's just IMHO. I'm well aware that other people's experiences with Sigil have been different from my own. Personally, I pretty much never encounter bugs in Sigil. This is selection bias, I know: when I do find a bug in Sigil, I fix it immediately. So my workflow is extremely bug-free. I realize regular users don't have this option.

You are right, "very" is too strong a word if thinking in terms of quantity of bugs or in relation to other applications. The emphasis was more from frustration at their nature. Spending an hour fine tuning a document only to have it irreperably corrupted to start again has happened so many times to me that it corrupts my view as "very buggy" when in fact that particular issue is quite possibly just "one" bug. But for me it is a showstopper. Of course there are othrs that I have seen others report too, but once you lose faith that your very next action may corrupt your book it all pales into significance really.

I'm not going to deny that there haven't been data-loosing bugs. There have been. Still, any major bugs in release versions are usually fixed fairly quickly. Major bugs in the betas and RC's are, well, in the betas and RC's. It's not an official release for a reason.

And I don't use betas or pre-release candidates for exactly that reason, no argument there. As I said above, I havent used any of the 0.4 versions yet.

Major bugs can't magically disappear over night. Someone has to notice them, report them and fix them. Step one is easy. Step three is harder, but it seems that step two is the hardest one. If I've said it once, I've said it a million times: I can't fix a bug that hasn't been reported. And the report has to be good, i.e. reproducible. Otherwise I'm shooting ducks in the dark.

Sadly the bugs I experience are extremely intermittent. I can't consistently reproduce them, without prolonged usage of Sigil. For instance the merging of book files - this is my #1 problem area. You can edit for an hour, save, try a merge and have that whole merge source page disappear off the planet. Close and reopen Sigil, try the merge again and it works.

0.4 is at RC1. It's called pre-release software for a reason.

Again, I said I wasn't using this yet - for precisely this "reason". IIRC people have reported that splitting files is still causing issues of files in the wrong order. Splitting and merging are my #1 usage of Sigil. Until this is fixed 100% I won't be touching 0.4 until it is officially released and others have verified this is dealt with.

Qt is absolutely awesome. It's like .NET for C++. On the other hand, QtWebKit, which is one module of many in the Qt Framework, is a smoldering piece of canine excrement.

I think we will agree to disagree on Qt being like .NET in any way, shape or form, but that like discussions of programming languages is a discussion best left for elsewhere. Clearly you like it, clearly I don't. Can I offer a cross platform alternative? No. That doesn't mean I have to like it. Let's leave it at that

This is exaggerated. Qt abstracts platform differences well enough that you very rarely have to venture off your dev platform to make sure that everything works on other platforms. As someone who has been working on a popular cross-platform application for the last two and a half years, I think I can say this with some authority.

Then you have been lucky, as I can just point to Calibre and negate this point. There are repeated posts from users saying the application does not behave or look like other Apple applications and demanding change. Linux users posting that certain file dialogs don't work on certain flavours of Linux. If I searched the Calibre codebase (or Kovid commented) I am sure there would be numerous examples of workarounds for certain platforms. You only have to hit one such case and you are in trouble without a box to test it on.

Making an epub editor is harder than it looks. Trust me.

Oh, I don't doubt that at all, which is why I said it was only a "passing" thought. I just don't have the time or enough desperation to justify the effort. But for a few missing features and bugs Sigil is the perfect answer for me - so close and yet so far...

I have yet to grow a third arm. I've been concentrating really hard on making it happen, but the bitch just won't come out.

Indeed - that was not a criticism of the amount of work you have put into Sigil, just a statement of the reality. Sigil has been Valloric and nothing but Valloric. If Valloric not available, Sigil not changed. It has been the harsh reality.

I don't blame you in any way for wanting to step away, it takes a certain type of developer to want to continue to maintain an app "forever". And like many I am grateful for your time and efforts to get it to this point, as we have had no real alternative. I wouldn't start a thread to raise my concerns, but since one existed, people asked why Calibre developers were not stepping in etc I merely offer my point of view. As a developer of 30-odd years, who has worked on Calibre for 8 months plus other open source projects, and who has locally patched/looked at the Sigil code I had perhaps a different perspective than just users wondering if there will be another release.

As I said in my original post, I sincerely do hope that something gets worked out and Sigil rules on...

[eping]

Could Sigil go commercial? or a branch to commercial?
such as enterprise edition, redesigned for publishers?

[Valloric]

Originally Posted by kiwidude

<snip>

I wrote a rather long response to many of your points, but then I accidentally pushed CTRL+R and refreshed the page, loosing my text. I'm too lazy to write it all out again.

Let's just say I agree with some of your points and disagree with others.

I'll rewrite one important section below:

Originally Posted by kiwidude

Indeed - that was not a criticism of the amount of work you have put into Sigil, just a statement of the reality. Sigil has been Valloric and nothing but Valloric. If Valloric not available, Sigil not changed. It has been the harsh reality.

This is painfully true, and not what I wanted or expected to happen. When I started Sigil, I figured others would want to join in fairly quickly. From what I gather, this didn't happen for several of reasons:
1. I already talked about the effects of C++ on potential developers. That's covered.
2. One long-time OSS developer brought out another interesting point in an email. He said I fixed things "too quickly". When someone noticed a major bug or a key missing feature, I'd hop on it. Development was fast and constant for a one-man project. Too fast and constant, he said. Devs usually join when they want something taken care of and the current devs can't/don't want to.

I found that intriguing. I can't say I agree with it entirely, but it's a perspective worthy of analysis. If true, then me leaving will have a positive effect on outside contributions.

Hah. Let's hope.

[mrmikel]

Myself personally, I have not experienced a crash for so long I can't remember when it was. But merging and splitting are not my big activities.

Kiwidude, many of your complaints seem like they can only be resolved by a complete rewrite in another programming language. Since Valloric, the pilot, has to repair the plane in mid-air, so to speak, that does not seem feasible.

Seems like I recall that calibre has had its interface issues as well.

Valloric started this out as a college project in C++ for reasons that were valid at the time...I guess they were since he turned out good enough to be hired for his programming skills. But that choice has limited the ability of others to help him with Sigil as well as his choice of programs that will interface to provide other functions like spell-check. The question is what to do now.

We Sigil users don't quite reach the critical mass to move forward as Calibre did because the program has been more narrowly aimed than Calibre. Calibre is a chef's knife rather than the Swiss Army knife that Calibre is. Speaking purely for myself, if Sigil had a spell-check facility, it would meet all the simple demands I make of it.

[kiwidude]

@mrmikel - my issues haven't been that Sigil crashes particularly often - it is stable in that regard which is a credit to Valloric. It is that it discards content or changes silently - which is the most dangerous kind of bug because you have to go looking to find out what you are now missing. This most frequently happens when using merge, but can also happen when using find/replace (where it does not actually do the replace which you only find out when you put focus back on the code view). I try to work around this now by saving before any merges and then double checking the content but every now and then you forget and get bitten badly.

Note also that it was not my intent to in any way turn this into a "Calibre is better because" type thread. I merely wished to indicate for any interested what attracted me to contribute to one project versus the other and it was the most recent example for me to use. Like any app ever written Calibre has pros and cons be it user interface based or technical. I just wished to share "from the outside" why I chose not to get more involved in the Sigil development, my perceived challenges for whoever does take it on and it's future as an OSS app. As always they are just my opinions which everyone is welcome to agree or disagree with

For myself, I would not care if Sigil became closed source. If Valloric in 2.5 years has struggled to get contributions, then both finding a maintainer / magicking up some contributors in a few months is a big ask without commercial involvement. Like mrmikel as I put in my first post here, if I had spell checking, the option of ncx mgmt and some bug fixes plus multi-select for merge/delete and that is all I want from Sigil - provided someone is maintaining the app I could care less whether it is closed or open source personally in this case.

[Valloric]

Originally Posted by kiwidude

@mrmikel - my issues haven't been that Sigil crashes particularly often - it is stable in that regard which is a credit to Valloric. It is that it discards content or changes silently - which is the most dangerous kind of bug because you have to go looking to find out what you are now missing.

The key dev goal of 0.4 was eliminating this. See issue 616 and all the issues it was blocked on (in the left sidebar).

Lots of other work went into 0.4 to make sure that Sigil makes fewer assumptions about how people want their data handled.

[EowynCarter]

Note also that it was not my intent to in any way turn this into a "Calibre is better because" type thread.

The only common point between calibre and sigil is than both are OSS.
For the rest, they fill absolutely distinct functions.

2. One long-time OSS developer brought out another interesting point in an email. He said I fixed things "too quickly". When someone noticed a major bug or a key missing feature, I'd hop on it. Development was fast and constant for a one-man project. Too fast and constant, he said. Devs usually join when they want something taken care of and the current devs can't/don't want to.

Well before seeing this, I was about to say : you fixed the bugs before they hit my nerves enough to think : "i'll do that myself".

[kiwidude]

Originally Posted by Valloric

2. One long-time OSS developer brought out another interesting point in an email. He said I fixed things "too quickly". When someone noticed a major bug or a key missing feature, I'd hop on it. Development was fast and constant for a one-man project. Too fast and constant, he said. Devs usually join when they want something taken care of and the current devs can't/don't want to.

I found that intriguing. I can't say I agree with it entirely, but it's a perspective worthy of analysis. If true, then me leaving will have a positive effect on outside contributions.

I know that similar exists with Calibre, in fact a thread commented on exactly this recently when someone said they wanted to get involved. It was pretty much suggested to them that it was pointless picking one of the easy issues up, as by the time they did so the fix would likely already be in the repo from the likes of Kovid or chaley. As developers there is something about low hanging fruit that is hard to resist.

Speaking for myself I am so irritated/embarassed in some cases when I leave a bug in a plugin that someone finds I want to redeem myself by fixing it immediately and have often had a release out within a few minutes of something being reported.

You also have the issue of time to review - unless you have 100% trust in the contributor it will frequently take longer to review and merge their patch than to just fix it yourself. Or sometimes it could be that it highlights some deeper bug/design issue that needs a lot more work than the obvious patch should be applied.

Undoubtedly there must have been an element of "well Valloric is still involved so he will fix it eventually so no need for me to bother". So perhaps Sigil will indeed become a "roll up your sleeves folks, as no-one else will do it for you" type app. Maybe. It's still C++ though...

Originally Posted by Valloric

The key dev goal of 0.4 was eliminating this. See issue 616 and all the issues it was blocked on (in the left sidebar).

Lots of other work went into 0.4 to make sure that Sigil makes fewer assumptions about how people want their data handled.

That is great to hear. I look forward to the official release

[LGN]

Hmm, suggestions about making a commercial Sigil I dislike. See, the original dev made the program 'big', does anyone realy think its good some commercial institution comes along, grabs all the code and change it, making it closed source and sell it? Besides GPL kinda makes that hard, I think it's a bit unfair.

I specially looked for an Open Source ePUB editor. Big publishers often have their inDesign workflow and don't give much about how the internals look of those converted epubs. Most then apply DRM to lock the ePUB.

And don't forget Sigil is mentioned in some (e)books about ebook publishing, some go as far as making a complete tutorial section.

Oh well, maybe some will just ignore ePUB v3, others will switch to indesign or whatever to do support ePUB v3. Others will just stick with the last stable Sigil. Others will go the unzip and some code editor route to make their ePUBs. And maybe a new project comes along starting all over again...

[charleski]

Originally Posted by kiwidude

Spending an hour fine tuning a document only to have it irreperably corrupted to start again has happened so many times to me that it corrupts my view as "very buggy" when in fact that particular issue is quite possibly just "one" bug. But for me it is a showstopper.

Ok, I've had that happen to me. The thing is, though, in every case, I could trace the fault back to me typing in mangled xhtml, particularly on global find/replaces, or doing something stupid like deleting the closing </style> tag in a header, etc. The 'bug' was behind the keyboard. This doesn't deny the fact that the data loss that happened in such cases was indeed very annoying. I think the only real bug that bothered me was the intermittent issue with find/replace commands being automatically reverted so you'd have to issue them twice.

Now, I think it's very valid to complain that Sigil should be more forgiving of users who type in mangled code, and it seems that the 0.4 release goes a long way to address that. Obviously, Sigil is at a stage where it needs to accommodate 'stupid users' who, like me, will occasionally forget to type a quotation mark or >. But this is more an issue of transferring functional demands from the user to the program than a bug per se.

Originally Posted by Valloric

When I started Sigil, I figured others would want to join in fairly quickly. From what I gather, this didn't happen for several of reasons:
1. I already talked about the effects of C++ on potential developers. That's covered.
2. One long-time OSS developer brought out another interesting point in an email. He said I fixed things "too quickly". When someone noticed a major bug or a key missing feature, I'd hop on it. Development was fast and constant for a one-man project. Too fast and constant, he said. Devs usually join when they want something taken care of and the current devs can't/don't want to.

There's also the issue that Sigil is your baby. You have a vision of how you want it to work and what you want it to do. This is, actually, largely related to the C++/python issue and the whole topic of plugins. A casual developer can write a plugin without delving into the main codebase and possibly corrupting code that already works. If the plugin's buggy, or it serves a function that the maintainer thinks is irrelevant, then that's fine - it's outside the main structure and users can utilise it or not as they choose.

But when a contributor has to alter the core of the code in order to implement a feature then you can (and should) worry about unintended effects on the overall design. The code looks well-modularised, so edits shouldn't actually break stuff outside the part that's being changed, but they can have an effect on overall functionality that might not be immediately manifest.

I've seen this happen before on an OSS project to which I contributed, where a new contributor joined and became very active - he produced a lot of very useful new code, but his design goals were clearly different to mine. After a few months he started rewriting modules I'd worked on and removing elements that I had inserted to satisfy my own design goals. At that point I quietly slipped away from the project - the new guy was doing a lot of good work (far more than me at that point) and it wasn't worth debating issues that were, at that stage, rather minor, but I could see that they had the potential to develop. If you have a good relationship with the other developers, then the obvious thing to do is discuss these issues and develop a common consensus, but if there is a one person who 'owns' the project by virtue of the amount of effort they spend on it, then you tend to be wary of stepping on his/her toes, which can often happen inadvertently.

[Phew, that was all rather long-winded.]

To tell the truth, I was happy with version 0.1.8, and used that for a long time. I only upgraded to 0.2 some time after it was released (I think it was the issue about global find/replace). And to be honest I've only just started trying the 0.4RC. Feature-bloat is definitely an issue. Take a look at Sigil's main (indeed, the only real) 'competitor' for ePub-editing, OxygenXML - it's a sprawling, massively-capable commercial program that has a load of complex functionality for editing and transforming XML and has advertised itself as an ePub editor for a while now. But only with the most recent version has it become able to do something as simple as find/replace code across all the files in an ePub. In terms of ePub editing, Sigil knocks this $400 program into a cocked hat, not because it has more features, but because it has the right ones.

There's something to be said for getting a core set of features working and then stopping at a particular point. I think Sigil could still develop a bit in terms of user-friendliness, and it will obviously need to accommodate ePub 3, but all this talk about commercial involvement makes me worry that a lean, focussed tool might turn into a sprawling giant.

[Hmm, long-winded again, sorry.]