New Sigil development blog

Valloric · 12-13-2009, 08:56 PM

I've been convinced (don't ask) that I should create a development blog for Sigil. Apparently it's a "great way to connect to one's users". We'll see.

Anyway, here it is: http://sigildev.blogspot.com/

It won't be high-volume, I'm guessing about a post a week, tops. It will vary depending whether I have the time and something interesting to say.

It will also be a place for posting any pertinent news, like for instance information about new releases. The release threads are hardly the way to go about it, since not everyone has an account on MobileRead (or is inclined to create one to post).

But the major reason for the blog is the ability to inform you about how the current release is progressing, what's being done (and why) and how it's being done. Tracking the live ChangeLog only goes so far. It also enables me to go into greater detail on certain changes/bug fixes/features or their design. For instance, the Find&Replace dialog went through several design iterations before I chose the final one. Wouldn't it be nice if you could have seen some screenshots? Maybe voted for one design over the other? Threads only go so far, and it's hard for new users to find/browse them when they drop off the radar.

In the end, this is an experiment. It may work and bring value to the users, or it may not. Here's to hoping it does.

kjk · 12-14-2009, 12:13 PM

Looks great so far, Valloric!

(plus, great blog layout-very "un-blogspotlike" in a good way.)

zelda_pinwheel · 12-14-2009, 12:17 PM

i think that's a great idea, valloric.

it's really interesting to see a bit of what's going on behind the scenes. thanks for taking the time to write about your work !

a thought, you might want to make this thread sticky, so people can find it easily.

Valloric · 12-14-2009, 12:58 PM

Quote:

Originally Posted by zelda_pinwheel

a thought, you might want to make this thread sticky, so people can find it easily.

I am planning to. When I created the thread I remembered you once told me that you think that new stickies are somehow less visible than regular new threads, and after thinking about it a bit, I tend to agree.

So I'm just going by your advice.

zelda_pinwheel · 12-14-2009, 01:02 PM

Quote:

Originally Posted by Valloric

I am planning to. When I created the thread I remembered you once told me that you think that new stickies are somehow less visible than regular new threads, and after thinking about it a bit, I tend to agree.

So I'm just going by your advice.

good to know.

i think regular readers tend to skim over the stickies area because they have already seen them. i could be wrong about that but it seems logical. actually i think someone else might have told that to me, first...

kovidgoyal · 12-14-2009, 01:07 PM

Don't see a way to leave comments on your blog posts. You should look at some more sophisticated encoding detection libraries like chardet. And don't rely on Qt for anything except drawing the GUI. I learned that the hard way

Fortunately since calibre is written in python, it was easy to find replacements.

Valloric · 12-14-2009, 02:11 PM

Quote:

Originally Posted by kovidgoyal

Don't see a way to leave comments on your blog posts.

Yes, the template seems to have gotten screwed up. I'll fix it as soon as I figure out how.

Quote:

Originally Posted by kovidgoyal

You should look at some more sophisticated encoding detection libraries like chardet.

Will do.

Quote:

Originally Posted by kovidgoyal

And don't rely on Qt for anything except drawing the GUI. I learned that the hard way

It seems I'm learning this lesson every day.

Quote:

Originally Posted by kovidgoyal

Fortunately since calibre is written in python, it was easy to find replacements.

But there's the problem: Sigil is C++, and finding replacements is hard. They have to be cross-platform too, and that's not as frequent a feature as I'd like.

kovidgoyal · 12-14-2009, 02:29 PM

Quote:

Originally Posted by Valloric

But there's the problem: Sigil is C++, and finding replacements is hard. They have to be cross-platform too, and that's not as frequent a feature as I'd like.

Yeah that was one of the major motivators for my choice of Python.

Valloric · 12-14-2009, 02:56 PM

Quote:

Originally Posted by kovidgoyal

Don't see a way to leave comments on your blog posts.

Should be fixed now.

Quote:

Originally Posted by kovidgoyal

You should look at some more sophisticated encoding detection libraries like chardet.

I took a look at it. There's a great python library (sigh). The same can't be said for C++. I'd have to extract the chardet code and all the code it depends on from mozilla trunk. Doesn't sound fun. It also sounds like waaaay too much work.

Besides, I was planning on using the ICU Character Set Detection some time in the future. I mean c'mon, it's ICU for Pete's sake. It's the gold standard of character conversions. Their way of encoding detection has to be good.

kovidgoyal · 12-14-2009, 03:26 PM

Ah well, hopefully ICU will do the trick. I chose chardet because it's optimized for documents from the web and in practice most encoding issues arise with HTML files from the web. And it has a great python library

Valloric · 12-14-2009, 03:57 PM

Quote:

Originally Posted by kovidgoyal

I chose chardet because it's optimized for documents from the web and in practice most encoding issues arise with HTML files from the web.

Yeah, but I only need the last part, the heuristic and statistical analysis. Sigil currently covers the following:

(X)HTML files that specify the encoding with the <meta> tag
(X)HTML files that specify the encoding with the XML "encoding" attribute
UTF-16 and UTF-32 (BE and LE) when they're not specified, through BOM detection
UTF-8 when not specified, through byte stream fingerprinting

All that's left are the files with an unspecified regional encoding. And for that I only need the analysis algorithms. And those from ICU are second to none.

But I'm not ruling out chardet yet. ICU is very big, and getting it to work under CMake... we'll it may turn out that pulling out the chardet sources and their dependencies from Mozilla trunk is less painful. We'll see.

Quote:

Originally Posted by kovidgoyal

And it has a great python library

Rub it in.

JayCeeEll · 12-14-2009, 04:24 PM

Wot no epub version of your blog.

For shame, for shame!

Valloric · 12-14-2009, 04:46 PM

Quote:

Originally Posted by JayCeeEll

Wot no epub version of your blog.

For shame, for shame!

I'd possibly make it generate an epub file on demand if I had control of the server, but I don't. And I don't see the use-case, really.

Valloric · 12-23-2009, 12:45 PM

For anyone interested, there's a new post on the blog that talks about the 0.2.0 redesign. Pictures, too.

KevinH · 12-29-2009, 08:15 PM

Hi,

I really like your blog and the ui for handling multiple xhtml and style files aat one time.

Very nice!

One question, my biggest issue is importing one very large xhtml files (typically generated by conversion from one large pml file into one large xhtml file). This is what makes the current ui so slow, which you have recognized and are redesigning for 0.2.0.

The issue is that the ui will be slow until I cut up the one large xhtml file into smaller pieces. So my question is how do you envision the ui responding to the insertion of a Sigil Chapter Break tag?

1. Will it immediately split the file and then "two separate files" will show up in the sidebar on the left? Will you then handle on the fly, updating links, tocs, spine, entries, etc? That might be hard to do but that may be what you have in mind.

Or

2. Will I instead have to set all of the Sigil chapter breakpoints first (slow in a very large file), and then convert it to epub (which will split the file into chapter size pieces) and then reload it so see the new sub files made from the inserted sigil chapter breaks?

If number 2, then what if I imbed proper Sigil Chapter breaks in my one big xhtml file in advance of loading it in Sigil (or whatever code you may want to reflect file split points)? Do you envision Sigil upon initial loading and seeing the splits - on the fly creating nicely numbered file pieces (content001.xhtml, ...) that would show up in the sidebar on the left of the ui?

Just wondering. Your Sigil program is wonderful but working from one big xhtml file is a pain, and your software does nice job updating links and things when it splits files, so using it to split large xhtml files into chapter size pieces on the fly (if I preinsert the right code where I want the breaks to be) would be a wonderful feature to have that would remove the need to work with one big xhtml in Sigil at all.

Thanks,

KevinH.

ps. I would have commented on your blog page but it did not accept my MobileRead login and password and I don't have any of the other categories of login-ids, listed here so I hope posting this question here is okay.

12-13-2009, 08:56 PM	#1
Valloric Created Sigil, FlightCrew Posts: 1,982 Karma: 350515 Join Date: Feb 2008 Device: Kobo Clara HD	New Sigil development blog I've been convinced (don't ask) that I should create a development blog for Sigil. Apparently it's a "great way to connect to one's users". We'll see. Anyway, here it is: http://sigildev.blogspot.com/ It won't be high-volume, I'm guessing about a post a week, tops. It will vary depending whether I have the time and something interesting to say. It will also be a place for posting any pertinent news, like for instance information about new releases. The release threads are hardly the way to go about it, since not everyone has an account on MobileRead (or is inclined to create one to post). But the major reason for the blog is the ability to inform you about how the current release is progressing, what's being done (and why) and how it's being done. Tracking the live ChangeLog only goes so far. It also enables me to go into greater detail on certain changes/bug fixes/features or their design. For instance, the Find&Replace dialog went through several design iterations before I chose the final one. Wouldn't it be nice if you could have seen some screenshots? Maybe voted for one design over the other? Threads only go so far, and it's hard for new users to find/browse them when they drop off the radar. In the end, this is an experiment. It may work and bring value to the users, or it may not. Here's to hoping it does.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre development	macellan	Calibre	3	05-21-2010 06:36 PM
PRS-600 App Development.	FlatlineTech	Sony Reader	3	04-26-2010 08:41 PM
iLiad How to get started in development?	mike b	iRex Developer's Corner	9	02-01-2007 03:38 AM
VadeMecum development	tovarish	Reading and Management	2	07-30-2006 06:34 PM
iLiad Hacking and/or development	arivero	iRex Developer's Corner	21	07-11-2006 10:45 AM

12-14-2009, 12:13 PM	#2
kjk . Posts: 3,408 Karma: 5647231 Join Date: Oct 2008 Device: never enough	Looks great so far, Valloric! (plus, great blog layout-very "un-blogspotlike" in a good way.)

12-14-2009, 12:17 PM	#3
zelda_pinwheel zeldinha zippy zeldissima Posts: 27,827 Karma: 921169 Join Date: Dec 2007 Location: Paris, France Device: eb1150 & is that a nook in her pocket, or she just happy to see you?	i think that's a great idea, valloric. it's really interesting to see a bit of what's going on behind the scenes. thanks for taking the time to write about your work ! a thought, you might want to make this thread sticky, so people can find it easily.

12-14-2009, 01:07 PM	#6
kovidgoyal creator of calibre Posts: 45,176 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Don't see a way to leave comments on your blog posts. You should look at some more sophisticated encoding detection libraries like chardet. And don't rely on Qt for anything except drawing the GUI. I learned that the hard way Fortunately since calibre is written in python, it was easy to find replacements.

12-14-2009, 03:26 PM	#10
kovidgoyal creator of calibre Posts: 45,176 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Ah well, hopefully ICU will do the trick. I chose chardet because it's optimized for documents from the web and in practice most encoding issues arise with HTML files from the web. And it has a great python library

12-14-2009, 04:24 PM	#12
JayCeeEll Connoisseur Posts: 87 Karma: 204 Join Date: Dec 2007 Location: Exeter, Devon, UK Device: PRS-300	Wot no epub version of your blog. For shame, for shame!

12-23-2009, 12:45 PM	#14
Valloric Created Sigil, FlightCrew Posts: 1,982 Karma: 350515 Join Date: Feb 2008 Device: Kobo Clara HD	For anyone interested, there's a new post on the blog that talks about the 0.2.0 redesign. Pictures, too.

12-29-2009, 08:15 PM	#15
KevinH Sigil Developer Posts: 8,467 Karma: 5703586 Join Date: Nov 2009 Device: many	Hi, I really like your blog and the ui for handling multiple xhtml and style files aat one time. Very nice! One question, my biggest issue is importing one very large xhtml files (typically generated by conversion from one large pml file into one large xhtml file). This is what makes the current ui so slow, which you have recognized and are redesigning for 0.2.0. The issue is that the ui will be slow until I cut up the one large xhtml file into smaller pieces. So my question is how do you envision the ui responding to the insertion of a Sigil Chapter Break tag? 1. Will it immediately split the file and then "two separate files" will show up in the sidebar on the left? Will you then handle on the fly, updating links, tocs, spine, entries, etc? That might be hard to do but that may be what you have in mind. Or 2. Will I instead have to set all of the Sigil chapter breakpoints first (slow in a very large file), and then convert it to epub (which will split the file into chapter size pieces) and then reload it so see the new sub files made from the inserted sigil chapter breaks? If number 2, then what if I imbed proper Sigil Chapter breaks in my one big xhtml file in advance of loading it in Sigil (or whatever code you may want to reflect file split points)? Do you envision Sigil upon initial loading and seeing the splits - on the fly creating nicely numbered file pieces (content001.xhtml, ...) that would show up in the sidebar on the left of the ui? Just wondering. Your Sigil program is wonderful but working from one big xhtml file is a pain, and your software does nice job updating links and things when it splits files, so using it to split large xhtml files into chapter size pieces on the fly (if I preinsert the right code where I want the breaks to be) would be a wonderful feature to have that would remove the need to work with one big xhtml in Sigil at all. Thanks, KevinH. ps. I would have commented on your blog page but it did not accept my MobileRead login and password and I don't have any of the other categories of login-ids, listed here so I hope posting this question here is okay.

Advert

Advert