Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Sigil

Notices

Reply
 
Thread Tools Search this Thread
Old 08-15-2009, 09:27 AM   #1
DanielCoffey
Connoisseur
DanielCoffey began at the beginning.
 
DanielCoffey's Avatar
 
Posts: 66
Karma: 26
Join Date: Jun 2009
Location: Edinburgh, UK
Device: Sony PRS-505
v0.1.2 very slow when editing on Mac

Valloric - if you are already aware of this problem just let me know but if not, I'll raise an issue for you...

=====

Sigil v0.1.2
Mac OS Leopard 10.5.8
iMac 24" 4Gb Ram

ISSUE : Sigil is very slow to open The_Deed_of_Paksenarrion.epub from Baen Webscriptions, taking around 90 seconds. The epub is 1.4Mb in size and contains around 30-ish chapters (1350 pages or so on my Sony 505).

It takes between 8-10 seconds to respond to a single character deletion and around 15 seconds to respond to a carriage return in the middle of a paragraph, displaying the Mac rotating beachball each time.

In addition, it opens this epub and displays it centered whereas my Sony 505 displays the epub as left justified... I not sure if this is just the default behaviour where justification is not set in the document.

Let me know if you want me to raise an official issue (and include the epub).

Cheers,

Daniel.
DanielCoffey is offline   Reply With Quote
Old 08-15-2009, 09:40 AM   #2
Valloric
Created Sigil, FlightCrew
Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.
 
Valloric's Avatar
 
Posts: 1,978
Karma: 350515
Join Date: Feb 2008
Device: Sony Reader PRS 505
Quote:
Originally Posted by DanielCoffey View Post
ISSUE : Sigil is very slow to open The_Deed_of_Paksenarrion.epub from Baen Webscriptions, taking around 90 seconds. The epub is 1.4Mb in size and contains around 30-ish chapters (1350 pages or so on my Sony 505).

It takes between 8-10 seconds to respond to a single character deletion and around 15 seconds to respond to a carriage return in the middle of a paragraph, displaying the Mac rotating beachball each time.
There have been some reports of issues like this. From what I can gather, it is very rare.

Tell me, does this happen with all epub books, or just this one? If it's just this one, then create the issue and attach the epub... otherwise I don't know how to help you. If it happens with all epub books, then it must be your system.

Quote:
Originally Posted by DanielCoffey View Post
In addition, it opens this epub and displays it centered whereas my Sony 505 displays the epub as left justified... I not sure if this is just the default behaviour where justification is not set in the document.
This is unrelated, and not a Sigil issue, but a Sony PRS-505 issue. The Mobile Digital Editions version on the 505's always displays books left justified, even if the markup in them specifies full justification. So your book is probably fully justified.

They fixed this in later versions of Mobile DE, hopefully we'll get it in the firmware update that's coming at the end of this month.
Valloric is offline   Reply With Quote
Old 08-15-2009, 10:16 AM   #3
DanielCoffey
Connoisseur
DanielCoffey began at the beginning.
 
DanielCoffey's Avatar
 
Posts: 66
Karma: 26
Join Date: Jun 2009
Location: Edinburgh, UK
Device: Sony PRS-505
Thanks for the quick reply...

Oddly enough, another smaller epub I have does it too, but to a much lesser degree. War For The Oaks epub (free from Tor) is 368Kb and the delays are smaller - around 10-12 seconds to open and only slightly sluggish responses when editing.

===

System : iMac 2.8GHz Intel Core 2 Duo, 4Gb RAM
OS : Mac OS X 10.5.8

Sigil idle, no ebook loaded, 0% CPU, 25Mb mem used

When opening War For The Oaks (368Kb), Sigil's CPU usage goes to 100% and its memory jumps from 25Mb used to 45Mb used then CPU usage settles at 0% once epub is loaded.

When entering carriage returns in the middle of a paragraph, CPU usage is about 25% but when deleting them, CPU usage hits 90-100% and it takes around 1s per deletion.

I then closed Sigil and reopened it before loading the 1.4Mb Deed of Paksenarrion epub. Activity monitor settled at 100% CPU usage and quickly reported Sigil (Not Responding) and memory usage climbed rapidly to 112.5Mb used. Once the epub was loaded, CPU settled back to 0%.

I am not sure if that degree of memory and CPU usage is what you would expect.
DanielCoffey is offline   Reply With Quote
Old 08-15-2009, 11:55 AM   #4
Valloric
Created Sigil, FlightCrew
Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.
 
Valloric's Avatar
 
Posts: 1,978
Karma: 350515
Join Date: Feb 2008
Device: Sony Reader PRS 505
Quote:
Originally Posted by DanielCoffey View Post
Oddly enough, another smaller epub I have does it too, but to a much lesser degree.
So what you're saying is that you have epubs for which this doesn't happen? Then attach the epub for which it does so I can take a look at what's going on.

Quote:
Originally Posted by DanielCoffey View Post
When opening War For The Oaks (368Kb), Sigil's CPU usage goes to 100% and its memory jumps from 25Mb used to 45Mb used then CPU usage settles at 0% once epub is loaded.
This is normal if it takes a few seconds. The loading is CPU intensive because of tidyLib which reads in your HTML and rewrites it completely.

Quote:
Originally Posted by DanielCoffey View Post
When entering carriage returns in the middle of a paragraph, CPU usage is about 25% but when deleting them, CPU usage hits 90-100% and it takes around 1s per deletion.
On my machine (Core 2 Duo, 4GB RAM), so far I have yet to see an editing task take more than 5% of CPU. But I've tried deleting paragraphs as you've noted, and this does seem to jump the usage to 100% for God knows what reason... but only on HUGE, 50k XHTML lines-of-code files. It's 2% CPU and instant response for average novel-length files (on my machine).

When editing huge files, there's going to be some unavoidable slowdown for Book View. It's still an embedded webkit-based browser.

Quote:
Originally Posted by DanielCoffey View Post
I then closed Sigil and reopened it before loading the 1.4Mb Deed of Paksenarrion epub. Activity monitor settled at 100% CPU usage and quickly reported Sigil (Not Responding) and memory usage climbed rapidly to 112.5Mb used. Once the epub was loaded, CPU settled back to 0%.

I am not sure if that degree of memory and CPU usage is what you would expect.
As I've said, loading is intensive. There's probably room for optimization there.

Even for huge files, memory usage should remain well below 100MB, between 40 and 70 MB. The Peak Memory Set has for me gone beyond 100MB (102MB actually), but that's for a split second during loading. It always goes down afterwards.

As reported, Sigil should be able to handle very large files even on weak hardware.

This is all on Windows though. I haven't had the chance to do performance testing for Macs much. From my limited testing with an old Macbook, it seems fine though.

But 90-second loading and several-second lag for typing? That should never happen. Attach that epub so I can take a look at it. If you're worried about your file being made available on the internet, just add the "Private" label to the issue which makes it visible only to the issue reporter (you) and the project developers (me).

Without the epub files in question I can only guess. If you provide me with it, and the same slowdown happens on my system, then I can tell you what the problem is (and hopefully I can fix it). If it doesn't happen on my system with the same file, then it must be your system that's at fault.
Valloric is offline   Reply With Quote
Old 08-15-2009, 12:33 PM   #5
DanielCoffey
Connoisseur
DanielCoffey began at the beginning.
 
DanielCoffey's Avatar
 
Posts: 66
Karma: 26
Join Date: Jun 2009
Location: Edinburgh, UK
Device: Sony PRS-505
I have created private issue 82 for you with the slow epub attached. I hope it behaves the same on your PC as it does on mine.

I must say that Calibre and Sigil make a wonderful pair of applications. You should be truly proud of all your hard work!
DanielCoffey is offline   Reply With Quote
Old 08-15-2009, 01:18 PM   #6
Valloric
Created Sigil, FlightCrew
Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.
 
Valloric's Avatar
 
Posts: 1,978
Karma: 350515
Join Date: Feb 2008
Device: Sony Reader PRS 505
Quote:
Originally Posted by DanielCoffey View Post
I have created private issue 82 for you with the slow epub attached. I hope it behaves the same on your PC as it does on mine.
It does. It most certainly does.

It just that Sigil is not the problem. The epub file is. The markup is... horrible, to say the least. For instance, every paragraph with text starts like this:
Code:
<p onmouseover="PNo(1032)">
And there is no javascript included in the book, from what I can tell. This only serves to slow down browsers. Also, between every two paragraphs with text there is code like this:
Code:
<p><a id="p1033" name="p1033"/></p>
Which does God knows what.

And I see what you mean when you said the text was centered. I thought you meant fully justified, but no, it is really centered. Removing the CSS style that applies "text-align: center" fixed this.

With that CSS style gone, and after removing all the useless "p/anchors" and the onmouseover handlers with some regexes in notepad++, your file now takes up 30MB less memory and can be nicely edited in Sigil with no lag. So CPU usage solved.

The memory consumption is still around 100MB, but your file is 80k XHTML lines-of-code. It is 1.4MB as an epub because epubs are compressed ZIP archives, and plain text gets compressed nicely. In Sigil, text is in uncompressed UTF-16 which means at least two bytes for every character, whereas your file is English in UTF-8, so only one byte per character stored. This effectively doubles the memory required to store your text in Sigil.

Now take into account that because of technical limitations of Qt widgets used for Code View and Book View, there are three text buffers instead of one, and that Book View is an embedded web browser... these things add up.

In the end it was your file that was causing the lag. Must have been an old version of calibre they were using when they created it, because that's some really painful markup.
Valloric is offline   Reply With Quote
Old 08-15-2009, 01:40 PM   #7
DanielCoffey
Connoisseur
DanielCoffey began at the beginning.
 
DanielCoffey's Avatar
 
Posts: 66
Karma: 26
Join Date: Jun 2009
Location: Edinburgh, UK
Device: Sony PRS-505
Fantastic! Er, I mean fantastic that it was reproducible, of course. I will feed your comments back to the owner of the Webscriptions site and get my copy cleaned up.

Thanks for looking at this and apologies for it taking time away from your other work and fun. Please feel free to close the issue (82).

As I said - well done for working on Sigil (even the logo is gorgeous!).
DanielCoffey is offline   Reply With Quote
Old 08-18-2009, 04:47 AM   #8
Dave Berk
Fanatic
Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.
 
Dave Berk's Avatar
 
Posts: 578
Karma: 138556
Join Date: May 2006
Device: PRS505
I also have the same problem. With one file. So it's probably a problem with the file itself. But opening the file with a text editor or an html editor is no problem and editing is fast then too.

It's the same file I attached with issues 87 & 88.
Dave Berk is offline   Reply With Quote
Old 08-18-2009, 09:29 AM   #9
Valloric
Created Sigil, FlightCrew
Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.
 
Valloric's Avatar
 
Posts: 1,978
Karma: 350515
Join Date: Feb 2008
Device: Sony Reader PRS 505
Quote:
Originally Posted by Dave Berk View Post
I also have the same problem. With one file. So it's probably a problem with the file itself. But opening the file with a text editor or an html editor is no problem and editing is fast then too.

It's the same file I attached with issues 87 & 88.
I'm guessing these other editors are not rendering out all of the XHTML files as one flow.

I'm also starting to believe the whole "one flow for editing" idea may not be so amazing.
Valloric is offline   Reply With Quote
Old 08-18-2009, 11:23 AM   #10
Dave Berk
Fanatic
Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.Dave Berk can fool all of the people all of the time.
 
Dave Berk's Avatar
 
Posts: 578
Karma: 138556
Join Date: May 2006
Device: PRS505
Quote:
Originally Posted by Valloric View Post
I'm guessing these other editors are not rendering out all of the XHTML files as one flow.
Yep.

Quote:
Originally Posted by Valloric View Post
I'm also starting to believe the whole "one flow for editing" idea may not be so amazing.
What other option is there?
Dave Berk is offline   Reply With Quote
Old 08-18-2009, 12:02 PM   #11
Valloric
Created Sigil, FlightCrew
Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.
 
Valloric's Avatar
 
Posts: 1,978
Karma: 350515
Join Date: Feb 2008
Device: Sony Reader PRS 505
Quote:
Originally Posted by Dave Berk View Post
What other option is there?
A poll/discussion thread on the future direction of Sigil's development (with this issue central) is forthcoming. Watch the forum.
Valloric is offline   Reply With Quote
Old 08-18-2009, 12:28 PM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,809
Karma: 5006091
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
My vote would be to split files over a certain size and use split files if they already exist.
kovidgoyal is online now   Reply With Quote
Old 08-18-2009, 05:06 PM   #13
Xenophon
curmudgeon
Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.
 
Xenophon's Avatar
 
Posts: 1,453
Karma: 5657812
Join Date: Jun 2006
Location: Pittsburgh, PA USA
Device: Kobo Aura (soon), (soon-to-be-ex)nook, (ex)PRS-700, (ex)PRS-500
Quote:
Originally Posted by Valloric View Post
It does. It most certainly does.

It just that Sigil is not the problem. The epub file is. The markup is... horrible, to say the least. For instance, every paragraph with text starts like this:
Code:
<p onmouseover="PNo(1032)">
And there is no javascript included in the book, from what I can tell. This only serves to slow down browsers. Also, between every two paragraphs with text there is code like this:
Code:
<p><a id="p1033" name="p1033"/></p>
Which does God knows what.

And I see what you mean when you said the text was centered. I thought you meant fully justified, but no, it is really centered. Removing the CSS style that applies "text-align: center" fixed this.

With that CSS style gone, and after removing all the useless "p/anchors" and the onmouseover handlers with some regexes in notepad++, your file now takes up 30MB less memory and can be nicely edited in Sigil with no lag. So CPU usage solved.

The memory consumption is still around 100MB, but your file is 80k XHTML lines-of-code. It is 1.4MB as an epub because epubs are compressed ZIP archives, and plain text gets compressed nicely. In Sigil, text is in uncompressed UTF-16 which means at least two bytes for every character, whereas your file is English in UTF-8, so only one byte per character stored. This effectively doubles the memory required to store your text in Sigil.

Now take into account that because of technical limitations of Qt widgets used for Code View and Book View, there are three text buffers instead of one, and that Book View is an embedded web browser... these things add up.

In the end it was your file that was causing the lag. Must have been an old version of calibre they were using when they created it, because that's some really painful markup.
I recognize those HTML parts! If you visit the Baen free library (for example) and read a book there on the Web -- that is, you view in your browser rather than downloading and reading locally -- you get html that contains these parts. What they appear to be is support for identifying paragraphs by number (those are paragraphs #1032 and #1033 in your quoted examples above). That was useful stuff when users were reporting typos and other bugs to the publisher from the Web, but serve no purpose in a downloaded eBook.

Kovid built the -Baen preprocessing switch for Calibre exactly to strip that stuff out (at my request). May I suggest that Sigil provide the same capability? It's a straight-forward bit of sed script hacking...

Xenophon

P.S. I'll poke Baen's web guy about removing that cruft from his eBook versions.
Xenophon is offline   Reply With Quote
Old 08-18-2009, 05:27 PM   #14
Valloric
Created Sigil, FlightCrew
Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.Valloric ought to be getting tired of karma fortunes by now.
 
Valloric's Avatar
 
Posts: 1,978
Karma: 350515
Join Date: Feb 2008
Device: Sony Reader PRS 505
Quote:
Originally Posted by Xenophon View Post
Kovid built the -Baen preprocessing switch for Calibre exactly to strip that stuff out (at my request). May I suggest that Sigil provide the same capability? It's a straight-forward bit of sed script hacking...
Sigil is an editor, and as such it strives to present the epub you loaded as faithfully as possible. When the search&replace dialog is done, it will be very easy to remove these directly. But Sigil will not remove them for you automatically. Maybe as an option in some pop-up import dialog. Maybe.

Until then (and after too, for those who want to) just run it through calibre to remove these. There's little point in duplicating functionality.

Quote:
Originally Posted by Xenophon View Post
P.S. I'll poke Baen's web guy about removing that cruft from his eBook versions.
Please do that. It would benefit all readers of those books, since it would speed up display and lower memory consumption on all Reading Systems.
Valloric is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Mac users that are Reader owners, help a new Mac owner AnemicOak Sony Reader 8 06-12-2010 04:36 PM
Slow editing avresbo Sigil 14 02-03-2010 10:07 PM
Slow to use? Mac Sidewinder Sigil 9 08-07-2009 07:48 PM
calibre - very slow conversion, very slow on PRS cremofix Calibre 3 06-10-2009 04:21 PM
Editing Kindle (.prc) metadata on a Mac GJN Kindle Developer's Corner 15 03-10-2009 01:22 PM


All times are GMT -4. The time now is 07:36 AM.


MobileRead.com is a privately owned, operated and funded community.