Extremely slow in long files

[arspr]

Hi Kovid,

I suppose this is a side effect of whatever coding condition, so it's not solvable, but nevertheless here I go.

I've been editing a book with very long html files (1 Mb and 750 kb). Any kind of editing action (Beautify all, Replace All) was extremely slow (up to a minute and more in a Core 2 Quad Q6600 @2.4 GHz - 4 GB RAM). And the software seemed hung while working. More over, if I clicked over the window, Windows put the "(Software not responding)" warning in its title bar.

I've split that files into "normal" chapters (now the biggest file is only 60 kb) and the very same actions run in just a few seconds.

[kovidgoyal]

When any change is made to a very long file, the entire syntax highlighting for the rest of the file might need to be recalculated (depending on the change). The HTML syntax highlighter in the editor actually keeps track of tag nesting in order to be able to provide inline language dependent spellchecking. Which means that editing long files will be slow. Split your files. Having huge files is almost never a good idea.

[Nebucad]

Originally Posted by kovidgoyal

When any change is made to a very long file, the entire syntax highlighting for the rest of the file might need to be recalculated (depending on the change). The HTML syntax highlighter in the editor actually keeps track of tag nesting in order to be able to provide inline language dependent spellchecking. Which means that editing long files will be slow. Split your files. Having huge files is almost never a good idea.

Is it possible to tweek the recalculation option?

I figured out calibre editor has little problems to handle big .html file in generally. Even splitting a file take a "long time". Once it caused an script error as well and i fear couldn't split the file at all. I wait until error poped up and push the stop btn. afterwards i could split the file. Later when the file droped blelow 700-800kb it works fine and faster.

I got editing timewaits if a file is bigger than 100kb or close to it. I noticed the recalculating starts if:

* adding "class=" into a tag, start at the point when I type "="
* type in "<" starts recalculation, even inside a html-Tag i.e. <TAG> "<" </TAG>
* if I remove the closing TAG (any TAG) i.e. <h1><b>BLA BLA</b><h1> removing the "</b>"-TAG cause an recalculating

To avoid the recalculating I always mark a TAG completely before I hit the del btn. and I start to remove the Opening-TAG first.

The 2nd step i use copy and past i.e. "class="xx"" this avoid recalculating as well.

If the file is smaller i guess until 50-70kb everything runs fine and fast enough for editing.

I suggest to set a delay for the calculating or increase the value.

[mrmikel]

Have you turned off showing spelling errors in code view? It makes quite a difference.

[Nebucad]

Originally Posted by mrmikel

Have you turned off showing spelling errors in code view? It makes quite a difference.

No i didn't i'll try this thx for the advice

[arspr]

Originally Posted by kovidgoyal

When any change is made to a very long file, the entire syntax highlighting for the rest of the file might need to be recalculated (depending on the change). The HTML syntax highlighter in the editor actually keeps track of tag nesting in order to be able to provide inline language dependent spellchecking. Which means that editing long files will be slow. Split your files. Having huge files is almost never a good idea.

Thanks for your explanation, Kovid.

Just a stupid hint I'm nearly sure you are currently aware of, but just in case.

Please notice how I pointed automated actions (Beautify All files, S&R All through all files) where syntax highlighter is useless while working...

I mean, if possible in the code, if they are the culprits for the pseudo-hang, and if you are not actually doing it, you could firstly turn syntax highlighting (and spell checking) off, then perform those global actions and then re-enable them.

Just thoughts...

[Nebucad]

Originally Posted by arspr

I mean, if possible in the code, if they are the culprits for the pseudo-hang, and if you are not actually doing it, you could firstly turn syntax highlighting (and spell checking) off, then perform those global actions and then re-enable them.

I suggest it as well.
Splitting the file might work, apparently not verry well. Even I splitt the file in small chunks (about 60kb each) won't prevent those psodo "hang-ups".

I could splitt it further below 20kb each but this seems to me redicules. Splitting just a file orientate on size wouldn't look always good, Splitting also means new page and pagebrakes have to make sense. additionally it's take his time to splilt a file into smaler chunks.

I've checked how Sigl control the file and this editor didn't has any problems with the .html size at all. I could editing the file whithout causing any "psode hang-ups". So i guess the editor limit his self.

It'll would be better if you can work fast with the editor instead of good looking syntax. You want to know how the code would look like? you use the preview window. Tidy up code you do it only once after work or for checking. you don't need it every second and not automatically.

[mrmikel]

Sigil doesn't have a problem because it enforces its own organization on the file when it brings it in. It doesn't have to try to figure things out. It also doesn't let you save when the file is badly formed. These give Sigil a leg up on that score, but these can be deficiencies as well if you are working with total junk in the first place.

People with brand new shiny e-readers and tablets can handle the big files, but there are lots of us out here with older but very serviceable machines. It is no problem to break at the start of a chapter. It takes a pretty large chapter, generally non-fiction to be over 250k long.

Also using the preview window with large files will slow it up because every change will require reformatting the whole file in the preview window. Leave the window unsync'ed and manually re-synch it to check your results.

Those having trouble might considering downloading a file at random out of the MR library and see how fast editing goes with that file. It might give a clue as to whether the file you are working on is a problem because it is full of extra code or something or whether the program is causing the problem.

[kovidgoyal]

If you are experiencing hangs with files of less than 100KB then there is something wrong with your machine. On my system with inline spell checking turned off, syntax highlighting for a 600KB HTML file takes under a second.

And with some performance improvements that will be in the next release syntax highlighting even with inline spell checking turned on takes less than a second for 600KB worth of HTML.

[Nebucad]

Originally Posted by mrmikel

Those having trouble might considering downloading a file at random out of the MR library and see how fast editing goes with that file. It might give a clue as to whether the file you are working on is a problem because it is full of extra code or something or whether the program is causing the problem.

Full of extra Code - could be.
The original file was a over 700 page non-fiction book (pdf-file 65MB). I use calibre to make a epub, usually i want to make a azw3 conversion but i think this didn't worked well so i made a epub conversion.

remote content
* all Hyperlinks
* all Ancors
* all Images (Links and physically)
* all CCS-Styles

I would call it as a "blank" book. I merge all small files into big one and splitt this file chapter by chapter. i got 14 chapters varios size. One Chapter got 300kB.

I´m using
* own css-formats and class
* HTML-TAGs like (table, ol, ul, li, p, pre, div, figure, figcapture, links, and image)
*Importing resized images

The file will be later converted to azw3 - This is what i actually do with the file.

I notice during work those "psodo hang-ups" and i notice this happend at file around 100kB. So i made the file smaller and work goes better. I got to the 300kB file and something else i notice. i use RegEX and Copy and Paste to speed up my work. It will take a while but if i make many actions the editing get slower and slower. I just have to save my work and quit the editor afterwards i can reopen it and everything runs faster.

I'm sorry for not beeing more precisely. I actually can't locate the problem what might could be.

It is the first time I'm using calibre-editor for this big projekt. I used to work with Sigil. thats the reason why i don't know such working lags aka (psodo-Hang-ups). So far i made steps which i never made in sigil, thats why I'm concern about it.

In my special case, the file might be corrupted or have extra code who cause those slow downs. I didn't noticed anything like this. Who knows we wait until the next editor update comes and than we'll see further.

My post should just be a hint that something dosen't worked well with the editing of big files. I know sound like nothing but more informaion i can't provide right now.

[mrmikel]

What results do you get when you run check book from the tools menu?

I also wonder if any antivirus program may be an issue.

I used to have a similar problem, but it was in Sigil!!

[Nebucad]

Originally Posted by mrmikel

What results do you get when you run check book from the tools menu?

I also wonder if any antivirus program may be an issue.

I used to have a similar problem, but it was in Sigil!!

No problems found.

Well isn't unusual. I fixed all "errors" and the only once who might popup will be an HTML missmatch i'm not done with editing yet.