Tolerance and parsing mistakes - Page 4

Toxaris · 01-22-2016, 02:21 PM

Perhaps Calibre empties the clipboard before starting the editor? Or could it be a difference between Qt for Python and Qt for C++?

DiapDealer · 01-22-2016, 02:27 PM

Quote:

Originally Posted by KevinH

Is there an easy way on Linux mint to fire up a Motif based Window Manager and then fire up Sigil to see if the problem goes away or not. If it stays, that would probably rule out my INCR theory.

I don't know about motif, but once I make sure something is in the clipboard, I experience the slow down on machines with the Xfce window manager, Gnome/metacity and x-Cinnamon/Mutter.

My question is: if this bug affects so many window-managers (or x11 in general), how is this NOT a huge deal being screamed about everywhere Qt5 and Linux intersect?

KevinH · 01-22-2016, 02:55 PM

Hi DiapDealer,

This is so strange. How can the presence of a single letter 'I' on the clipboard at launch have such an extreme impact on the text selection highlighting speed if it has nothing to do with how Qt/Sigil and the system clipboard are talking to each other.

That said unless Toxaris is right and PyQt clears the clipboard on start, I can't see any other differences. I tried this with QtCreator and the stuff on the clipboard had no impact on its highlighting speed in their text editor. So it must be how we are using it in Sigil but ...

KevinH

Quote:

Originally Posted by DiapDealer

I don't know about motif, but once I make sure something is in the clipboard, I experience the slow down on machines with the Xfce window manager, Gnome/metacity and x-Cinnamon/Mutter.

KevinH · 01-22-2016, 03:02 PM

BTW ... I fired up Sigil with a blank clipboard. Copied a single line to the clipboard. Exited Sigil, fired up calibre and pasted the line and it was copied perfectly.

So calibre is NOT clearing the clipboard when starting up ....

Also after loading a long book in Sigil, I clicked on BookView and was able to select huge swathes of text with no slowdown at all. So whatever it is, it is something to do with CodeView in some way.

So this is getting stranger yet ...

KevinH

KevinH · 01-22-2016, 03:12 PM

Here is another interesting observation. With something in the clipboard, I started up Sigil. Then opened an epub3 ebook that had css, javascripts, and of course a content.opf to go with its xhtml files.

I tried selecting text in the xhtml and css tabs and they appeared slow. I tried selecting text n the javascript tab and in the content.opf and both were fast! No slowdown. The javascript one did not have any syntax highlighting. So i thought it might be the syntax highlighting we do. But then the content.opf did have syntax highlighting and it was fast.

The css and xhtml both have syntax highlighting and are slow. A CodeView widget is used for all of these. The only thing different is the type of Tab widget being used.

Hmm....

KevinH

DiapDealer · 01-22-2016, 03:13 PM

I feel a breakthrough on the horizon.

KevinH · 01-22-2016, 03:43 PM

Hi,

Well we can rule out Syntax Highlighting as the culprit. I set it to NONE for FlowTabs.cpp and it had no effect. The scrolling was still quite slow for xhtml files. So I just have to figure out what a FlowTab and CSSTab are doing that the OPFTab and XMLTab is not.

KevinH

KevinH · 01-22-2016, 05:08 PM

Hi,

BTW, one source of the problem is that QPlainTextEdit textChanged() signal is emitted by by simply highlighting a selection of even syntax highlighting. The problem is the base text in the PlainTextEdit has not changed one bit. What a silly design for emitting a textChanged() signal. We use this signal to know when to update the Preview window.

What any of this has to do with the Clipboard having something when Sigil is started is very very strange. For a while everthing worked even with things in the clipboard(), it it went back to its buggy ways.

This is so frustrating.

KevinH

KevinH · 01-23-2016, 11:40 AM

Okay, I can now recreate this in a new way that really tells me this is a bug in Qt.

1. open firefox or any browser to any page
2. xsel -bc
3. sigil
4. Play with text highlighting in default file 1 - it is fast
5. Click on your browser window and select some text (any text) and copy it to the clipboard, then return to sigil
6. Now try highlighting text again in Sigil it will be slow
7. highlight a small piece of text in Sigil and copy it to the clipbord
8. Now try highlighting text again in Sigil and it will be fast again

So the mere presence of any data not from Sigil in the clipboard will freak things out.

I searched and found this bug:

https://bugreports.qt.io/browse/QTBUG-38585

Which reports a problem with sleeping too much in clipboardWaitForEvent in qclipboard_x11.cpp causing slowdowns and delays.

There was a patch supplied but of course Qt ignored it and then someone at Qt claimed the file qclipboard_x11.cpp no longer exists so they abandoned the patch completely.

Of course, during the interval of qclipboard_x11.cpp was removed and replaced by qxcbclipboard.cpp and it has the exact same issue. Any time it wants to communicate with an outside app (another app that owns the current clipboard contents) it sits in a wait loop for 50000 us or 50 ms. Since highlighting text in X11 is a clipboard activity, as long as anything foreign is in the clipboard, it will sit and sleep many many times here while we are trying to highlight something waiting to talk to the owner of the current clipboard contents.

This problem does not exist in Windows or Mac because the system clipboard is global and there is no such thing as owners of clipboard contents, and because simple selection (highlighting text) is not a clipboard activity on Mac and Windows.

My guess is Qt will never fix it given how they ignored the problem and the patch offered and then abandoned it when they moved to qxcbclipboard.cpp which almost always uses the event loop approach. Sigil has about 3 or 4 other bug reports at Qt for years now that have never seen any action. One includes a use after free that could be used to attack the application. I have no hopes of anyone at Qt even looking at this issue at all, let alone fixing it.

So the question is how can we work around it on Linux? Given simply copying anything to the clipboard inside Sigil will prevent the issue (replacing the foreign ownership) there should be something Sigil can do. I am thinking of having Sigil check for ownership of the item on the clipboard and if necessary copying it and then pasting it back as its new owner. This should be doable.

Anyone have any pull with Qt about bug fixes for Linux that could get them to at least fix this, would be nice.

KevinH

DiapDealer · 01-23-2016, 12:31 PM

I still can't help but think that Sigil is doing something in Code View (or somewhere) that exacerbates the bug, no? Otherwise, why would it not manifest in Book View, or in the countless other Qt-based Linux apps that don't experience the same lag in text-selection?

Don't get me wrong ... if you come up with a successful workaround for the bug, I'll be happier than a pig in slop. It's just that I still don't understand why Code View-only?

KevinH · 01-23-2016, 12:52 PM

Hi DiapDealer,

"I'll be happier than a pig in slop." Well it's time to start slopping around! To test my theory, I tried my workaround of taking ownership of the clip when an external clip is added. This seems to fix the problem 100% on my Linux box. See my latest commit to Sigil Master.

So if simply changing ownership of a piece of clipboard text (by copying it out and pasting it back unchanged) and not changing anything else in Sigil, stops the bug completely, then I would say there is a bug in qxcbclipboard.cpp. Nothing like that should happen or happens on any other platform.

As to why Sigil, I think Sigil is a heavy user of XHTMLSyntaxHighlighting and when you add to that when you highlight anything in Sigil in xhtml you are sending the textChanged() signal out to the FlowTab and/or CSSTab (when nothing has ever changed in the underlying QPlainTextEdit widget - a bad design of Qt's as far as I can tell - and then emitting another signal from there to tell the MainWindow to tell Preview to Update itself. So there are a lot of signals flying around when someone on Linux / X11 simply selects the text! No other platform has a problem with it.

Anyway, please test my "copying out and pasting back fix" and let me know if the funny highlighting of code problem goes away on your machine. It does on mine!

Once it is confirmed, I will close the slow highlighting issue on Sigil's github.

KevinH

DiapDealer · 01-23-2016, 01:26 PM

Oh, man, it's looking good so far! Thanks for digging deep on this one.!

Toxaris · 01-23-2016, 02:43 PM

So it was indeed a difference between Qt for Python and Qt for C++. Very good digging and hunting Kevin!

roger64 · 01-23-2016, 08:11 PM

@KevinH

Wow! congratulations for squashing this sneaky one! Thank you also for making us live online this impressive hunting chase.

Count me another happy pig.

eschwartz · 01-23-2016, 08:57 PM

Thanks for tracking this down, Kevin!

Enjoying my now much faster dev build.

01-22-2016, 03:02 PM	#49
KevinH Sigil Developer Posts: 8,501 Karma: 5703586 Join Date: Nov 2009 Device: many	BTW ... I fired up Sigil with a blank clipboard. Copied a single line to the clipboard. Exited Sigil, fired up calibre and pasted the line and it was copied perfectly. So calibre is NOT clearing the clipboard when starting up .... Also after loading a long book in Sigil, I clicked on BookView and was able to select huge swathes of text with no slowdown at all. So whatever it is, it is something to do with CodeView in some way. So this is getting stranger yet ... KevinH Last edited by KevinH; 01-22-2016 at 03:05 PM.

01-22-2016, 05:08 PM	#53
KevinH Sigil Developer Posts: 8,501 Karma: 5703586 Join Date: Nov 2009 Device: many	Hi, BTW, one source of the problem is that QPlainTextEdit textChanged() signal is emitted by by simply highlighting a selection of even syntax highlighting. The problem is the base text in the PlainTextEdit has not changed one bit. What a silly design for emitting a textChanged() signal. We use this signal to know when to update the Preview window. What any of this has to do with the Clipboard having something when Sigil is started is very very strange. For a while everthing worked even with things in the clipboard(), it it went back to its buggy ways. This is so frustrating. KevinH Last edited by KevinH; 01-22-2016 at 06:10 PM.

01-23-2016, 11:40 AM	#54
KevinH Sigil Developer Posts: 8,501 Karma: 5703586 Join Date: Nov 2009 Device: many	Okay, I can now recreate this in a new way that really tells me this is a bug in Qt. 1. open firefox or any browser to any page 2. xsel -bc 3. sigil 4. Play with text highlighting in default file 1 - it is fast 5. Click on your browser window and select some text (any text) and copy it to the clipboard, then return to sigil 6. Now try highlighting text again in Sigil it will be slow 7. highlight a small piece of text in Sigil and copy it to the clipbord 8. Now try highlighting text again in Sigil and it will be fast again So the mere presence of any data not from Sigil in the clipboard will freak things out. I searched and found this bug: https://bugreports.qt.io/browse/QTBUG-38585 Which reports a problem with sleeping too much in clipboardWaitForEvent in qclipboard_x11.cpp causing slowdowns and delays. There was a patch supplied but of course Qt ignored it and then someone at Qt claimed the file qclipboard_x11.cpp no longer exists so they abandoned the patch completely. Of course, during the interval of qclipboard_x11.cpp was removed and replaced by qxcbclipboard.cpp and it has the exact same issue. Any time it wants to communicate with an outside app (another app that owns the current clipboard contents) it sits in a wait loop for 50000 us or 50 ms. Since highlighting text in X11 is a clipboard activity, as long as anything foreign is in the clipboard, it will sit and sleep many many times here while we are trying to highlight something waiting to talk to the owner of the current clipboard contents. This problem does not exist in Windows or Mac because the system clipboard is global and there is no such thing as owners of clipboard contents, and because simple selection (highlighting text) is not a clipboard activity on Mac and Windows. My guess is Qt will never fix it given how they ignored the problem and the patch offered and then abandoned it when they moved to qxcbclipboard.cpp which almost always uses the event loop approach. Sigil has about 3 or 4 other bug reports at Qt for years now that have never seen any action. One includes a use after free that could be used to attack the application. I have no hopes of anyone at Qt even looking at this issue at all, let alone fixing it. So the question is how can we work around it on Linux? Given simply copying anything to the clipboard inside Sigil will prevent the issue (replacing the foreign ownership) there should be something Sigil can do. I am thinking of having Sigil check for ownership of the item on the clipboard and if necessary copying it and then pasting it back as its new owner. This should be doable. Anyone have any pull with Qt about bug fixes for Linux that could get them to at least fix this, would be nice. KevinH Last edited by KevinH; 01-23-2016 at 01:09 PM.

01-23-2016, 12:52 PM	#56
KevinH Sigil Developer Posts: 8,501 Karma: 5703586 Join Date: Nov 2009 Device: many	Hi DiapDealer, "I'll be happier than a pig in slop." Well it's time to start slopping around! To test my theory, I tried my workaround of taking ownership of the clip when an external clip is added. This seems to fix the problem 100% on my Linux box. See my latest commit to Sigil Master. So if simply changing ownership of a piece of clipboard text (by copying it out and pasting it back unchanged) and not changing anything else in Sigil, stops the bug completely, then I would say there is a bug in qxcbclipboard.cpp. Nothing like that should happen or happens on any other platform. As to why Sigil, I think Sigil is a heavy user of XHTMLSyntaxHighlighting and when you add to that when you highlight anything in Sigil in xhtml you are sending the textChanged() signal out to the FlowTab and/or CSSTab (when nothing has ever changed in the underlying QPlainTextEdit widget - a bad design of Qt's as far as I can tell - and then emitting another signal from there to tell the MainWindow to tell Preview to Update itself. So there are a lot of signals flying around when someone on Linux / X11 simply selects the text! No other platform has a problem with it. Anyway, please test my "copying out and pasting back fix" and let me know if the funny highlighting of code problem goes away on your machine. It does on mine! Once it is confirmed, I will close the slow highlighting issue on Sigil's github. KevinH Last edited by KevinH; 01-23-2016 at 01:11 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Understanding two reported mistakes	roger64	Editor	11	02-19-2014 02:47 AM
Manage gramm. mistakes?	Dalirose	Onyx Boox	1	01-01-2014 12:22 PM
5 Mistakes Of New Fiction Writers	VydorScope	Writers' Corner	67	07-08-2012 07:36 AM
Noob install mistakes	Shiryas	Calibre	2	04-05-2010 01:30 AM
Mistakes in The Hobbit	AlexBell	News	58	05-12-2009 08:26 PM

01-22-2016, 02:21 PM	#46
Toxaris Wizard Posts: 4,520 Karma: 121692313 Join Date: Oct 2009 Location: Heemskerk, NL Device: PRS-T1, Kobo Touch, Kobo Aura	Perhaps Calibre empties the clipboard before starting the editor? Or could it be a difference between Qt for Python and Qt for C++?

01-22-2016, 03:12 PM	#50
KevinH Sigil Developer Posts: 8,501 Karma: 5703586 Join Date: Nov 2009 Device: many	Here is another interesting observation. With something in the clipboard, I started up Sigil. Then opened an epub3 ebook that had css, javascripts, and of course a content.opf to go with its xhtml files. I tried selecting text in the xhtml and css tabs and they appeared slow. I tried selecting text n the javascript tab and in the content.opf and both were fast! No slowdown. The javascript one did not have any syntax highlighting. So i thought it might be the syntax highlighting we do. But then the content.opf did have syntax highlighting and it was fast. The css and xhtml both have syntax highlighting and are slow. A CodeView widget is used for all of these. The only thing different is the type of Tab widget being used. Hmm.... KevinH

01-22-2016, 03:13 PM	#51
DiapDealer Grand Sorcerer Posts: 28,382 Karma: 203720150 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	I feel a breakthrough on the horizon.

01-22-2016, 03:43 PM	#52
KevinH Sigil Developer Posts: 8,501 Karma: 5703586 Join Date: Nov 2009 Device: many	Hi, Well we can rule out Syntax Highlighting as the culprit. I set it to NONE for FlowTabs.cpp and it had no effect. The scrolling was still quite slow for xhtml files. So I just have to figure out what a FlowTab and CSSTab are doing that the OPFTab and XMLTab is not. KevinH

01-23-2016, 12:31 PM	#55
DiapDealer Grand Sorcerer Posts: 28,382 Karma: 203720150 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	I still can't help but think that Sigil is doing something in Code View (or somewhere) that exacerbates the bug, no? Otherwise, why would it not manifest in Book View, or in the countless other Qt-based Linux apps that don't experience the same lag in text-selection? Don't get me wrong ... if you come up with a successful workaround for the bug, I'll be happier than a pig in slop. It's just that I still don't understand why Code View-only?

01-23-2016, 01:26 PM	#57
DiapDealer Grand Sorcerer Posts: 28,382 Karma: 203720150 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	Oh, man, it's looking good so far! Thanks for digging deep on this one.!

01-23-2016, 02:43 PM	#58
Toxaris Wizard Posts: 4,520 Karma: 121692313 Join Date: Oct 2009 Location: Heemskerk, NL Device: PRS-T1, Kobo Touch, Kobo Aura	So it was indeed a difference between Qt for Python and Qt for C++. Very good digging and hunting Kevin!

01-23-2016, 08:11 PM	#59
roger64 Wizard Posts: 2,624 Karma: 3120635 Join Date: Jan 2009 Device: Kindle PW3 (wifi)	@KevinH Wow! congratulations for squashing this sneaky one! Thank you also for making us live online this impressive hunting chase. Count me another happy pig.

01-23-2016, 08:57 PM	#60
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Thanks for tracking this down, Kevin! Enjoying my now much faster dev build.

Advert

Advert