[2 Bugs] + [Enhancement] Book Browser, CSS, URLs, Oh My! - Page 2

DiapDealer · 10-02-2018, 04:15 PM

@Tex: any more on bug number 2? Are you still able to consistently reproduce it on your end?

Never mind. Apparently, I can't read. You have to have Book View open when deleting the files and not Code View. I can consistently reproduce it on Windows with your Test epub. But there has to be something else in play ... because I can delete multiple css files from other epubs with identical circumstances (page open in Book View that has both css files linked whenever deleting). Nope. Wrong about that too.

100% reproduceable on Windows.

KevinH · 10-02-2018, 04:44 PM

Still can not reprduce it on a Mac. Have you tried with Linux?

DiapDealer · 10-02-2018, 05:04 PM

Quote:

Originally Posted by KevinH

Still can not reprduce it on a Mac. Have you tried with Linux?

I just did. I can't reproduce it at all on Linux. It only seems to affect Windows (I've reproduced it with the stock 0.9.10 as well as the latest build from master). Which is odd in itself. Makes me want to look for ifdefs.

KevinH · 10-02-2018, 07:19 PM

For bug 2 on Windows, my guess is when BookView has an html file open that links two or more css files, the Windows webkit is somehow placing a read only lock or is waiting to load/read that last css file that prevents Utility:SDeleteFile from actually succeeding. In QFile I see there are an error() and errorString() methods that should tell you why the delete was not successful. So you might want to add some qDebug print statements in Misc/Utility.cpp in SDeleteFile() when the result of the QFile remove is not successful so that we can see why and then try the bug2 testcase.

KevinH · 10-02-2018, 07:22 PM

BTW, my guess is you will see a permission based error or file in use error of some sort happening that is in Windows webkit that does not happen with Mac and Linux versions of webkit.

DiapDealer · 10-02-2018, 07:28 PM

Thanks for the pointer. I'll check it out.

DiapDealer · 10-02-2018, 08:28 PM

Quote:

Originally Posted by KevinH

BTW, my guess is you will see a permission based error or file in use error of some sort happening that is in Windows webkit that does not happen with Mac and Linux versions of webkit.

Exactly so. file.errorString() returns:

"The process cannot access the file because it is being used by another process."

KevinH · 10-02-2018, 10:30 PM

So something in Webkit has open that file and has not closed it. On Linux and Mac, all resources/images used by an xhtml page are read in and actually cached. You can even delete the resource/image files and until you flush the cache the xhtml page will look unchanged.

So somehow Webkit on Windows is not bothering to close a resource file (in this case a css file) after reading it into its cache or it is doing some kind of smart lazy loading. Not sure which. Either way this is a Qt Webkit bug that we either ignore or try to workaround in some way.

I just do not understand why the same permission bug does not happen due to Preview which is also a Webkit window.

In other words, why does BV cause this permission issue when CV with Preview does not?

Something is strange!

KevinH · 10-02-2018, 10:48 PM

From google, you may be able to use ProcessExplorer to find what other software has a handle to the file.

Common causes include antivirus scanners, shared drives like DropBox or Google Drive, continuous backup software, etc. So perhaps something on Windows looking inside temp is scanning the files.

I wonder if the same problem exists if we use Sigil preferences to set Sigil's temp directory to some place private to try to prevent that or if it is something happening inside of Qt/Sigil itself.

DiapDealer · 10-03-2018, 08:59 AM

Well, I've verified that changing the temp directory has no effect on the issue. That's about all I'll be able to test until later this evening when I get back to my dev machine.

I will say that the problem manifests on different versions of Windows on different hardware with different software installed (as well as different versions of Sigil, Qt5, and QtWebKit). Some machines are virtual--with no kind of anti-malware/scanning software installed--and some physical. With the direct connection to Book View being open being the trigger, I don't think it's a third-party software interference issue, myself.

I remember something similar happening a while back where multiple image files (or all image files) were selected on Windows (when one or more of the images were PNGs). The problem then turned out to be Book View and Preview being open at the same time. The workaround was to temporarily close Preview if it was open (at the same time as Book View) and relaunch it after the delete if necessary. That workaround is still in place, though, and Preview is, in fact, closing when trying to delete the multiple css files (if it and BV are open at the same time). Plus ... I've made sure that Preview is not open at all when testing this new bug.

I can't help but think there's some kind of weird connection there.

EDIT: I think the workaround evolved from this discussion. But in that case, Sigil was actually crashing when PNGs were being deleted with BV an Preview open at the same time.

KevinH · 10-03-2018, 09:31 AM

That old bug BTW was related to two different but incompatible versions of
libpng being on a system. Not sure how this happens inside webkit but if the wrong header file is found during a build, chaos with libpng differences causes memory corruption.

DiapDealer · 10-03-2018, 09:45 AM

It's got to be timing thing, no? Single css files can be deleted when BV is open (and even the first of the css files in a multiple file delete is successful with BV open). But the Windows-only thing has me baffled.

Like bug #1, something is interfering with the recursive delete (if it is recursive). Is Book View still trying to re-render from the first css file deletion while the second is still in the process of trying to be deleted?

KevinH · 10-03-2018, 10:45 AM

BookView itself should have only opened, read, and closed those resource files when loading the webpage. Since changing temp did not matter, my guess is that it is not antivirus or other external software. So it must be something in Qt or Sigil. The funny thing is if you go and load section4 which only includes one css file link, no problems happen according to Tex's original post.

BTW, BV and PV have the same parent class that does much of the work for both but the same problem does not happen in PV.

I will look inside the BV and tab classes to see if I can see any code that might be relevent.

KevinH · 10-03-2018, 11:22 AM

After looking at the code, I agree with you that it does appear to be a timing issue. Each html resource each time it is updated builds a list of linked resources (css and images) and sets up signals to each to catch deletes and underlying file changes so that html resource inside a flowtab can tell its BV window to be reload itself automatically.

The same issue does not happen in Preview as it does not use FlowTabs at all and just peridically reloads itself.

So here is the problematic sequence of events (I think):

1. the first linked css file is deleted and its deleted signal in heard by Section1 associated html resource (see ResourceObjects/HTMLResource.cpp) which in turn emits a LinkedResourceUpdated signal.

2. the FlowTab holding the BookView receives this LinkedResourceUpdated signal and its LinkedResourceModified slot is invoked (see src/Tabs/FlowTab.cpp) which in turn clears the webkit caches and if the tab is at the front tries to reload its html.

3. reloading the html causes webkit to try to load both the first and second css files into its cache in a background thread asynchronously.

4. meanwhile the return from the slot happens and the second css file is now in the process of being deleted but it cannot if that exact same css file is still trying to be read back into the webkit cache by another thread

So somehow we need to NOT invalidate the cache in webkit and try reloading until ALL deleted resources have been removed.

Right now, we have no structure in place to do that as each resource in isolation when it is being deleted emits the signal that starts this process, never knowing if they are the last delete of many or not.

I really can not see an easy way to handle this right now. Perhaps breaking the LinkedResourceUpdated signal and creating some all changes done signal of some sort.

This may take some thinking and again it is just another reason for BV to go away.

KevinH · 10-03-2018, 11:29 AM

Adding a caution, the above is just a guess based on my reading of the code since I can not recreate this issue on a Mac inside a debugger.

Perhaps, in Misc/Utility.cpp in SDeleteFile, we simply check the return code from remove and if needed delay/sleep a small amount of time and then try the delete one more time before giving up hoping that during the sleep BV's webkit thread finishes loading in its resources.

Alternatively, we could force BV to switch to CV during deletes and then switch back afterwards?

10-02-2018, 04:15 PM	#16
DiapDealer Grand Sorcerer Posts: 27,549 Karma: 193191846 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	@Tex: any more on bug number 2? Are you still able to consistently reproduce it on your end? Never mind. Apparently, I can't read. You have to have Book View open when deleting the files and not Code View. I can consistently reproduce it on Windows with your Test epub. But there has to be something else in play ... because I can delete multiple css files from other epubs with identical circumstances (page open in Book View that has both css files linked whenever deleting). Nope. Wrong about that too. 100% reproduceable on Windows. Last edited by DiapDealer; 10-02-2018 at 04:26 PM.

10-03-2018, 08:59 AM	#25
DiapDealer Grand Sorcerer Posts: 27,549 Karma: 193191846 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	Well, I've verified that changing the temp directory has no effect on the issue. That's about all I'll be able to test until later this evening when I get back to my dev machine. I will say that the problem manifests on different versions of Windows on different hardware with different software installed (as well as different versions of Sigil, Qt5, and QtWebKit). Some machines are virtual--with no kind of anti-malware/scanning software installed--and some physical. With the direct connection to Book View being open being the trigger, I don't think it's a third-party software interference issue, myself. I remember something similar happening a while back where multiple image files (or all image files) were selected on Windows (when one or more of the images were PNGs). The problem then turned out to be Book View and Preview being open at the same time. The workaround was to temporarily close Preview if it was open (at the same time as Book View) and relaunch it after the delete if necessary. That workaround is still in place, though, and Preview is, in fact, closing when trying to delete the multiple css files (if it and BV are open at the same time). Plus ... I've made sure that Preview is not open at all when testing this new bug. I can't help but think there's some kind of weird connection there. EDIT: I think the workaround evolved from this discussion. But in that case, Sigil was actually crashing when PNGs were being deleted with BV an Preview open at the same time. Last edited by DiapDealer; 10-03-2018 at 09:03 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
bugs in book browser	Reinhard	Sigil	25	07-31-2018 02:07 PM
Possible Error / Enhancement - Book Editor, Spell Check Right-Click Sub-Menu	WizkidTX	Editor	7	07-22-2015 09:33 PM
PRS-T1 Two tiny bugs when reading a book	opitzs	Sony Reader	10	07-08-2013 07:10 PM
[Program Enhancement] Adding Empty Book	Gunnerp245	Calibre	3	12-24-2012 07:32 AM
Enhancement (Delete formats from Book Details Panel)	JohnnyBook	Library Management	5	07-21-2012 08:17 AM

10-02-2018, 04:44 PM	#17
KevinH Sigil Developer Posts: 7,644 Karma: 5433388 Join Date: Nov 2009 Device: many	Still can not reprduce it on a Mac. Have you tried with Linux?

10-02-2018, 07:19 PM	#19
KevinH Sigil Developer Posts: 7,644 Karma: 5433388 Join Date: Nov 2009 Device: many	For bug 2 on Windows, my guess is when BookView has an html file open that links two or more css files, the Windows webkit is somehow placing a read only lock or is waiting to load/read that last css file that prevents Utility:SDeleteFile from actually succeeding. In QFile I see there are an error() and errorString() methods that should tell you why the delete was not successful. So you might want to add some qDebug print statements in Misc/Utility.cpp in SDeleteFile() when the result of the QFile remove is not successful so that we can see why and then try the bug2 testcase.

10-02-2018, 07:22 PM	#20
KevinH Sigil Developer Posts: 7,644 Karma: 5433388 Join Date: Nov 2009 Device: many	BTW, my guess is you will see a permission based error or file in use error of some sort happening that is in Windows webkit that does not happen with Mac and Linux versions of webkit.

10-02-2018, 07:28 PM	#21
DiapDealer Grand Sorcerer Posts: 27,549 Karma: 193191846 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	Thanks for the pointer. I'll check it out.

10-02-2018, 10:30 PM	#23
KevinH Sigil Developer Posts: 7,644 Karma: 5433388 Join Date: Nov 2009 Device: many	So something in Webkit has open that file and has not closed it. On Linux and Mac, all resources/images used by an xhtml page are read in and actually cached. You can even delete the resource/image files and until you flush the cache the xhtml page will look unchanged. So somehow Webkit on Windows is not bothering to close a resource file (in this case a css file) after reading it into its cache or it is doing some kind of smart lazy loading. Not sure which. Either way this is a Qt Webkit bug that we either ignore or try to workaround in some way. I just do not understand why the same permission bug does not happen due to Preview which is also a Webkit window. In other words, why does BV cause this permission issue when CV with Preview does not? Something is strange!

10-02-2018, 10:48 PM	#24
KevinH Sigil Developer Posts: 7,644 Karma: 5433388 Join Date: Nov 2009 Device: many	From google, you may be able to use ProcessExplorer to find what other software has a handle to the file. Common causes include antivirus scanners, shared drives like DropBox or Google Drive, continuous backup software, etc. So perhaps something on Windows looking inside temp is scanning the files. I wonder if the same problem exists if we use Sigil preferences to set Sigil's temp directory to some place private to try to prevent that or if it is something happening inside of Qt/Sigil itself.

10-03-2018, 09:31 AM	#26
KevinH Sigil Developer Posts: 7,644 Karma: 5433388 Join Date: Nov 2009 Device: many	That old bug BTW was related to two different but incompatible versions of libpng being on a system. Not sure how this happens inside webkit but if the wrong header file is found during a build, chaos with libpng differences causes memory corruption.

10-03-2018, 09:45 AM	#27
DiapDealer Grand Sorcerer Posts: 27,549 Karma: 193191846 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	It's got to be timing thing, no? Single css files can be deleted when BV is open (and even the first of the css files in a multiple file delete is successful with BV open). But the Windows-only thing has me baffled. Like bug #1, something is interfering with the recursive delete (if it is recursive). Is Book View still trying to re-render from the first css file deletion while the second is still in the process of trying to be deleted?

10-03-2018, 10:45 AM	#28
KevinH Sigil Developer Posts: 7,644 Karma: 5433388 Join Date: Nov 2009 Device: many	BookView itself should have only opened, read, and closed those resource files when loading the webpage. Since changing temp did not matter, my guess is that it is not antivirus or other external software. So it must be something in Qt or Sigil. The funny thing is if you go and load section4 which only includes one css file link, no problems happen according to Tex's original post. BTW, BV and PV have the same parent class that does much of the work for both but the same problem does not happen in PV. I will look inside the BV and tab classes to see if I can see any code that might be relevent.

10-03-2018, 11:22 AM	#29
KevinH Sigil Developer Posts: 7,644 Karma: 5433388 Join Date: Nov 2009 Device: many	After looking at the code, I agree with you that it does appear to be a timing issue. Each html resource each time it is updated builds a list of linked resources (css and images) and sets up signals to each to catch deletes and underlying file changes so that html resource inside a flowtab can tell its BV window to be reload itself automatically. The same issue does not happen in Preview as it does not use FlowTabs at all and just peridically reloads itself. So here is the problematic sequence of events (I think): 1. the first linked css file is deleted and its deleted signal in heard by Section1 associated html resource (see ResourceObjects/HTMLResource.cpp) which in turn emits a LinkedResourceUpdated signal. 2. the FlowTab holding the BookView receives this LinkedResourceUpdated signal and its LinkedResourceModified slot is invoked (see src/Tabs/FlowTab.cpp) which in turn clears the webkit caches and if the tab is at the front tries to reload its html. 3. reloading the html causes webkit to try to load both the first and second css files into its cache in a background thread asynchronously. 4. meanwhile the return from the slot happens and the second css file is now in the process of being deleted but it cannot if that exact same css file is still trying to be read back into the webkit cache by another thread So somehow we need to NOT invalidate the cache in webkit and try reloading until ALL deleted resources have been removed. Right now, we have no structure in place to do that as each resource in isolation when it is being deleted emits the signal that starts this process, never knowing if they are the last delete of many or not. I really can not see an easy way to handle this right now. Perhaps breaking the LinkedResourceUpdated signal and creating some all changes done signal of some sort. This may take some thinking and again it is just another reason for BV to go away.

Advert

Advert

10-03-2018, 11:29 AM	#30
KevinH Sigil Developer Posts: 7,644 Karma: 5433388 Join Date: Nov 2009 Device: many	Adding a caution, the above is just a guess based on my reading of the code since I can not recreate this issue on a Mac inside a debugger. Perhaps, in Misc/Utility.cpp in SDeleteFile, we simply check the return code from remove and if needed delay/sleep a small amount of time and then try the delete one more time before giving up hoping that during the sleep BV's webkit thread finishes loading in its resources. Alternatively, we could force BV to switch to CV during deletes and then switch back afterwards?