Slow switching of Views - Long thread

[theducks]

Things that need to be 'calculated', rather than looked up (read direct), will slow things down.

Any formula (column value), Any column Icon or color needs additional processing after the DATA from the DB has been read. (I will assume the same applies to Grid View, which I do not use)

[BetterRed]

Since we're talking about Virtual Library switching and we've eliminated most everything else, my suspicions have fallen on the VL search expressions, mine are simple. I guess there are performance implications if they're complex.

One way to test this would be to switch directly from a restricted VL to the All Books VL. For me that's Ctrl+Alt+Up-arrow. But that's a customised setting.

On Windows the default shortcut, Ctrl+Esc, doesn't work because Microsoft uses it to show the Start Menu. As an ardent user of keyboard shortcuts why anyone would use that sequence for anything other than 'Get me outta here… Now!' escapes me.

Maybe others know of search expression examples that could be used in a VL, which have performance implications. chaley?

BR

[chaley]

Originally Posted by BetterRed

Maybe others know of search expression examples that could be used in a VL, which have performance implications. chaley?

BR

For a library with approx 250 books, a search expression shouldn't be slow unless they reference custom columns with many references to disk information. And even then it shouldn't be as slow as the OP reports.

One exception: I have seen slow searches when the comments columns contains huge amounts of text, the searches aren't prefixed, and the columns-to-search when prefixes aren't used isn't set (Preferences / Search / Limit the searched metadata. In one case the person was copying most of the book into the comments.

[BetterRed]

Originally Posted by chaley

...
One exception: I have seen slow searches when the comments columns contains huge amounts of text, the searches aren't prefixed, and the columns-to-search when prefixes aren't used isn't set (Preferences / Search / Limit the searched metadata. In one case the person was copying most of the book into the comments.

Yes, I thought of that after I posted. The OP mentioned a "mm_annotations" custom column, which I see is also long text. I suggest he excludes that column too.

If that solves the problem I'll move this convo to fresh a Lib Mgt sub-forum thread.

BR

[jnikkir]

Originally Posted by BetterRed

Yes, I thought of that after I posted. The OP mentioned a "mm_annotations" custom column, which I see is also long text. I suggest he excludes that column too.

Tried that... no dice. The annotations aren't that long, they just include highlighted passages (maybe 4 max per book), and they're only present in 10-20 books.

Originally Posted by BetterRed

Since we're talking about Virtual Library switching and we've eliminated most everything else, my suspicions have fallen on the VL search expressions, mine are simple. I guess there are performance implications if they're complex.

Switching virtual libraries/VL tabs is instantaneous. What hangs is the loading of the VM View. For example, I'll click on a VL tab, immediately be able to see all the listed books, but it's still showing the column setup of the VM View from the previous library; then I'll wait a few seconds and then the correct columns for that Virtual Library's VM view kick in. The lag happens when switching between VM Views in the All Books tab, too.

All my search expressions are simple:

• not #vendor:"=AO3" not tags:"=Manga"
• tags:"Manga"
• #vendor:"AO3"

I've run a ton of tests in the portable version: loaded a copy of my current library, tried removing all kinds of columns and metadata, tried creating smaller new libraries from scratch and adding custom columns to those, then making different VM views...

The fastest setup was when I dropped 25 eBook files directly into a new, clean library; didn't add any custom columns; made some VM views. Switching between those VM views was fast. Not instant, but faster than a second.

I also tried making a new library and importing my full original library (with settings) into it, then deleting all but 30 books. Still slow. A little faster than my original install, but still a couple of seconds.


Honestly I'm driving myself a bit batty with all the testing, so I'm probably just going to give up and live with it.

I really appreciate everyone's help and suggestions, though!

[BetterRed]

Originally Posted by jnikkir

. . .
Switching virtual libraries/VL tabs is instantaneous. What hangs is the loading of the VM View. For example, I'll click on a VL tab, immediately be able to see all the listed books, but it's still showing the column setup of the VM View from the previous library; then I'll wait a few seconds and then the correct columns for that Virtual Library's VM view kick in.

That's bizzaro.

Originally Posted by jnikkir

The lag happens when switching between VM Views in the All Books tab, too.

That's bizzaro².

Originally Posted by jnikkir

Honestly I'm driving myself a bit batty with all the testing, so I'm probably just going to give up and live with it.

I really appreciate everyone's help and suggestions, though!

One other suggestion, have you tried running calibre in Windows Safe Mode. I can't imagine how, but maybe a 3rd party gadget/extension/add-on is sticking its spoon into the pudding,

BR

[JimmXinu]

I don't know if it will actually help any, but here's a quick and dirty version of VM that spits out some times between different events.

You have to run in debug to see the output and look at the code to see what it means, but it should pinpoint exactly where the time is going.

From VM's point of view that is.

I have every expectation that the longest time for basically everyone will be where I see it: "7.4" self.gui.library_view.apply_state(...) which sets the viewable columns and sorts.

Look for the self.print_timediff(...) calls in action.py.

[jnikkir]

Originally Posted by BetterRed

One other suggestion, have you tried running calibre in Windows Safe Mode. I can't imagine how, but maybe a 3rd party gadget/extension/add-on is sticking its spoon into the pudding

Good suggestion. I tried this, but got an error. It said my database is corrupt... so I couldn't access that library. It doesn't do that when I'm not in Safe Mode... Is this something I need to look into?? =/

Traceback (most recent call last):
  File "calibre\gui2\ui.py", line 853, in library_moved
  File "calibre\db\legacy.py", line 187, in __init__
  File "calibre\db\legacy.py", line 44, in create_backend
  File "calibre\db\backend.py", line 461, in __init__
  File "calibre\db\backend.py", line 1213, in user_version
  File "calibre\db\backend.py", line 926, in conn
  File "calibre\db\backend.py", line 339, in __init__
apsw.CantOpenError: CantOpenError: unable to open database file

Originally Posted by JimmXinu

I don't know if it will actually help any, but here's a quick and dirty version of VM that spits out some times between different events.

You have to run in debug to see the output and look at the code to see what it means, but it should pinpoint exactly where the time is going.

From VM's point of view that is.

I have every expectation that the longest time for basically everyone will be where I see it: "7.4" self.gui.library_view.apply_state(...) which sets the viewable columns and sorts.

Look for the self.print_timediff(...) calls in action.py.

Okay wow, thank you SO MUCH for doing this! I installed, manually CTRL+Tabbed through my VM views with no VL open, then clicked though my VL tabs (which have Action Chains associated w/ them), to compare the speeds between two. It actually does look like maybe the Action Chain switches have the biggest lag? I could have sworn there wasn’t much difference, but if I’m reading this right, it looks like there is. Maybe I've been looking for the cause of the lag in entirely the wrong place...

calibre Debug log
calibre 5.38 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.22000 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.22000')
Python 3.8.5
Windows: ('10', '10.0.22000', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 47, 0) && DeDRM (10, 0, 2) && Package KFX (from KFX Input) (1, 47, 0) && Action Chains (1, 16, 3) && Annotations (1, 16, 0) && Count Pages (1, 11, 2) && FanFicFare (4, 10, 0) && Generate Cover (2, 1, 0) && KFX metadata reader (from KFX Input) (1, 47, 0) && KFX Input (1, 47, 0) && Set KFX metadata (from KFX Output) (1, 59, 0) && KFX Output (1, 59, 0) && KindleUnpack - The Plugin (0, 83, 1) && Obok DeDRM (10, 0, 0) && Reading List (1, 8, 0) && SmartEject (2, 4, 0) && View Manager (1, 7, 0) && WordDumb (3, 18, 4)
calibre 5.38 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.22000 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.22000')
Python 3.8.5
Windows: ('10', '10.0.22000', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 47, 0) && DeDRM (10, 0, 2) && Package KFX (from KFX Input) (1, 47, 0) && Action Chains (1, 16, 3) && Annotations (1, 16, 0) && Count Pages (1, 11, 2) && FanFicFare (4, 10, 0) && Generate Cover (2, 1, 0) && KFX metadata reader (from KFX Input) (1, 47, 0) && KFX Input (1, 47, 0) && Set KFX metadata (from KFX Output) (1, 59, 0) && KFX Output (1, 59, 0) && KindleUnpack - The Plugin (0, 83, 1) && Obok DeDRM (10, 0, 0) && Reading List (1, 8, 0) && SmartEject (2, 4, 0) && View Manager (1, 7, 0) && WordDumb (3, 18, 4)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 95.81139489194499 x 95.9160839160839
Using calibre Qt style: True
[0.00] Starting up...
[0.01] Showing splash screen...
[0.44] splash screen shown
[0.44] Initializing db...
[0.47] db initialized
[0.47] Constructing main UI...
DEBUG:    0.0 Annotations plugin: Error loading libiMobileDevice. This hasn't worked for a while, and is blacklisted in calibre v3.
DEBUG:    0.0 Annotations plugin: Error is:  No module named 'calibre.devices.idevice'
DEBUG:    0.0 Annotations::reader_app_support.py - loading translations
DEBUG:    0.0 Annotations::common_utils.py - loading translations
DEBUG:    0.0 Annotations::appearance.py - loading translations
DEBUG:    0.0 Annotations::config.py - loading translations
DEBUG:    0.0 Annotations::annotated_books.py - loading translations
DEBUG:    0.0 Annotations::action.py - loading translations
DEBUG:    0.0 obok::utilities.py - loading translations
DEBUG:    0.0 obok::dialogs.py - loading translations
DEBUG:    0.0 obok::config.py - loading translations
DEBUG:    0.0 obok::action_err.py - loading translations
[2.41] main UI initialized...
[2.41] Hiding splash screen
Action chains: _call_plugins_action_chains_resources(): start
Action Chains: _call_plugins_action_chains_resources(): finished
Action chains: formulas: running on_templates_update()
Action chains: _call_plugins_on_modules_update(): start
Action Chains: _call_plugins_on_modules_update(): finished
Starting QuickView
[3.98] splash screen hidden
[3.98] Started up in 3.98 seconds with 227 books
switch_view(All Books)
1:0.0
2:0.0
3:0.0
4:0.0
5:0.0
6:0.0
7.1:0.0
7.2:0.0
7.3:0.0
7.4:0.0
7.5:0.0
7.6:0.0
7.10:0.0
7:0.0
8:0.03127121925354004
9:0.0
10:0.0
switch_view(Fanfic)
1:0.0
2:0.015655994415283203
3:0.06296420097351074
4:0.0
5:0.0
6:0.0
7.1:0.0
7.2:0.0
7.3:0.0
7.4:1.192173957824707
7.5:0.07853269577026367
7.6:0.0
7.10:0.0
7:0.0
8:0.031271934509277344
9:0.0
10:0.0
switch_view(Manga)
1:0.0
2:0.0
3:0.10957932472229004
4:0.0
5:0.0
6:0.0
7.1:0.0
7.2:0.0
7.3:0.0
7.4:1.117962121963501
7.5:0.0
7.6:0.0
7.10:0.0
7:0.0
8:0.03127408027648926
9:0.0
10:0.0
switch_view(Prose)
1:0.0010290145874023438
2:0.0
3:0.06935787200927734
4:0.0
5:0.0
6:0.0
7.1:0.0
7.2:0.0
7.3:0.0
7.4:0.12209963798522949
7.5:0.015333890914916992
7.6:0.0
7.10:0.0
7:0.0
8:0.031244516372680664
9:0.0
10:0.0
switch_view(All Books)
1:0.0
2:0.0
3:0.11008644104003906
4:0.0
5:0.0
6:0.0
7.1:0.0
7.2:0.0
7.3:0.0
7.4:0.7217462062835693
7.5:0.0
7.6:0.0
7.10:0.0
7:0.0
8:0.03124237060546875
9:0.0
10:0.0
switch_view(Manga)
1:0.0
2:0.0
3:0.09266543388366699
4:0.0
5:0.0
6:0.0
7.1:0.0
7.2:0.0
7.3:0.0
7.4:0.8940434455871582
7.5:0.0
7.6:0.0
7.10:0.0
7:0.0
8:0.03124237060546875
9:0.0
10:0.0
switch_view(Fanfic)
1:0.0
2:0.0
3:0.07855987548828125
4:0.0
5:0.0
6:0.0
7.1:0.0
7.2:0.0
7.3:0.0
7.4:1.535933256149292
7.5:0.0
7.6:0.0
7.10:0.0
7:0.0
8:0.03168678283691406
9:0.0
10:0.0
switch_view(Prose)
1:0.0
2:0.0
3:0.15700888633728027
4:0.0
5:0.0
6:0.0
7.1:0.0
7.2:0.0
7.3:0.0
7.4:1.6943953037261963
7.5:0.015617609024047852
7.6:0.0
7.10:0.0
7:0.0
8:0.03121209144592285
9:0.0
10:0.005539655685424805
switch_view(All Books)
1:0.0
2:0.0
3:0.09420442581176758
4:0.0
5:0.0
6:0.0
7.1:0.0
7.2:0.0
7.3:0.0
7.4:0.7216336727142334
7.5:0.0
7.6:0.0
7.10:0.0
7:0.0
8:0.03173184394836426
9:0.0
10:0.0
Action Chains: Starting event: VL Tab Changed
Action chains: formulas: running on_templates_update()
calling method (init_cache) of obj (selection_count)
Action chains: formulas: running on_templates_update()
calling method (flush_cache) of obj (selection_count)
Action Chains: Starting chain: VL View
Action Chains: VL View: starting action No. 1: Switch To VL View
debug1: vl: Fanfiction
debug2: view: Fanfic
7.1:5.5109899044036865
7.2:0.0
7.3:0.0
7.4:2.2270941734313965
7.5:0.0
7.6:0.0
7.10:0.0
Action Chains: chain (VL View) finished in: 0:00:02.496662
Action Chains: Is db modified by event: True
Action Chains: event (VL Tab Changed) finished in: 0:00:02.513281
Action Chains: Signal vl_tab_changed emitted
Action Chains: Starting event: VL Tab Changed
Action chains: formulas: running on_templates_update()
calling method (init_cache) of obj (selection_count)
Action chains: formulas: running on_templates_update()
calling method (flush_cache) of obj (selection_count)
Action Chains: Starting chain: VL View
Action Chains: VL View: starting action No. 1: Switch To VL View
debug1: vl: Manga
debug2: view: Manga
7.1:2.437694787979126
7.2:0.0
7.3:0.0
7.4:1.8519632816314697
7.5:0.0
7.6:0.0
7.10:0.0
Action Chains: chain (VL View) finished in: 0:00:02.040398
Action Chains: Is db modified by event: True
Action Chains: event (VL Tab Changed) finished in: 0:00:02.056019
Action Chains: Signal vl_tab_changed emitted
Action Chains: Starting event: VL Tab Changed
Action chains: formulas: running on_templates_update()
calling method (init_cache) of obj (selection_count)
Action chains: formulas: running on_templates_update()
calling method (flush_cache) of obj (selection_count)
Action Chains: Starting chain: VL View
Action Chains: VL View: starting action No. 1: Switch To VL View
debug1: vl: Prose
debug2: view: Prose
7.1:3.3334391117095947
7.2:0.0
7.3:0.0
7.4:0.21960043907165527
7.5:0.0
7.6:0.0
7.10:0.0
Action Chains: chain (VL View) finished in: 0:00:00.423471
Action Chains: Is db modified by event: True
Action Chains: event (VL Tab Changed) finished in: 0:00:00.439093
Action Chains: Signal vl_tab_changed emitted
Action Chains: Starting event: VL Tab Changed
Action chains: formulas: running on_templates_update()
calling method (init_cache) of obj (selection_count)
Action chains: formulas: running on_templates_update()
calling method (flush_cache) of obj (selection_count)
Action Chains: Starting chain: VL View
Action Chains: VL View: starting action No. 1: Switch To VL View
debug1: vl: _ALL_BOOKS
debug2: view: All Books
7.1:3.4159934520721436
7.2:0.0
7.3:0.0
7.4:0.7062489986419678
7.5:0.0
7.6:0.0
7.10:0.0
Action Chains: chain (VL View) finished in: 0:00:01.015570
Action Chains: Is db modified by event: True
Action Chains: event (VL Tab Changed) finished in: 0:00:01.033187
Action Chains: Signal vl_tab_changed emitted
Action Chains: Starting event: VL Tab Changed
Action chains: formulas: running on_templates_update()
calling method (init_cache) of obj (selection_count)
Action chains: formulas: running on_templates_update()
calling method (flush_cache) of obj (selection_count)
Action Chains: Starting chain: VL View
Action Chains: VL View: starting action No. 1: Switch To VL View
debug1: vl: Manga
debug2: view: Manga
7.1:6.749933481216431
7.2:0.0
7.3:0.0
7.4:1.3303160667419434
7.5:0.015621662139892578
7.6:0.0
7.10:0.0
Action Chains: chain (VL View) finished in: 0:00:01.506669
Action Chains: Is db modified by event: True
Action Chains: event (VL Tab Changed) finished in: 0:00:01.526304
Action Chains: Signal vl_tab_changed emitted
Action Chains: Starting event: VL Tab Changed
Action chains: formulas: running on_templates_update()
calling method (init_cache) of obj (selection_count)
Action chains: formulas: running on_templates_update()
calling method (flush_cache) of obj (selection_count)
Action Chains: Starting chain: VL View
Action Chains: VL View: starting action No. 1: Switch To VL View
debug1: vl: _ALL_BOOKS
debug2: view: All Books
7.1:2.164708375930786
7.2:0.0
7.3:0.0
7.4:0.9017114639282227
7.5:0.007542610168457031
7.6:0.0
7.10:0.0
Action Chains: chain (VL View) finished in: 0:00:01.199462
Action Chains: Is db modified by event: True
Action Chains: event (VL Tab Changed) finished in: 0:00:01.217078
Action Chains: Signal vl_tab_changed emitted
Action Chains: Starting event: VL Tab Changed
Action chains: formulas: running on_templates_update()
calling method (init_cache) of obj (selection_count)
Action chains: formulas: running on_templates_update()
calling method (flush_cache) of obj (selection_count)
Action Chains: Starting chain: VL View
Action Chains: VL View: starting action No. 1: Switch To VL View
debug1: vl: Prose
debug2: view: Prose
7.1:2.987982988357544
7.2:0.0
7.3:0.0
7.4:1.4749689102172852
7.5:0.0
7.6:0.0
7.10:0.0
Action Chains: chain (VL View) finished in: 0:00:01.695132
Action Chains: Is db modified by event: True
Action Chains: event (VL Tab Changed) finished in: 0:00:01.710650
Action Chains: Signal vl_tab_changed emitted
Action Chains: Starting event: VL Tab Changed
Action chains: formulas: running on_templates_update()
calling method (init_cache) of obj (selection_count)
Action chains: formulas: running on_templates_update()
calling method (flush_cache) of obj (selection_count)
Action Chains: Starting chain: VL View
Action Chains: VL View: starting action No. 1: Switch To VL View
debug1: vl: Fanfiction
debug2: view: Fanfic
7.1:2.9287922382354736
7.2:0.0
7.3:0.0
7.4:2.2907092571258545
7.5:0.012963533401489258
7.6:0.0
7.10:0.0
Action Chains: chain (VL View) finished in: 0:00:02.619679
Action Chains: Is db modified by event: True
Action Chains: event (VL Tab Changed) finished in: 0:00:02.641620
Action Chains: Signal vl_tab_changed emitted
Action Chains: Starting event: VL Tab Changed
Action chains: formulas: running on_templates_update()
calling method (init_cache) of obj (selection_count)
Action chains: formulas: running on_templates_update()
calling method (flush_cache) of obj (selection_count)
Action Chains: Starting chain: VL View
Action Chains: VL View: starting action No. 1: Switch To VL ViewTraceback (most recent call last):
  File "calibre\gui2\ui.py", line 135, in __init__
  File "calibre\gui2\ui.py", line 153, in init_iaction
  File "calibre_plugins.fanficfare_plugin.__init__", line 121, in load_actual_plugin
  File "calibre\customize\__init__.py", line 624, in load_actual_plugin
  File "importlib\__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "calibre\customize\zipplugin.py", line 191, in exec_module
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 80, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "zipimport.py", line 259, in load_module
  File "C:\Users\jnikk\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\__init__.py", line 132, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1342, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1316, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1297, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 414, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 649, in spec_from_file_location
  File "zipimport.py", line 191, in get_filename
  File "zipimport.py", line 709, in _get_module_code
  File "zipimport.py", line 536, in _get_data
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\jnikk\\AppData\\Roaming\\calibre\\plugins\\FanFicFare.zip'

debug1: vl: Prose
debug2: view: Prose
7.1:2.425004005432129
7.2:0.0
7.3:0.0
7.4:2.5235352516174316
7.5:0.0
7.6:0.0
7.10:0.0
Action Chains: chain (VL View) finished in: 0:00:02.805604
Action Chains: Is db modified by event: True
Action Chains: event (VL Tab Changed) finished in: 0:00:02.821227
Action Chains: Signal vl_tab_changed emitted

[BetterRed]

Originally Posted by jnikkir

Good suggestion. I tried this, but got an error. It said my database is corrupt... so I couldn't access that library. It doesn't do that when I'm not in Safe Mode... Is this something I need to look into?? =/

I doubt the database is corrupt, more likely something in Safe mode is preventing the db from being opened - which is the absolute converse of normal expectations - bizarro³ on steroids with flummox on the side.

If you have one of the SQLite utilities you could try poking the library database with it in Safe Mode - I use this one ==>> DB Browser for SQLite

BR

[JimmXinu]

Originally Posted by jnikkir

Okay wow, thank you SO MUCH for doing this! I installed, manually CTRL+Tabbed through my VM views with no VL open, then clicked though my VL tabs (which have Action Chains associated w/ them), to compare the speeds between two. It actually does look like maybe the Action Chain switches have the biggest lag? I could have sworn there wasn’t much difference, but if I’m reading this right, it looks like there is. Maybe I've been looking for the cause of the lag in entirely the wrong place...

When you were switching Views, the longest time (up to ~1.5s) is during self.gui.library_view.apply_state(...) , as expected. That's still a bit longer than I'd honestly expect, but not insanely so.

When doing VL changes, it looks like Action Chains is calling VM's apply_column_and_sort() instead of switch_view(), which I would have expected.

For one thing, it means that the "7.1" time can't be trusted--VM is resetting the time at the start of switch_view(). What you see there for 7.1 includes all the wall-clock time passed since the last interaction. You may be right, but you can't trust this output to tell you.

(Presumably Action Chains is also reading the VM config data needed to call apply_column_and_sort()? I don't know what it does...)


I'm not usually involved in debugging whole libraries. Is it appropriate to ask for a zipped copy of the library to look at? If it's not too private or too copyrighted, that is.

[ownedbycats]

Originally Posted by BetterRed

If you have one of the SQLite utilities you could try poking the library database with it in Safe Mode - I use this one ==>> DB Browser for SQLite

Personally, I like SQLiteStudio.

[jnikkir]

Originally Posted by BetterRed

I doubt the database is corrupt, more likely something in Safe mode is preventing the db from being opened - which is the absolute converse of normal expectations - bizarro³ on steroids with flummox on the side.

If you have one of the SQLite utilities you could try poking the library database with it in Safe Mode - I use this one ==>> DB Browser for SQLite

BR

I'm just going with the theory that I screwed something up somewhere because I really don't know what I'm doing I don't know anything about how to use SQLite or similar... I feel like I'm wading into waters that are too deep for my swimming ability!

Originally Posted by JimmXinu

When doing VL changes, it looks like Action Chains is calling VM's apply_column_and_sort() instead of switch_view(), which I would have expected.

For one thing, it means that the "7.1" time can't be trusted--VM is resetting the time at the start of switch_view(). What you see there for 7.1 includes all the wall-clock time passed since the last interaction. You may be right, but you can't trust this output to tell you.

(Presumably Action Chains is also reading the VM config data needed to call apply_column_and_sort()? I don't know what it does...)

Hmmm, okay. That does explain why some of those numbers are so high... I did think that was weird. It was calculating how long it took me to click on the next library, not from my library clicks.

So this is my Action Chains script... I just copy/pasted from this post. Is there something I should change in that, to make it call switch_view() instead?

PHP Code:

from calibre_plugins.action_chains.actions.base import ChainAction

class MyAction(ChainAction):

    # VL View
    name = 'VL View Action'

    def run(self, gui, settings, chain):
        pass
import copy
from functools import partial

from qt.core import (QApplication, Qt, QWidget, QVBoxLayout, QHBoxLayout, QGridLayout,
                     QGroupBox, QAbstractTableModel, QModelIndex, QSizePolicy,
                     QToolButton, QSpacerItem, QIcon, QBrush, pyqtSignal)

from calibre import prints
from calibre.constants import DEBUG
from calibre.gui2 import error_dialog
from calibre.gui2.widgets2 import Dialog

from calibre_plugins.action_chains.actions.base import ChainAction
from calibre_plugins.action_chains.common_utils import get_icon, ViewLog
from calibre_plugins.action_chains.database import get_valid_vls
from calibre_plugins.action_chains.gui.delegates import ComboDelegate
from calibre_plugins.action_chains.gui.models import UP, DOWN
from calibre_plugins.action_chains.gui.views import TableView
import calibre_plugins.action_chains.config as cfg

ALL_BOOKS = '_ALL_BOOKS'
KEY_TABS_VIEWS_TABLE_STATE = 'tabsViewsTableStates'

def get_vls(db):
    vls = get_valid_vls(db)
    vls.insert(0, ALL_BOOKS)
    return vls

def view_manager_views(gui):
    try:
        import calibre_plugins.view_manager.config as vm_cfg
        views = vm_cfg.get_library_config(gui.current_db)[vm_cfg.KEY_VIEWS]
        return views
    except:
        import traceback
        print(traceback.format_exc())
        return []

class TabsModel(QAbstractTableModel):

    error = pyqtSignal(str, str)

    def __init__(self, plugin_action, tabs_config=[]):
        QAbstractTableModel.__init__(self)
        self.tabs_config = tabs_config
        self.plugin_action = plugin_action
        self.gui = self.plugin_action.gui
        self.db = self.gui.current_db
        self.col_map = ['tab_name','view_name','errors']
        self.editable_columns = ['tab_name','view_name']
        #self.hidden_cols = ['errors']
        self.hidden_cols = []
        self.col_min_width = {
            'tab_name': 300,
            'view_name': 300
        }
        all_headers = [_('VL'), _('View'), _('errors')]
        self.headers = all_headers

    def rowCount(self, parent):
        if parent and parent.isValid():
            return 0
        return len(self.tabs_config)

    def columnCount(self, parent):
        if parent and parent.isValid():
            return 0
        return len(self.headers)

    def headerData(self, section, orientation, role):
        if role == Qt.DisplayRole and orientation == Qt.Horizontal:
            return self.headers[section]
        elif role == Qt.DisplayRole and orientation == Qt.Vertical:
            return section + 1
        return None

    def data(self, index, role):
        if not index.isValid():
            return None;
        row, col = index.row(), index.column()
        if row < 0 or row >= len(self.tabs_config):
            return None
        tab_config = self.tabs_config[row]
        col_name = self.col_map[col]
        value = tab_config.get(col_name, '')
        error = tab_config.get('errors', '')

        if role in [Qt.DisplayRole, Qt.UserRole, Qt.EditRole]:
            if col_name == 'errors':
                if error:
                    return error
            else:
                return value

        elif role == Qt.DecorationRole:
            if col_name == 'errors':
                if error:
                    return QIcon(get_icon('dialog_error.png'))
                
        elif role == Qt.ToolTipRole:
            if col_name == 'errors':
                if error:
                    return error

        elif role == Qt.ForegroundRole:
            color = None
            if error:
                color = Qt.red
            if color is not None:
                return QBrush(color)

        return None

    def setData(self, index, value, role):
        done = False

        row, col = index.row(), index.column()
        tab_config = self.tabs_config[row]
        val = str(value).strip()
        col_name = self.col_map[col]
        
        if role == Qt.EditRole:
            # make sure no duplicate event entries
            if col_name == 'tab_name':
                old_name = self.data(index, Qt.DisplayRole)
                names = self.get_names()
                if old_name in names:
                    names.remove(old_name)
                if val in names:
                    msg = _('Duplicate vls')
                    details = _('Name ({}) is used in more than one entry'.format(val))
                    self.error.emit(msg, details)
                else:
                    tab_config[col_name] = val
            else:
                tab_config[col_name] = val
            done = True
            
        return done

    def flags(self, index):
        flags = QAbstractTableModel.flags(self, index)
        if index.isValid():
            tab_config = self.tabs_config[index.row()]
            col_name = self.col_map[index.column()]
            if col_name in self.editable_columns:
                flags |= Qt.ItemIsEditable
        return flags

    def insertRows(self, row, count, idx):
        self.beginInsertRows(QModelIndex(), row, row + count - 1)
        for i in range(0, count):
            tab_config = {}
            tab_config['tab_name'] = ''
            tab_config['view_name'] = ''
            self.tabs_config.insert(row + i, tab_config)
        self.endInsertRows()
        return True

    def removeRows(self, row, count, idx):
        self.beginRemoveRows(QModelIndex(), row, row + count - 1)
        for i in range(0, count):
            self.tabs_config.pop(row + i)
        self.endRemoveRows()
        return True

    def move_rows(self, rows, direction=DOWN):
        srows = sorted(rows, reverse=direction == DOWN)
        for row in srows:
            pop = self.tabs_config.pop(row)
            self.tabs_config.insert(row+direction, pop)
        self.layoutChanged.emit()

    def get_names(self):
        names = []
        col = self.col_map.index('tab_name')
        for row in range(self.rowCount(QModelIndex())):
            index = self.index(row, col, QModelIndex())
            name = self.data(index, Qt.DisplayRole)
            # empty name belong to separators, dont include
            if name:
                names.append(name)
        return names

    def validate(self):
        for tab_config in self.tabs_config:
            errors = []
            tab_name = tab_config['tab_name']
            view_name = tab_config['view_name']
            if tab_name not in get_vls(self.db):
                errors.append(_('VL is not available'))
            if view_name not in view_manager_views(self.gui):
                errors.append(_('View is not available'))
            if errors:
                tab_config['errors'] = ' ::: '.join(errors)

class TabsTable(TableView):

    def __init__(self, parent):
        TableView.__init__(self, parent)
        self.plugin_action = parent.plugin_action
        self.doubleClicked.connect(self._on_double_clicked)
        self.gui = self.plugin_action.gui
        self.db = self.gui.current_db
        self.horizontalHeader().setStretchLastSection(False)
        #self.setShowGrid(False)

    def set_model(self, _model):
        self.setModel(_model)
        _model.error.connect(lambda *args: error_dialog(self, *args, show=True))
        self.col_map = _model.col_map

        # Hide columns
        for col_name in _model.hidden_cols:
            col = self.col_map.index(col_name)
            self.setColumnHidden(col, True)

        self.tabs_delegate = ComboDelegate(self, get_vls(self.db))
        self.setItemDelegateForColumn(self.col_map.index('tab_name'), self.tabs_delegate)

        self.views_delegate = ComboDelegate(self, view_manager_views(self.gui))
        self.setItemDelegateForColumn(self.col_map.index('view_name'), self.views_delegate)

        self.resizeColumnsToContents()
        # Make sure every other column has a minimum width
        for col_name, width in _model.col_min_width.items():
            col = self.col_map.index(col_name)
            self._set_minimum_column_width(col, width)

    def _on_double_clicked(self, index):
        m = self.model()
        col_name = m.col_map[index.column()]
        if col_name == 'errors':
            tab_config = m.tabs_config[index.row()]
            details = tab_config.get('errors', '')
            self._view_error_details(details)

    def _view_error_details(self, details):
        ViewLog(_('Errors details'), details, self)

class ConfigWidget(QWidget):

    def __init__(self, plugin_action):
        QWidget.__init__(self)
        self.plugin_action = plugin_action
        self.gui = plugin_action.gui
        self.db = self.gui.current_db
        #self.tabs_config = tabs_config
        self._init_controls()

    def _init_controls(self):
        self.setWindowTitle(_('Configure views'))
        self.l = l = QVBoxLayout()
        self.setLayout(l)

        settings_l = QGridLayout()
        l.addLayout(settings_l)

        _table_gb = QGroupBox(_('Views'))
        _table_l = QHBoxLayout()
        _table_gb.setLayout(_table_l)
        l.addWidget(_table_gb)
        
        self._table = TabsTable(self)
        _table_l.addWidget(self._table)
        
        _model = self._model = TabsModel(self.plugin_action)
        _model.validate()
        self._table.set_model(_model)
        self._table.selectionModel().selectionChanged.connect(self._on_table_selection_change)
        
        # restore table state
        state = cfg.plugin_prefs.get(KEY_TABS_VIEWS_TABLE_STATE, None)
        if state:
            self._table.apply_state(state)

        # Add a vertical layout containing the the buttons to move up/down etc.
        button_layout = QVBoxLayout()
        _table_l.addLayout(button_layout)
        
        move_up_button = self.move_up_button = QToolButton(self)
        move_up_button.setToolTip(_('Move row up'))
        move_up_button.setIcon(QIcon(I('arrow-up.png')))
        button_layout.addWidget(move_up_button)
        spacerItem1 = QSpacerItem(20, 40, QSizePolicy.Minimum, QSizePolicy.Expanding)
        button_layout.addItem(spacerItem1)

        add_button = self.add_button = QToolButton(self)
        add_button.setToolTip(_('Add row'))
        add_button.setIcon(QIcon(I('plus.png')))
        button_layout.addWidget(add_button)
        spacerItem2 = QSpacerItem(20, 40, QSizePolicy.Minimum, QSizePolicy.Expanding)
        button_layout.addItem(spacerItem2)

        delete_button = self.delete_button = QToolButton(self)
        delete_button.setToolTip(_('Delete row'))
        delete_button.setIcon(QIcon(I('minus.png')))
        button_layout.addWidget(delete_button)
        spacerItem4 = QSpacerItem(20, 40, QSizePolicy.Minimum, QSizePolicy.Expanding)
        button_layout.addItem(spacerItem4)

        move_down_button = self.move_down_button = QToolButton(self)
        move_down_button.setToolTip(_('Move row down'))
        move_down_button.setIcon(QIcon(I('arrow-down.png')))
        button_layout.addWidget(move_down_button)

        move_up_button.clicked.connect(partial(self._table.move_rows,UP))
        move_down_button.clicked.connect(partial(self._table.move_rows,DOWN))
        add_button.clicked.connect(self._table.add_row)
        delete_button.clicked.connect(self._table.delete_rows)
        
        self._on_table_selection_change()

        self.setMinimumSize(400, 300)
        l.addStretch(1)

    def _on_table_selection_change(self):
        sm = self._table.selectionModel()
        selection_count = len(sm.selectedRows())
        self.delete_button.setEnabled(selection_count > 0)
        self.move_up_button.setEnabled(selection_count > 0)
        self.move_down_button.setEnabled(selection_count > 0)

    def save_table_state(self):
        # save table state
        cfg.plugin_prefs[KEY_TABS_VIEWS_TABLE_STATE] = self._table.get_state()

    def load_settings(self, settings):
        self._model.tabs_config = settings['tabs_config']
        self._model.validate()
        self._model.layoutChanged.emit()

    def save_settings(self):
        self.save_table_state()
        
        settings = {}
        tabs_config = self._table.model().tabs_config
        # remove error keys from event_members
        for tab_config in tabs_config:
            try:
                del tab_config['errors']
            except:
                pass
        settings['tabs_config'] = tabs_config
        return settings


class SwitchToVLView(ChainAction):

    name = 'Switch To VL View'      

    def run(self, gui, settings, chain):
        idx = gui.vl_tabs.currentIndex()
        vl = str(gui.vl_tabs.tabData(idx) or '').strip() or ALL_BOOKS
        print('debug1: vl: {}'.format(vl))
        if vl:
            view = self.vl_view_lookup(vl, settings['tabs_config'])
            print('debug2: view: {}'.format(view))
            if view:
                if not view in view_manager_views(gui):
                    if DEBUG:
                        prints('Action Chains: Switch To VL View: view ({}) is not available'.format(view))
                    return                  
                self.switch_view(gui, view, vl)      
            else:
                if DEBUG:
                    prints('Action Chains: Switch To VL View: VL Tab ({}) has no configured view'.format(vl))

    def vl_view_lookup(self, vl, tabs_config):
        for tab_config in tabs_config:
            if vl == tab_config['tab_name']:
                return tab_config['view_name']

    def switch_view(self, gui, view, vl):
        view_manager = gui.iactions.get('View Manager')
        if view_manager:
            import calibre_plugins.view_manager.config as vm_cfg
            library_config = vm_cfg.get_library_config(gui.current_db)
            view_info = copy.deepcopy(library_config[vm_cfg.KEY_VIEWS][view])
            ####
            view_info[vm_cfg.KEY_APPLY_VIRTLIB] = False
            ####
            selected_ids = gui.library_view.get_selected_ids()
            # Persist this as the last selected view
            if library_config.get(vm_cfg.KEY_LAST_VIEW, None) != view:
                library_config[vm_cfg.KEY_LAST_VIEW] = view
                vm_cfg.set_library_config(gui.current_db, library_config)

#            if view_info.get(vm_cfg.KEY_APPLY_VIRTLIB,False):
#                view_manager.apply_virtlib(view_info[vm_cfg.KEY_VIRTLIB])
            if view_info[vm_cfg.KEY_APPLY_RESTRICTION]:
                view_manager.apply_restriction(view_info[vm_cfg.KEY_RESTRICTION])
            if view_info[vm_cfg.KEY_APPLY_SEARCH]:
                view_manager.apply_search(view_info[vm_cfg.KEY_SEARCH])
            view_manager.apply_column_and_sort(view_info)

            gui.library_view.select_rows(selected_ids)
            view_manager.current_view = view
            view_manager.rebuild_menus()
        else:
            if DEBUG:
                prints('Action Chains: Switch To VL View: View Manager Plugin not available')

    def validate(self, settings):
        if not settings:
            return (_('Settings Error'), _('You must configure this action before running it'))
        return True

    def config_widget(self):
        return ConfigWidget 


Originally Posted by JimmXinu

I'm not usually involved in debugging whole libraries. Is it appropriate to ask for a zipped copy of the library to look at? If it's not too private or too copyrighted, that is.

That's really nice of you to offer but I don't think I'm comfortable sharing my whole library... Would it work to share a ZIP of a library with a smaller selection, with all eBook formats removed? "Remove all formats from selected books"? I made a dupe library that way (copied the structure from my main library), and it's giving me the same lags as my full library.

[ownedbycats]

Just the duplicated test library should work for these purposes; I've done it before. You don't need to send any of the files either, just the metadata.db.

[BetterRed]

Originally Posted by jnikkir

. . .

That's really nice of you to offer but I don't think I'm comfortable sharing my whole library... Would it work to share a ZIP of a library with a smaller selection, with all eBook formats removed? "Remove all formats from selected books"? I made a dupe library that way (copied the structure from my main library), and it's giving me the same lags as my full library.

Ah-ha - If you still have that laggy library with no format files, we can use it to 'prove' the lag is not due to file system access.

Secure a copy of it with your file manager (with calibre closed), then delete all the Author and Book folders so that it looks something this:

**

Then start calibre and access that library (you won't see any covers) and do whatever to see if it's still laggy - assuming it is you could zip that library, upload it somewhere, and PM the link to Jim.

BR

** you might not have the …backup.json and …restore.db files, that's OK.

[jnikkir]

Originally Posted by BetterRed

Ah-ha - If you still have that laggy library with no format files, we can use it to 'prove' the lag is not due to file system access.

Secure a copy of it with your file manager (with calibre closed), then delete all the Author and Book folders so that it looks something this:

Attachment 192598 **

Then start calibre and access that library (you won't see any covers) and do whatever to see if it's still laggy - assuming it is you could zip that library, upload it somewhere, and PM the link to Jim.

BR

** you might not have the …backup.json and …restore.db files, that's OK.

Yep, it's still laggy with no book/author folders in the library folder. =? Just PM'd the .zip to Jim.

Thank you all again for the help. I really appreciate it.