[GUI Plugin] Quality Check

[dunhill]

I have friends who have asked me to translate it and when using it, maybe not 100%, but the use they give it works for them. I leave it up to you, I just put my grain of sand

[chaley]

Originally Posted by dunhill

I have friends who have asked me to translate it and when using it, maybe not 100%, but the use they give it works for them. I leave it up to you, I just put my grain of sand

I released it as version 1.11.0.

[dunhill]

thank you

[jindroush]

Hi guys,
can anyone responsible for the release do this change in the future releases?

file: check_fix.py
function: check_and_update_file_sizes
original line:
db_size = db.sizeof_format(book_id, fmt, index_is_id=True)
replace by:
db_size = next(db.backend.execute( "SELECT uncompressed_size FROM data WHERE book=? AND format=?", ( book_id, fmt ) ))[0]

Without this change Calibre never consults db_size actually, as sizeof_format is touching disk, not db.

How to test the functionality:
a) Add any ebook to calibre
b) Manually modify the filesize on the disk in calibre's storage folder
c) Check by Quality Check/Fix/Check and repair book sizes - the original code does not find any difference, the fixed code finds one and fixes it.

[chaley]

Originally Posted by jindroush

Hi guys,
can anyone responsible for the release do this change in the future releases?

file: check_fix.py
function: check_and_update_file_sizes
original line:
db_size = db.sizeof_format(book_id, fmt, index_is_id=True)
replace by:
db_size = next(db.backend.execute( "SELECT uncompressed_size FROM data WHERE book=? AND format=?", ( book_id, fmt ) ))[0]

I am not going to add direct sql access to calibre's tables.

What I will do is talk with Kovid about me adding API to base calibre to get that number. If he agrees then after the API change is released I will change the plugin appropriately.

Also note that your change isn't sufficient. The plugin calls

db.format_metadata(book_id, fmt, update_db=True, commit=True)

Note that it doesn't use the named parameter use_cache, which by default is true. Use_cache overrides update_db so the database is not updated. I will fix that as well if we get that far.

[chaley]

Kovid agreed with my changes.

Attached is beta version 1.11.1 that uses the changes discussed in my last post. Check and repair book sizes will work if you run from current calibre source or after calibre 5.9 is released, otherwise it shows an error.

[jindroush]

Thx. My fix was simple dirty patch which worked - funny thing about use_cache must have been introduced later.

[chaley]

Originally Posted by jindroush

funny thing about use_cache must have been introduced later.

The problem has been there since at least calibre 4.0. It would work sometimes. If use_cache is true and the book's format metadata is in the cache then update_db was ignored. I changed it to give priority to update_db, ignoring use_cache if update_db is true.

[Rob557]

When I click on "Quality Check / Search ePub", Calibre now freezes and I need to use Windows 10 Task Manager to close the program. I have used that very useful feature of Quality Check for years without trouble.

When using Search ePub, it normally displays something like the last 10 search criteria so that you can select one, or enter a new one. Recently I've been using 4 different sets of multiple search criteria to find ePubs that have various OCR problems that I decide whether to try to fix, or download a cleaner version of the ePub.

My problem started today when I decided to clear out the current list of search criteria and I did that by putting in some dummy criteria, running each one against a single ePub. The new criteria were something like $$A, $$B, ..., $$J. Once the old ones were cleared out, I then clicked on Search ePub so that I could put in a new criteria list, but instead of the normal listing of most-recently used criteria, Calibre simply froze as soon as I clicked on "Search ePub".

The following are different things I've tried to do (each time failing and having to force-terminate and restart Calibre ... I'm using version 5.12 for Calibre and version 1.11 for Quality Check):
a) change Calibre libraries,
b) change search scope (one book, several books, full library),
c) choose "repair" Calibre when re-install version 5.12,
d) uninstall Quality Check, reboot Calibre, reinstall Quality Check version 1.11, reboot Calibre.

I also tried other Calibre functionality and other features of Quality Check before trying Seach ePub, and nothing else seems to be a problem other than Calibre freezing as soon as I click on Search ePub.

My thoughts go in the direction that there is a problem with the new set of prior search criteria that has been stored, but I don't know if there is any way to reset that list if simply uninstalling the Quality Check plugin and re-installing it does not result in removing the prior criteria.

Apologies for the length of my comments. I'd welcome any suggestions. I'm assuming for now that the problem is with the operation of the plugin rather than a problem with Calibre itself ?

[davidfor]

@Rob557: It is almost definitely something in the search criteria breaking things. We'd need to see exactly what it was to have an idea. I don't know exactly how it does the search (regex, plain text or something else) but you might have hit a combination that needs some escaping to work properly.

You could try running calibre in debug mode to see if the log shows anything. Though you will probably need to start it from the command-line using "calibre-debug -g".

Uninstalling/repairing calibre does not touch the configuration. Neither does uninstalling a plugin touch its configuration. The search options are in the file "gui.json". This is in the calibre configuration directory. The simplest way to get to that is from the Miscellaneous page of the preferences. Edit this with calibre closed. Look for "quality check plugin:search epub dialog:search_opts". Following this is the epub search configuration. Mine looks like:

  "quality check plugin:search epub dialog:search_opts": {
    "ignore_case": true,
    "previous_finds": [
      "dresden"
    ],
    "scope_css": false,
    "scope_html": true,
    "scope_ncx": false,
    "scope_opf": false,
    "scope_plaintext": false,
    "scope_zip": false,
    "show_all_matches": false
  },

Deleting the whole block will remove the configuration. Deleting whatever is between the square brackets after "previous_finds" should be enough to reset the searches.

[ownedbycats]

Originally Posted by davidfor

Deleting the whole block will remove the configuration. Deleting whatever is between the square brackets after "previous_finds" should be enough to reset the searches.

Also copy the file to another folder before doing this. Calibre's config jsons like to reset everything if you accidentally delete the wrong character. I've done it more than once.

[Rob557]

Hi davidfor ... thank you for providing potential solutions for this.
With Calibre turned off, the following was the content of that section of the file "gui.json":

  "quality check plugin:search epub dialog:search_opts": {
    "ignore_case": false,
    "previous_finds": [
      "$$0",
      "$$9",
      "$$8",
      "$$3$$1$$2$$4$$5$$6$$7",
      "$$3$$1$$2$$4$$5$$6",
      "$$3$$1$$2$$4$$5",
      "$$3$$1$$2$$4",
      "$$3$$1$$2",
      "$$3$$1",
      "$$3"
    ],
    "scope_css": false,
    "scope_html": false,
    "scope_ncx": false,
    "scope_opf": false,
    "scope_plaintext": true,
    "scope_zip": false,
    "show_all_matches": true
  },

I replaced it with the following:

  "quality check plugin:search epub dialog:search_opts": {
    "ignore_case": false,
    "previous_finds": [
      "test"
    ],
    "scope_css": false,
    "scope_html": false,
    "scope_ncx": false,
    "scope_opf": false,
    "scope_plaintext": true,
    "scope_zip": false,
    "show_all_matches": true
  }

Unfortunately when I started up Calibre and tried Search ePub, the problem was not resolved.
Tomorrow I'll try the approach of running Calibre in debug mode. I've never done that before so not quite sure what to look for. Calibre will run okay up to the point of clicking on Search ePub. When it freezes, I'm not sure where to look for a 'crash' report.
Thanks again.

[Rob557]

I gave it a quick try and now see how the log file appears in the Command window and how to copy the result ... when Calibre freezes, nothing further is shown in the log window identifying that it froze, but there do seem to be other warning messages during the boot process that may be relevant:

Microsoft Windows [Version 10.0.18363.1440]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>calibre-debug -g
calibre 5.12 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.18362 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 3.8.5
Windows: ('10', '10.0.18362', '', 'Multiprocessor Free')
Interface language: None
Failed to initialize plugin: 'C:\\Users\\D\\AppData\\Roaming\\calibre\\plugins\\Barnes & Noble.zip'
Traceback (most recent call last):
  File "calibre\customize\ui.py", line 703, in initialize_plugins
  File "calibre\customize\ui.py", line 61, in load_plugin
  File "calibre\customize\zipplugin.py", line 293, in load
  File "importlib\__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "calibre\customize\zipplugin.py", line 192, in exec_module
  File "calibre_plugins.barnes_noble.__init__", line 11, in <module>
ImportError: cannot import name 'quote' from 'urllib' (C:\Program Files\Calibre2\app\bin\python-lib.bypy.frozen\urllib\__init__.pyc)
Failed to initialize plugin: 'C:\\Users\\D\\AppData\\Roaming\\calibre\\plugins\\DeDRM.zip'
Traceback (most recent call last):
  File "calibre\customize\ui.py", line 703, in initialize_plugins
  File "calibre\customize\ui.py", line 61, in load_plugin
  File "calibre\customize\zipplugin.py", line 293, in load
  File "importlib\__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "calibre\customize\zipplugin.py", line 185, in exec_module
  File "calibre\customize\zipplugin.py", line 181, in get_code
  File "calibre_plugins.dedrm.__init__", line 132
    except Exception, e:
                    ^
SyntaxError: invalid syntax
Failed to initialize plugin: 'C:\\Users\\D\\AppData\\Roaming\\calibre\\plugins\\FictionDB.zip'
Traceback (most recent call last):
  File "calibre\customize\ui.py", line 703, in initialize_plugins
  File "calibre\customize\ui.py", line 61, in load_plugin
  File "calibre\customize\zipplugin.py", line 293, in load
  File "importlib\__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "calibre\customize\zipplugin.py", line 192, in exec_module
  File "calibre_plugins.fictiondb.__init__", line 11, in <module>
ImportError: cannot import name 'quote' from 'urllib' (C:\Program Files\Calibre2\app\bin\python-lib.bypy.frozen\urllib\__init__.pyc)
Successfully initialized third party plugins: Count Pages (1, 11, 0) && EpubMerge (2, 11, 0) && Find Duplicates (1, 8, 4) && Generate Cover (2, 0, 1) && Goodreads (1, 5, 1) && Modify ePub (1, 6, 2) && Quality Check (1, 11, 0) && Reading List (1, 7, 1) && Resize Cover (1, 1, 0) && Search The Internet (1, 8, 1)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 101.6 x 101.6
Using calibre Qt style: True
[0.00] Starting up...
[0.01] Showing splash screen...
[0.39] splash screen shown
[0.39] Initializing db...
[3.50] db initialized
[3.51] Constructing main UI...
{'default': False, 'width': 400, 'height': 680}
Resize Covers::rebuild_menus - size_data={'default': False, 'width': 400, 'height': 680}
{'default': False, 'width': 420, 'height': 680}
Resize Covers::rebuild_menus - size_data={'default': False, 'width': 420, 'height': 680}
{'default': True, 'width': 435, 'height': 680}
Resize Covers::rebuild_menus - size_data={'default': True, 'width': 435, 'height': 680}
{'default': False, 'width': 450, 'height': 680}
Resize Covers::rebuild_menus - size_data={'default': False, 'width': 450, 'height': 680}
{'default': False, 'width': 460, 'height': 680}
Resize Covers::rebuild_menus - size_data={'default': False, 'width': 460, 'height': 680}
{'default': False, 'width': 480, 'height': 680}
Resize Covers::rebuild_menus - size_data={'default': False, 'width': 480, 'height': 680}
{'default': False, 'width': 520, 'height': 680}
Resize Covers::rebuild_menus - size_data={'default': False, 'width': 520, 'height': 680}
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
EpubMerge: DEBUG: 2021-03-10 20:19:11,970: calibre_plugins.epubmerge.epubmerge_plugin(158): macmenuhack file_path:C:\Users\D\AppData\Roaming\calibre\plugins\fanficfare_macmenuhack.txt
[13.44] main UI initialized...
[13.44] Hiding splash screen
Starting QuickView
[15.55] splash screen hidden
[15.56] Started up ...

I'll check tomorrow whether any other clues emerge. Thanks.

[ownedbycats]

FictionDB plugin is defunct now and not updated for 5.x; it's probably not doing any harm sitting in there but I'd delete it. Also update DeDRM and the Barnes & Noble plugin.

Not much help other than getting rid of the errors though.

[Rob557]

Hi ownedbycats,

Originally Posted by ownedbycats

Also copy the file to another folder before doing this. Calibre's config jsons like to reset everything if you accidentally delete the wrong character. I've done it more than once.

Good advice. I made a copy prior to editing, just in case.

Originally Posted by ownedbycats

FictionDB plugin is defunct now and not updated for 5.x; it's probably not doing any harm sitting in there but I'd delete it. Also update DeDRM and the Barnes & Noble plugin. Not much help other than getting rid of the errors though.

I looked again but don't seem to have any of those plugins listed in Preferences as user-installed plugins, and have no prompts for updating those particular ones. I couldn't tell which error messages those items would be producing ... maybe something else is causing the error messages and might help explain why Calibre is freezing?