[GUI Plugin] Quality Check

[capink]

One of the good predictors of epub quality is the size of the css file. epubs with css less than 1 Kb usually turn out to be of bad quality. I checked my library that way outside calibre by extracting the size of the css for each epub. My coding skills are not really up to the task of adding such functionality to this plugin. Is someone is still maintaining the plugin he might consider adding this.

[theducks]

Originally Posted by capink

One of the good predictors of epub quality is the size of the css file. epubs with css less than 1 Kb usually turn out to be of bad quality. I checked my library that way outside calibre by extracting the size of the css for each epub. My coding skills are not really up to the task of adding such functionality to this plugin. Is someone is still maintaining the plugin he might consider adding this.

Boy is that a wide shot.

IMHO that is a possible sign of bloat
Here is my basic stylesheet (modeled after Webcriptions of old), 609 bytes and it includes some stuff some would have kittens over.

body{
    display: block;
    font-size: 1.2em;
    margin-bottom: 0;
    margin-left: 2pt;
    margin-right: 2pt;
    margin-top: 0;
    padding-left: 0;
    padding-right: 0;
    text-align: justify
    }
	
.indented{
	display: block;
   margin: 0.5em 0 0 0;
   text-indent: 1.5em;
	}
.nonindented{
	display: block;
   margin: 0.5em 0 0 0;
   text-indent: 0;
	}
	
.chapno {
   display: block;
   font-size: 1.5em;
   margin: 1em 0;
   border: 0;
   padding: 0;
   text-indent: 0;
   text-align: center;
}
.scene {
	display: block;
	margin: 1em 0; 
	text-align: center;
 }

[DNSB]

Originally Posted by theducks

Boy is that a wide shot.

IMHO that is a possible sign of bloat
Here is my basic stylesheet (modeled after Webcriptions of old), 609 bytes and it includes some stuff some would have kittens over.

I just checked my collection and found ~120 book that had a CSS file under 1KB. Removing those that had multiple CSS files, of the ~50 remaining, several did look as if the CSS had been trimmed to remove unused styles but overall, none of them looked bad. Perhaps a bit simple layout but eminently readable.

[JSWolf]

One thing I really dislike is the margin shortcut. It doesn't need to exist. Just use the full margin commands as it's easier to read.

What I do is load the eBook into Calibre. I then delete every useless HTML file and if there are multiple CSS, I delete the ones I no longer need and merge the rest. I also have Calibre remove unused CSS in the CSS file and HTML. Then I add in my own body and p classes. Then I make any other changes needed.

The problem is that if I drop in my own full CSS, I end up having to figure out what from the publisher CSS goes with my CSS and what I cannot dump as I don't have a version. It's too much hassle. it's a lot easier to modify what's there.

[theducks]

Request: for Check (triggered by a user question in Library Management)

Check (structure?) for EPUB Type: 2 OR 3 <selection

[icallaci]

Is there any possibility of adding a check for invalid CSS properties to this wonderful plugin? It does everything else I need, so a check for invalid CSS properties would make my life complete. Thank you for this very useful tool.

[kboogie222]

Sadly I am not a engineer, but if anyone finds the challenge interesting I'm happy to help with testing, etc. Thanks everyone for making this such a great application and plugin.

USER STATEMENT
As a Calibre user, I would like to be able to detect books with "Bad Breaks" so that I can repair or replace them with more readable versions.

BACKGROUND
As a result of poor conversion there are often books that have "Bad Breaks", where a line break is inserted mid sentence. This results in a new line that typically begins with a lowercase letter. This is very common and likely one of the biggest quality and readability issues with many Calibre user libraries. Unfortunately, there is no easy way to search an entire library and identify books that have Bad Breaks.

ACCEPTANCE CRITERIA
* User is able to search an entire library for books that have bad breaks
* User can search mobi, epub and azw formats
* User is able to set a threshold for number of bad breaks identified
* Results are displayed in filtered view

EXTRA CREDIT
* User is able to sample book and set page size or word count for sample size
* User can search additional document formats

[snarkophilus]

Originally Posted by kboogie222

USER STATEMENT
As a Calibre user, I would like to be able to detect books with "Bad Breaks" so that I can repair or replace them with more readable versions.

Great idea, but I can see this getting hairy quickly! Some books use lower case chapter names, so an algorithm that was smart enough to pick lower case letters at the start of a paragraph style instead of a chapter name style would be nice. Maybe something that counts the number of time a style was used? I've also seen cases where a paragraph that finishes with a lower case letter or a comma and the next starts with an upperc case character are still "bad breaks".

I'm sure there's someone in the Sigil world who has built up a fancy regex to find many of these. (Quick search...) There are some examples here, here, here and here.

Definitely a handy one if it could be implemented.

[theducks]

Originally Posted by snarkophilus

Great idea, but I can see this getting hairy quickly! Some books use lower case chapter names, so an algorithm that was smart enough to pick lower case letters at the start of a paragraph style instead of a chapter name style would be nice. Maybe something that counts the number of time a style was used? I've also seen cases where a paragraph that finishes with a lower case letter or a comma and the next starts with an upperc case character are still "bad breaks".

I'm sure there's someone in the Sigil world who has built up a fancy regex to find many of these. (Quick search...) There are some examples here, here, here and here.

Definitely a handy one if it could be implemented.

I have about 5 or 6 that I use in Sigil. Only 2, do I ever run using the 'All" button. The rest I step thru (it is still a very fast operation) , some times I do skip the replace .
There are still EXCEPTIONS . (lots of Publishers Boilerplate should not be touched).
I am currently reading a book that has a acronym that starts with a lower case letter.
A.M. or P.M. will fail. (One of my searches dos fix Mr. Mrs. ... splits )
Still, Nothing beats the human eyeball for spotting errors

[kboogie222]

Originally Posted by snarkophilus

Great idea, but I can see this getting hairy quickly! Some books use lower case chapter names, so an algorithm that was smart enough to pick lower case letters at the start of a paragraph style instead of a chapter name style would be nice.

Thanks so much for the direction, the links are super helpful and contain some useful regex and thinking around identifying the problems and fixing the problems. It seems like this has been a big challenge in the community that dates back over decade or longer.

Judging from the conversation, fixing the problem would take some finesse, and likely some human judgement. I'm a little nervous to even take that on, hah.. But clearly you all have been thinking about some improved approaches over the "Line un-wrap factor" that exists in Calibre.

From an identification perspective it sounds like we have two challenges; 1) accurately identifying the Bad Breaks via regex, and 2) implementing a regex search across an entire library.

Strictly from an identification vantage, do you think the regex posted here would do a decent job of identifying the breaks for the purpose of a quality check? Would it ignore the title edge case? Are there other edge cases that you would consider for the purposes of quality check and finding books with this problem?

Find: </p>\s+<p class="calibre2">([a-z])
Replace: \1 (a space followed by \1)

I wish I knew more about the Quality Check plugin architecture. Once we had a tuned up regex fingerprint, is the Quality Check plugin capable of searching across a library? Would it be straight forward to implement the search with an adjustable threshold and sample size?

This is really interesting, thanks so much for the direction on this!

[snarkophilus]

Originally Posted by kboogie222

Strictly from an identification vantage, do you think the regex posted here would do a decent job of identifying the breaks for the purpose of a quality check? Would it ignore the title edge case? Are there other edge cases that you would consider for the purposes of quality check and finding books with this problem?

I don't think the regex needs to be especially complex. We might need to think about cases where there's a span at the end of the paragraph too, so looking for things like paragraphs that finish with [letter]</span></p> for example. As mentioned in a few of the threads I linked to, songs and verses often finish a line without any punctuation, so they are likely to throw out false positives. That said a regex like this

[\w",](</span>)?</p>

might be a good place to start?? Of course, one could always make the detection regex configurable in the plugin too!

I wish I knew more about the Quality Check plugin architecture. Once we had a tuned up regex fingerprint, is the Quality Check plugin capable of searching across a library? Would it be straight forward to implement the search with an adjustable threshold and sample size?

The "run across the library" thing looks quite easy at a glance. Picking a Quality Check check that looks in HTML files there's not that much code involved:

Spoiler:

    def check_epub_address(self):
        RE_ADDRESS = re.compile(r'</address>', re.UNICODE)

        def evaluate_book(book_id, db):
            path_to_book = db.format_abspath(book_id, 'EPUB', index_is_id=True)
            if not path_to_book:
                self.log.error('ERROR: EPUB format is missing: ', get_title_authors_text(db, book_id))
                return False
            try:
                with ZipFile(path_to_book, 'r') as zf:
                    for resource_name in self._manifest_worthy_names(zf):
                        extension = resource_name[resource_name.rfind('.'):].lower()
                        if extension in NON_HTML_FILES:
                            continue
                        else:
                            data = zf.read(resource_name).lower()
                            if RE_ADDRESS.search(data):
                                return True
                    return False

            except InvalidEpub as e:
                self.log.error('Invalid epub:', e)
                return False
            except:
                self.log.error('ERROR parsing book: ', path_to_book)
                self.log(traceback.format_exc())
                return False

        self.check_all_files(evaluate_book,
                             no_match_msg='No searched ePub books have \<address\> smart tags',
                             marked_text='epub_address_tags',
                             status_msg_type='ePub books for <address> smart tags')

Instead of just looking for at least one match for the regex, you could count the number of times the broken sentence regex appears and return "true" if more than certain (configurable?) threshold.

It seems like your original goal of detecting all epubs in a library that have possible broken sentences doesn't seem that hard (he says!). Fixing those automatically? No thanks

I'm still very new to Calibre plugins, so I may be leading you down the wrong path. So take all that I said about with a grain of salt, especially if someone more knowledgeable says something that contradicts me

[snarkophilus]

Originally Posted by snarkophilus

As mentioned in a few of the threads I linked to, songs and verses often finish a line without any punctuation, so they are likely to throw out false positives.

Indeed, Stephen King's Christine has 145 matches for just [a-z]</p> and 245 matches for [a-z,]</p>. Almost all of these were in song verses at the start of each chapter, but there were three missing periods at the end of sentences, one comma that should have been a period and one actual occurance of a break mid-sentence.

[JSWolf]

Is anyone working on making this plugin work with Python 3?

[davidfor]

Originally Posted by JSWolf

Is anyone working on making this plugin work with Python 3?

Can you report what errors you are seeing?

[JSWolf]

Originally Posted by davidfor

Can you report what errors you are seeing?

I'm not seeing any errors as I'm not using the 4.99+ beta. I'm just keeping track of what I use that's been updated.