[GUI Plugin] Quality Check

[kiwidude]

I will definitely add a check for series duplicates, that is an excellent idea. I think that will pickup one of the main frustrations people have where two series with the same number are actually completely different (and you get into numbering issues etc). Ideally you want your series names to be unique.

Missing books in a series? I'm not convinced. It isn't going to be able to tell you that you are missing #3 in a trilogy that you have #1 and #2 for. It will also return false positives in the scenarios where say you have an omnibus edition that covers books 1 & 2, so a supposed "gap" isn't really a gap. Then there is the whole issue of numbering of series indexes for magazines, people who always use partials etc. So I'm not going to include it at this point.

Having a mixture of full and partial indexes is not an "error" for many users. I like a lot of people intermix related short stories for a series by putting them in with a "x.1" index, with the main full books having the normal 1,2,3 index values. So I'm also not convinced on that one.

As for magazine related ones, they fall into the same category as "Missing parts" to me. These are not issues with the quality of the metadata (or conversions) of books in your library, which is what this plugin is aimed at. A lot of these are fairly bespoke to how you are storing and using the series index field. Maybe I can be convinced further down the track to add such checks but for now it is out of scope.

As for title case information, the algorithm is aimed at English names. No there is no customisation capability. It is all based on some open source code that is part of the Calibre library. So I would suggest not using the check (or the Calibre title case function) if it does not give you the results you would like. If you want it enhanced, as it is a Calibre function you should go through the normal feature request for Calibre route.

[drMerry]

Thank for telling reasons to in- or exclude functions.

I can agree with most of them.
But on the missing books I think there is a point that you did not mention.
Of-course there is the false positive problem. But the Missing-part-of-series is not a function you would use daily. But when it IS available, it would be a good tool to find books you still would like to have. (for example if you collect a specific series).

You will not see books that are missing with a higher index of the last book you have, but this is the way it has to be. (You also do not fetch isbn-numbers on picture-only pdf files although they are inside the book).
A work around for collectors knowing there are more books would be to add an empty book with the last series-index.

Another option would be: Series with 1 book. Some Ebooks set the series option (often with name of author). Than you have a series with 1 book (#1) that is not a real series. Would be nice to filter this books also (this kind of series do not have set the series-index so the index is set to the default calibre option)

[kiwidude]

Of your original suggestions other than duplicate series, the missing books in a sequence is one that I might eventually consider, though it has two issues I can see. The first is identifying the series increment to know a book is missing. For some people/series they use 0.01, 0.02, 0.03 etc. For "normal" series you have 1,2,3. For magazines you have said you use 2011.01, 2011.02 etc. So how can you differentiate those from someone who has a series of 1, 1.1, 1.2, 2, 3, 5? The second issue is to do with visual presentation of the results but you could kludge something together for that if you can solve the first issue.

As for books with one series, yes that could be useful for people with badly messed metadata I guess. Of course there are genuine reasons for having a book with only one in the series, such as when an author starts a new trilogy or you only own one book in the series and haven't created wishlist items for the rest. However like the "Check title with series" option it is something that just brings the results back and it is up to you to look through and identify genuine series issues versus false positives.

Speaking of which, I suggested on the find duplicates thread an idea I had for the "Check title with series" function which would apply to a Series with 1 book check and a number of other quality check functions. That is the ability to specify an exclusion for a book from a particular check.

For instance I modify author sort on some books to remove co-authors so that both sorting and exporting to my Kindle works "properly" to keep my series together. Those books are repeatedly flagged as having author sort problems, even though I know I am happy with them being so. If I could set an exclusion on those books for author sort then the next time I run that check they will not appear. Likewise for other checks as well. I think this could be fairly useful but is a fair amount of work and would need some menu restructuring, extra dialogs etc. You need to be able to review your exclusions, remove them and so on. It's something i will keep in mind for the future.

[theducks]

Originally Posted by kiwidude

Of your original suggestions other than duplicate series, the missing books in a sequence is one that I might eventually consider, though it has two issues I can see. The first is identifying the series increment to know a book is missing. For some people/series they use 0.01, 0.02, 0.03 etc. For "normal" series you have 1,2,3. For magazines you have said you use 2011.01, 2011.02 etc. So how can you differentiate those from someone who has a series of 1, 1.1, 1.2, 2, 3, 5? The second issue is to do with visual presentation of the results but you could kludge something together for that if you can solve the first issue.

You might have a Guess function (if ticked)
Look at all the indexes for a given series

All integers = basic numbering (default)
4 digit integers with decimals (cant be 2 digit only, 2010.1 is October) is issue date
What do you do with big gaps?


The scheme I use makes it difficult

123 is an omnibus of vols 1, 2, 3
4 is a normal index
4.n is something(s) that slots in between the volume and next. Shorts, fanfic
0.n has no exact placement/unknown (n for ordering of the un-ordered )
yyyy.ii year.issue (Analog Double (2 month) Issues ruin this)

[kiwidude]

@theducks - yes, exactly that sort of wide ranging series index usage is why I've got a big question mark over the practicality of implementation. Particularly where you have fractionals, there is just so much variability in how people have chosen to use the index to "bring together" stuff. If you limited it to just (for instance) looking at the integer portion of the number then that might be workable, but of questionable value given all the cases it "wouldn't" match.

[theducks]

Originally Posted by kiwidude

@theducks - yes, exactly that sort of wide ranging series index usage is why I've got a big question mark over the practicality of implementation. Particularly where you have fractionals, there is just so much variability in how people have chosen to use the index to "bring together" stuff. If you limited it to just (for instance) looking at the integer portion of the number then that might be workable, but of questionable value given all the cases it "wouldn't" match.

Can a PI be made to only function with/when a specific filter is in place? Series:series name(s)

then
radio buttons for the test pattern (avoids pattern errors by users).
spin-dial 'Ignore gaps bigger than"
tick: 'Ignore fractional portion'

[kiwidude]

The plugin already respects search restrictions. But it just sounds horrifically complicated, requiring intimate knowledge of the particular series you are examinging and how they are numbered. It would be orders of magnitude simpler just to visually scan down your books sorted by the series column

[theducks]

Originally Posted by kiwidude

The plugin already respects search restrictions. But it just sounds horrifically complicated, requiring intimate knowledge of the particular series you are examinging and how they are numbered. It would be orders of magnitude simpler just to visually scan down your books sorted by the series column

Been my plan all along. It has been working for me.
It was the other users that felt the need.

Now if you really wanted a Project
Find the missing Titles and make an 'Empty Book' record to hole their place

[drMerry]

The others == drMerry?

Well, there are problems, I knew it when I started this question.
But main problem is the false-positive.
This is why the quality-check is a 'helping' function, not a 'dictator'.
Just like the title in series it will show some info.
I think it is a lot of programming, so only start if you're motivated

I think you have to have one thing clear: It is an option presented to the user. The user is not forced to use it to have the benefit of the other functions.
So I can imagine theducks would not use this function because he has to much 'errors' while he uses the index in an other way than the 'normal index numbering' (what is normal...)

But think of the duplicate finder. If I would ask you at this moment to give me a plugin that could do all functions duplicate finder 1.0.2 does, you would say it would be a hard thing to program.
The way you solved that was for a great part by adding separate functions and let the user combine them. (with or without pre filtering the books to test)
I think that is the way you could work at this moment to.

To make it 'simple' you could start with a one-book option. Just select one book (or in left hand one series) en process this.
Give the user an option-screen to select some details (trial and error, only main-indexing, 2digit sub-indexing, 1digit sub-indexing)
That would give a result for the selected series. You could also add an option like last index: #, equals last index

Just some suggestions, but like I said. It may be a long walk so only start when motivated

[DoctorOhh]

I have been editing my ePubs in Sigil and using the Tweak ePub feature in calibre to remove the margins. I would verify that the margins went away in Calibre's viewer. This worked fine until yesterday when I realized my reader uses ADE to render its ePubs and ADE recognizes and uses the page-template.xpgt file for margins. This means I have a bunch of books that looked margin free in Calibre's viewer but really have a margin in them.

As of Calibre 0.7.53 calibre removes the Adobe page template margins during conversion. My problem is identifying the ePubs that have a page-template.xpgt file.

Can you add identifying which books have a page-template.xpgt file to the QC plugin? Once I have these identified I can reconvert the books with the same settings and have Calibre remove the Adobe page template margins.

Thanks for all your work and considering this request.

[kiwidude]

Originally Posted by dwanthny

Can you add identifying which books have a page-template.xpgt file to the QC plugin? Once I have these identified I can reconvert the books with the same settings and have Calibre remove the Adobe page template margins.

Thanks for all your work and considering this request.

I can do that. I vaguely recall a thread referring to this issue as these .xpgt files being spawn of the devil from the margin issues they caused. And that recently a Calibre change was made to do that margin removal, am I correct? What scenario do they cause a problem on? I.e. Will a user need to do an ePub to ePub conversion or is it just for converting to other formats like mobi? Is there any reason not to offer an option to just remove these files from the ePub?

[DoctorOhh]

Originally Posted by kiwidude

I can do that. I vaguely recall a thread referring to this issue as these .xpgt files being spawn of the devil from the margin issues they caused. And that recently a Calibre change was made to do that margin removal, am I correct?

You are correct as of Calibre 0.7.53 calibre removes the Adobe page template margins during conversion.

Originally Posted by kiwidude

What scenario do they cause a problem on? I.e. Will a user need to do an ePub to ePub conversion or is it just for converting to other formats like mobi?

If the page-template controls the margins, and they often do, then a conversion epub - epub or epub - other will need to occur to remove the margins via calibre.

Originally Posted by kiwidude

Is there any reason not to offer an option to just remove these files from the ePub?

Quite a few folks in this thread seem to remove the file and all references to it and are happy with the results. If I had the option to just remove the file I would do it. I have already proofed the files in calibre's viewer which seems to ignore this file.

As of April 1 Calibre handles removing the margins from the page-template during conversion. So identifying the older books that have this page-template is my minimum need. Once identified I'll have no problem running them through a ePub to ePub conversion and let calibre deal with it. But I would not be against having the option to remove the file and any references to the file.

[drMerry]

Originally Posted by drMerry

...
2. Where do you get the 'Title Case Information'?
I have some problems with it, All books having the word macht have to be written as MacHt (capital H). I think I made a mistake somewhere but I do not know where.
I also would like change this behaviour so only the first word and names get capitals, but I think I have to change a regex than.

This problem solved in next release of Calibre...

[jackie_w]

Originally Posted by kiwidude

Is there any reason not to offer an option to just remove these files from the ePub?

None at all, but it would be a lot less coding effort to leave the .xpgt in place but "neutralise" its margins and I believe this is what the current calibre convert does.

If you want to surgically remove the .xpgt I think you would need to:

• Find the .xpgt and delete it.
• Remove the reference to it in the .opf manifest.
• Remove the <link> reference to it in every html file.

I guess the decision is yours

[kiwidude]

@jackie - thx for the detail. My only thought is that as mentioned previously on this thread there is a perception rightly or wrongly that it can be "scary" to convert a file. Which if I didn't offer a way to remove it you are still forced to do.

Not that I am committing to do it at this point of course, I'm just intelligence gathering...