Many Series are Out of Sequence in the Drawer

[DoctorOhh]

I don't know when this happened, could have been yesterday with the update or it could have been months ago, but I went to manually look up a series and it was not in the series list where I expected to find it. As it turns out many are out of order.

Here is the example, I was looking up Sisters of the Moon and it wasn't under S. Under the S I have the following that are out of place:

Asian Saga
Vorkosigan Saga
Tales of the Sazi
Island in the Sea of Time
Golden Age of the Solar Clipper
Black Company: Book of the South
Bio of a Space Tyrant
Emily the Strange
Anna Strong

Based on these out of place Series under S I guessed that Sisters of the Moon was under M and sure enough I found it there.

I searched and found others out of place:

Sabina Kane under K
Confessions of Georgia Nicolson under N
Aurora Teagarden under T
Star Wars: The Thrawn Trilogy under T
Once Upon a Time under T
Sisterhood of the Traveling Pants under T
Witch World: The Turning under T
Sembia: Gateway to the Realms under R
Quest of the Riddle Master under R
Riftwar: Legacy under R where it should be
Inspector Ian Rutledge Mysteries under R
Nathan "Hollywood Nate" Weiss under "
America 2040 under 2
Area 51 under 5
Brothers of the Absinthe Club under A
Ophelia and Abby under A
Kissed by an Angel under A
Shadows of the Apt under A

I could go on for pages. Is mine a unique corruption? Is there anything you might want me to try? I did run clean database after I discovered the problem. Now my database is clean and out of order.

Titles and Authors seem fine. It just seems to be Series.

[chaley]

Doesn't happen for me, but then I might not have the right data.

CC uses the series sort value that comes from calibre, controlled by the tweak save_template_title_series_sorting. If the tweak is not 'strictly_alphabetic' then calibre uses the title_sort rules for the series. My guess is that you have changed the title sort rules so that "the" and "and" are significant no matter where they appear in the series (or title). Could you check that?

[DoctorOhh]

Originally Posted by chaley

Doesn't happen for me, but then I might not have the right data.

Ok so most likely it is unique to me. All I usually use in CC is my To Be Read tag so this could have been this way for a long time.

Originally Posted by chaley

My guess is that you have changed the title sort rules so that "the" and "and" are significant no matter where they appear in the series (or title). Could you check that?

I checked it and it looked normal, however it was bold. I reset the tweak and now it is not bold. Still looks the same to me. When I get home I'll connect and see if the metadata changes. Thanks for the feedback.

[chaley]

Originally Posted by DoctorOhh

I checked it and it looked normal, however it was bold. I reset the tweak and now it is not bold. Still looks the same to me. When I get home I'll connect and see if the metadata changes. Thanks for the feedback.

Changing the tweak won't force a CC metadata sync, so even if it is fixed CC won't see the change. Best would be to pick one or two of the "bad" ones and open Edit Metadata (Single) on them. That will update the last_mod time so that CC will re-download the metadata.

Difficult-to-see single character changes in regexps can have a huge effect. I hope the reset fixes it.

[DoctorOhh]

Originally Posted by chaley

Changing the tweak won't force a CC metadata sync, so even if it is fixed CC won't see the change. Best would be to pick one or two of the "bad" ones and open Edit Metadata (Single) on them. That will update the last_mod time so that CC will re-download the metadata.

Difficult-to-see single character changes in regexps can have a huge effect. I hope the reset fixes it.

Thanks I'll take it from here.

If things were so askew shouldn't that have shown (which it didn't) in calibre's library?

[chaley]

Originally Posted by DoctorOhh

If things were so askew shouldn't that have shown (which it didn't) in calibre's library?

I would think so, which is one thing that is making me nervous. Assuming both tweaks are set to 'library_order' then the regexp should have applied when sorting by series on the spreadsheet or in the tag browser.

The fact that CC doesn't download metadata when one changes the tweak might be significant. It could be that the books in question were updated in CC while you were experimenting and weren't subsequently updated. Yea, I know I am reaching.

[jackie_w]

Some more info. Having seen Doc's post, I've just looked at my Series groupings.

Step1. Of 106 books, 2 appeared under the wrong Series first letter. Both had Series value Inspector Alan Grant, but appeared under 'G' instead of 'I'. This is very odd as I also have books with Series values Inspector Banks and Inspector Vaara which are both correctly grouped under 'I'. In the calibre library Tag Browser panel, all these books are correctly grouped under Series 'I'.

My Tweaks, from day 1 of them being available, have been set as:
title_series_sorting = 'strictly_alphabetic'
save_template_title_series_sorting = 'strictly_alphabetic'
and have never changed since.

Step 2. In calibre I opened Edit Metadata and saved (with no changes) so that the metadata.opf file has today's date. After reconnecting CC, both books are still incorrectly grouped under 'G'.

Step 3. In calibre I changed one of the two books to have Series value Inspector Grant. After reconnecting CC this changed book is now correctly grouped under 'I'. Then I changed the 2nd book in the same way. Both books now correctly grouped in CC.

Step 4. In calibre change both books back to their original Series value of Inspector Alan Grant. Reconnect CC, both books still correctly grouped under 'I' with their original value of Inspector Alan Grant.

Conclusion, I've no idea

Edit: I've now looked through all my other first letter groups (Title, Authors, Tags, pubdate, #awards) and have not found any similar problems.

[chaley]

Originally Posted by jackie_w

Some more info. Having seen Doc's post, I've just looked at my Series groupings.

Step1. Of 106 books, 2 appeared under the wrong Series first letter. Both had Series value Inspector Alan Grant, but appeared under 'G' instead of 'I'. This is very odd as I also have books with Series values Inspector Banks and Inspector Vaara which are both correctly grouped under 'I'. In the calibre library Tag Browser panel, all these books are correctly grouped under Series 'I'.

Ignoring the "Inspector Al" in front, we see that "an Grant" became "Grant" (under G). Before the "huh", read on.

My Tweaks, from day 1 of them being available, have been set as:
title_series_sorting = 'strictly_alphabetic'
save_template_title_series_sorting = 'strictly_alphabetic'
and have never changed since.

Step 2. In calibre I opened Edit Metadata and saved (with no changes) so that the metadata.opf file has today's date. After reconnecting CC, both books are still incorrectly grouped under 'G'.

Did CC actually transfer the metadata? It could be that Kovid has gotten quite clever and not updated last_modified if there weren't any changes.

Step 3. In calibre I changed one of the two books to have Series value Inspector Grant. After reconnecting CC this changed book is now correctly grouped under 'I'. Then I changed the 2nd book in the same way. Both books now correctly grouped in CC.

OK, this change would definitely have changed last_mod, so metadata would transfer.

Step 4. In calibre change both books back to their original Series value of Inspector Alan Grant. Reconnect CC, both books still correctly grouped under 'I' with their original value of Inspector Alan Grant.

Conclusion, I've no idea

goodness reigns.

I have a theory. Going through both calibre and CC, I find that if a book was added to CC (and thus to calibre) via CC's "scan for new books", CC used an incorrect regexp to guess at the series sort. It did not anchor the pattern, so the (A|An|The) could appear anywhere, as described above for "Inspector Alan Grant". This incorrect metadata would be stored in CC when it sent the book to calibre. If that metadata was never changed in calibre then the error would remain in CC. Is there any significant chance that this scenario fits the two books?

The only other thing I can think of is that there could have been a short time when calibre "incorrectly" computed the series sort. Any books sent to CC during that time would have the wrong value. When the problem was corrected (if ever there was a problem), calibre would immediately start to use the right values but CC would be stuck with the wrong ones. I know I am grasping at straws, but that is about all I have left. Other than the one case described above, CC gets that value from calibre, and I can't see any way that calibre would generate the wrong value.

[jackie_w]

At one time - up until sometime in Oct 2013 - I was using calibre Save-to-Disk to transfer books to Dropbox (I won't bore you with the whys and wherefores) then synching them to android. So CC would have been finding new books by scanning rather than by direct transfer. Edit: For clarification, I don't think there have ever been any books in CC or on Dropbox which didn't originate from the calibre library.

Looking at the 2 books in question before today's metadata changes:

Dropbox:
Tey, Josephine - Inspector Alan Grant 03 - The Franchise Affair.epub
created on 25/04/2013

Calibre:
metadata.opf 10/10/2013
The Franchise Affair - Josephine Tey.epub 26/02/2013

Dropbox:
Tey, Josephine - Inspector Alan Grant 05 - The Daughter of Time.epub
created on 30/10/2012

Calibre:
metadata.opf 10/10/2013
The Daughter of Time - Josephine Tey 26/02/2013

Does any of this help/hinder your theory?

[chaley]

Originally Posted by jackie_w

Does any of this help/hinder your theory?

It supports it. The two books in question *could* have been added to CC via "scan" and never had their metadata updated from calibre. In this case they would have the wrong series sort.

FWIW: I have fixed the regexp problem in CC.

[DoctorOhh]

Originally Posted by chaley

It could be that the books in question were updated in CC while you were experimenting and weren't subsequently updated.

My sort tweak was the standard:

Spoiler:

# Set the list of words considered to be "articles" for sort strings

per_language_title_sort_articles = {'ron': ('Un\\s+', 'O\\s+', 'Ni\xc5\x9fte\\s+'), 'tur': ('Bir\\s+',), 'por': ('A\\s+', 'O\\s+', 'Os\\s+', 'As\\s+', 'Um\\s+', 'Uns\\s+', 'Uma\\s+', 'Umas\\s+'), 'ita': ('Lo\\s+', 'Il\\s+', "L'", 'La\\s+', 'Gli\\s+', 'I\\s+', 'Le\\s+'), 'nld': ('De\\s+', 'Het\\s+', 'Een\\s+'), 'spa': ('El\\s+', 'La\\s+', 'Lo\\s+', 'Los\\s+', 'Las\\s+', 'Un\\s+', 'Una\\s+', 'Unos\\s+', 'Unas\\s+'), 'fra': ('Le\\s+', 'La\\s+', "L'", 'Les\\s+', 'Un\\s+', 'Une\\s+', 'Des\\s+'), 'ell': ('O\\s+', 'I\\s+', 'To\\s+', 'Ta\\s+', 'Tus\\s+', 'Tis\\s+', "'Enas\\s+", "'Mia\\s+", "'Ena\\s+", "'Enan\\s+"), 'eng': ('A\\s+', 'The\\s+', 'An\\s+'), 'swe': ('En\\s+', 'Ett\\s+', 'Det\\s+', 'Den\\s+', 'De\\s+'), 'deu': ('Der\\s+', 'Die\\s+', 'Das\\s+', 'Den\\s+', 'Ein\\s+', 'Eine\\s+', 'Einen\\s+'), 'afr': ("'n\\s+", 'Die\\s+')}

default_language_for_title_sort = None

title_sort_articles = '^(A|The|An)\\s+'

and I have never experimented with this tweak. Turns out the default tweak has had international language specific articles added, that is why it was slightly different, but my calibre library sorts perfectly.

Originally Posted by chaley

It supports it. The two books in question *could* have been added to CC via "scan" and never had their metadata updated from calibre. In this case they would have the wrong series sort.

My tweaks are library order. The last (many moons ago) time I loaded books I put them on the device using number only to identify the books. I do not believe I have had scan for books set since well before my last load. I won't actually have a chance to look at it closer until Wednesday night Thursday morning.

Originally Posted by chaley

FWIW: I have fixed the regexp problem in CC.

Would clean database have used this regex?

[DoctorOhh]

I reset the tweak, restarted calibre (v1.25) and sent a new book to CC in the Sisters of the Moon series and it was under M and not S. Fortunately my To Be Read tag works great. In case your curious I emailed you the log file after I sent the last book.

[chaley]

Originally Posted by DoctorOhh

I reset the tweak, restarted calibre (v1.25) and sent a new book to CC in the Sisters of the Moon series and it was under M and not S. Fortunately my To Be Read tag works great. In case your curious I emailed you the log file after I sent the last book.

I can't find any reason for this to happen. But clearly it does, so oh yes, I am curious.

The current logs won't help much because they don't display the series sort values. I have changed both CC and calibre to log the series sort values. My guess is that we will need to wait until you have both of these changes before we can make any progress on this.

[DoctorOhh]

Originally Posted by chaley

I can't find any reason for this to happen. But clearly it does, so oh yes, I am curious.

The current logs won't help much because they don't display the series sort values. I have changed both CC and calibre to log the series sort values. My guess is that we will need to wait until you have both of these changes before we can make any progress on this.

No problem, when I get both changes I'll check back in.

[chaley]

Originally Posted by DoctorOhh

I reset the tweak, restarted calibre (v1.25) and sent a new book to CC in the Sisters of the Moon series and it was under M and not S. Fortunately my To Be Read tag works great. In case your curious I emailed you the log file after I sent the last book.

I think I can explain this now. If the series is already in CC's database then when setting up grouping for a book, CC will use the series_sort that is already in the database instead of the one coming with the book. Because of this, the only way to correct the series misplacement in the grouping drawer is to delete all the books in that series then resend them. At least, I think that will do it.

Of course, if it works then we are left with the question about how the error got there in the first place. Still working on that.