Send to Device with series order

[marie44]

I read a lot of romance, sci-fi, fantasy - which are often parts of some series or another and so are all identified as such in calibre. So yeah - LOTS of collections if I let Sony create collections from series, since I often only have one or two in a series.

I effectively have 2 tag columns - the default one that I use more loosely for organization and themes w/in calibre, and then I created a "Sony" tags column that I use much more strictly - genres & subgenres only. ("fantasy" "fantasy:urban" "romance" "romance:historical" "romance:contemporary", etc). The "Sony" tags column is the only that creates collections on my device; it's much easier to browse that way.

[jackie_w]

Originally Posted by marie44

The plugboard that jackie_w described, as I understand it, was supposed to change it so that when I sort by author the titles in a series will display by series order rather than alphabetical by title. And it works, but only some of the time. (I just double checked, and if I sort by "title" rather than "author" I have the same issues - some series sort by series #, and some sort alphabetically). I didn't know if perhaps there was something in the plugboard that needs to be different for a 650, since jackie_w has a 505. Or does the 505 workaround just not work for the 650?

Hi Marie,

I'm glad that my info got you part way to your goal.

Just to clarify... All the info I posted above was nothing to do with Collections at all. To get all books sorted by Author/Series/Series_index/Title I was using an item from the main menu of the PRS505 called "Books by Author" rather than the main menu item labelled "Collections". I suppose it's possible that the PRS650 doesn't have these menu items.

However, since my previous post I HAVE been tinkering with Collections and some more of chaley's fantastic new features.

Like you I have edited my Sony Device Interface plugin so that Collections are only created from Tags (genre in my case). The books were sorted by Title by default. Title, here seems to mean the Title in the Calibre library. Confusing, or what.

Nevertheless I got my Collections correctly sorted like this:

• Created a new column in my library using Preferences - Add your own columns
  

  Lookup name: coll505sort
  Col hdg: Coll_505_sort
  Col type: Col built from other cols
  Template: {author_sort}-{series}{series_index:0>2s||-}{title}

  You'll never see this field on your reader, so you don't need to get elaborate. It's only used for sorting.
  
• Close Calibre and restart so the new column is added.
  
• Then back into Preferences - Tweaks and set
  

  sony_collection_sorting_rules = [(['*'], '#coll505sort')]

• Close Calibre and restart so the new tweak is added.
  
• Connect PRS505 to Calibre, let it do its stuff and disconnect.
  
• Then when I went into Collections and opened one of them all the books were in the order defined by my new custom column
  
• NB. All the work done in the Plugboard is still as I described earlier in the thread, i.e the Collection work is in addition

Perhaps you could see if this works on your PRS650.

[chaley]

Originally Posted by jackie_w

Then back into Preferences - Tweaks and set

sony_collection_sorting_rules = [(['*'], '#coll505sort')]

It is great to see confirmation that the tweak works for someone other than me.

[marie44]

@jackie_w: I assumed, erroneously, obviously, that whatever sorting rules applied to the "books" option (the option on the 650's homepage that lists all books on the device) would also apply to collections. I never use the "books" option, so I had to go check it out - sure enough, the plugboard does work there! With one minor glitch - the titles are listed in reverse numerical/alphabetical order (Author Archer, Book 4, Book 3, Book 2, etc). (Authors are however listed in correct order). Not a huge deal since I don't use that menu, but any guesses as to what I've got set wrong that makes it reverse like that?

As to chaley's tweak - how do I set this up in conjunction with what I already have set up to create collections from my "sony" tags:

sony_collection_renaming_rules={'#sony':''}

I tried to just copy the new tweak and put it directly below the one I already have, but that didn't seem to do anything as far as I could tell - not that I really expected it to. I'm assuming I need to combine the two, but I'm not sure how.

Thanks!

[chaley]

Originally Posted by marie44

As to chaley's tweak - how do I set this up in conjunction with what I already have set up to create collections from my "sony" tags:

sony_collection_renaming_rules={'#sony':''}

I tried to just copy the new tweak and put it directly below the one I already have, but that didn't seem to do anything as far as I could tell - not that I really expected it to. I'm assuming I need to combine the two, but I'm not sure how.

Thanks!

Using the same column to sort won't work for two reasons. 1) all the books would have the same value, so the sort will fall back to title, and 2) that field is multiple, so all will sort on the first one.

What order do you want the books to be in within each collection? If it is the same as in #2 in your original post, then you should:
1) create a new custom column of kind 'column built from other columns. Call it whatever you want, perhaps sony_sort.
2) Put

{series}{series_index:>2s|| - }{title}

into the template.
3) Restart calibre so the column comes into existence.

At this point you should see the column off to the right. Check that the contents are what you want. If not, you can double-click on the column to edit the template.

Now...
4) Tell calibre to sort collections using that column. To do so, change the sony_collection_sorting_rules tweak to be:

[(['*'], '#sony_sort')]

Next time metadata is refreshed (on next connect if metadata management is set to automatic), the collections will be rebuilt and resorted.

If you wish, you can hide the #sony_sort column. There is no need for it to take up screen space.

edit and side comment: I note that your cat looks a lot like my eldest, Lizzie. The 'M' on Lizzie's forehead seems to be a bit more pronounced than your cat's, but the coloring is very similar.

[jackie_w]

Originally Posted by chaley

It is great to see confirmation that the tweak works for someone other than me.

You're very welcome. You keep writing 'em, I'll keep using 'em.

[marie44]

Note to self: do not try messing with this stuff when tired, b/c I conflate completely separate tweaks (collection renaming and collection sort). Erg. Sorry about that.

@chaley: Well, I tried the new column and the tweak built off it, and I initially thought it worked, but it appears to only be affecting some series and not others. I can see at least three different issues: Series A is sorted numerically by series in one collection but not another (sorted alphabetically there). Series B sorts almost correctly, except book 15 which is listed first (Books 1-14 list correctly) - and it behaves the same in two different collections. Series C is sorting alphabetically across collections and not by series #. I did try removing some series and putting them back on, but that didn't seem to make a difference.

(and yes, metadata management is set to automatic).

[chaley]

Originally Posted by marie44

@chaley: Well, I tried the new column and the tweak built off it, and I initially thought it worked, but it appears to only be affecting some series and not others. I can see at least three different issues: Series A is sorted numerically by series in one collection but not another (sorted alphabetically there). Series B sorts almost correctly, except book 15 which is listed first (Books 1-14 list correctly) - and it behaves the same in two different collections. Series C is sorting alphabetically across collections and not by series #. I did try removing some series and putting them back on, but that didn't seem to make a difference.

(and yes, metadata management is set to automatic).

Do all of the books have a series and series index? Those that do not will sort by title.

Please look at a series that is sorted incorrectly and check that it appears in a different order than you get if you sort calibre's library by the new column. You can make this easier by searching for the series, then clicking on the sort column header to sort it. It the orders are the same but not what you want, then we need to play with the column. If the orders are different, then I have a problem somewhere.

If there is a difference, then the first place to start is to capture calibre's debug output when the collections are built. To do that, start a command shell/box using whatever is appropriate for your operating system (on windows it is start, run, type cmd).
1) Make note of what folder you are in (on windows it will be the prompt).
2) Ensure your device is not plugged in.
3) Type

calibre-debug -g > caldebug.txt 2>&1

calibre should open
4) plug in your device. Wait for calibre to recognize it and show you the device view.
5) quit calibre.
6) Using a file explorer, go to the folder you noted in step 1. There should be a file caldebug.txt in it.
7) Look at the file caldebug.txt. Remove any information you consider private. If you remove too much, the file will become useless to me.
8) Post that file here. Alternatively, you can send it to calbugs at charles.haleys.org (make the obvious substitution).

[Glubbish]

Hi Chaley.
I am jumping on this thread as I have much the same problem
ie:
on the sony:
- collections are sorted correctly, series in order (when sorted by title)
- books (list all) shows series out of order (when sorted by author)

Here is my debug output
Starting up...
Using proxies: {'ftp': '{removed}', 'http': '{removed}'}
Started up in 52.3129999638
Job: 1 Get device information finished
No details available.
DEBUG: 0.0 PRS505: starting fetching books for card None
DEBUG: 0.0 USBMS: Fetching list of books from device. oncard= None
DEBUG: 0.0 USBMS: dirs are: G:\ database/media/books
DEBUG: 15.3 USBMS: count found in cache: 1061, count of files in metadata: 1061, need_sync: False
DEBUG: 15.3 USBMS: Finished fetching list of books from device. oncard= None
DEBUG: 15.4 Building XMLCache... {0: 'G:\\database\\cache\\media.xml'}
DEBUG: 16.1 Done building XMLCache...
DEBUG: 16.1 Updating JSON cache: 0
DEBUG: 16.1 Start build_id_playlist_map
DEBUG: 16.2 Removing playlist id: 2236 Purchased
DEBUG: 16.2 after cleaning playlists
DEBUG: 16.3 Finish build_id_playlist_map. Found 338
DEBUG: 16.5 Finished updating JSON cache: 0
DEBUG: 16.5 PRS505: finished fetching books for card None
DEBUG: 16.5 PRS505: starting fetching books for card carda
DEBUG: 16.5 USBMS: Fetching list of books from device. oncard= carda
DEBUG: 16.6 Building XMLCache... {0: 'G:\\database\\cache\\media.xml'}
DEBUG: 17.0 Done building XMLCache...
DEBUG: 17.0 PRS505: finished fetching books for card carda
DEBUG: 17.1 PRS505: starting fetching books for card cardb
DEBUG: 17.1 USBMS: Fetching list of books from device. oncard= cardb
DEBUG: 17.1 Building XMLCache... {0: 'G:\\database\\cache\\media.xml'}
DEBUG: 17.5 Done building XMLCache...
DEBUG: 17.5 PRS505: finished fetching books for card cardb
Job: 2 Get list of books on device finished
No details available.
DEBUG: 19.6 PRS505: started sync_booklists
DEBUG: 19.7 Building XMLCache... {0: 'G:\\database\\cache\\media.xml'}
DEBUG: 20.2 Done building XMLCache...
DEBUG: 20.3 PRS505: collection fields: [u'series', u'tags']
Device using plugboard device_db PRS505 [[u'{series}{series_index:0>2s| - | - }{title}', u'title']]
DEBUG: 20.3 PRS505: use plugboards [[u'{series}{series_index:0>2s| - | - }{title}', u'title']]
DEBUG: 20.3 Starting update [u'series', u'tags']
DEBUG: 20.3 Start build_id_playlist_map
DEBUG: 20.6 Removing playlist id: 2236 Purchased
DEBUG: 20.6 after cleaning playlists
DEBUG: 21.0 Finish build_id_playlist_map. Found 338
DEBUG: 21.0 Updating XML Cache: 0
DEBUG: 29.4 Timezone votes: 0 GMT, 110 LTZ, use_tz_var=True
DEBUG: 29.4 Starting update_playlists [u'series', u'tags'] 0
DEBUG: 29.4 Starting get_collections: on_connect
DEBUG: 29.4 Renaming rules: {}
DEBUG: 29.4 Sorting rules: [(['*'], '#sony_sort')]
DEBUG: 29.7 update_playlists: finished building maps
DEBUG: 29.8 Finishing update_playlists
DEBUG: 29.8 In update/ Starting refresh of device_collections
DEBUG: 29.8 Start build_id_playlist_map
DEBUG: 29.9 after cleaning playlists
DEBUG: 30.0 Finish build_id_playlist_map. Found 338
DEBUG: 30.0 Running fix_ids()
DEBUG: 31.2 Finished running fix_ids()
DEBUG: 31.2 Finished update
DEBUG: 31.9 USBMS: starting sync_booklists
DEBUG: 40.8 USBMS: finished sync_booklists
DEBUG: 40.8 PRS505: finished sync_booklists
Job: 3 Send metadata to device finished
No details available.
get_metadata cache: 6370 calls, 83.28% misses

Thanks Glubbish

[chaley]

Originally Posted by Glubbish

Hi Chaley.
I am jumping on this thread as I have much the same problem
ie:
on the sony:
- collections are sorted correctly, series in order (when sorted by title)
- books (list all) shows series out of order (when sorted by author)

Thanks for posting the debug log. What you describe and what the debug output shows makes sense, mostly. Unfortunately I don't have a 650 (or an anything 50), so I can't be sure what is happening.

First note: calibre controls directly only the order of collections, and it seems that setting the tweak had the intended effect. The order of the other lists is controlled by the Sony, and the collections order tweak won't have any effect on them.

As regards the non-collections lists, I imagine that the *50 devices use the title_sort attribute in lieu of title when sorting. I note that your plugboard sets title, but it does not set title_sort. Are the books lists (not collection) sorted by the original title, perhaps with the leading articles removed? If so, that is evidence that my impression is correct.

If I am right -- the 650 is sorting by title_sort instead of title -- then you should be able to fix it by try adding a plugboard line that sets title_sort to the same value as title. My 300 throws title_sort away so I can't test this, but it is very easy to imagine that this will fix your problem. If #sony_sort is set to [u'{series}{series_index:0>2s| - | - }{title}', u'title'], then the plugboard could be [{#sony_sort}] -> title_sort instead of using the longer sequence.

Let me know what happens.

[Glubbish]

Hi Chaley,

Sorry but I am a complete neophyte as far as this goes.
Setting the column to
[u'{series}{series_index:0>2s| - | - }{title}', u'title']
ended up with the literals, I am not sure what the u is supposed to do.
I tried also without the [] but it looks like
u'<series> - <index> - <title>',u'title'
where the names in <> are the actual book values, the rest are literal.

Thanks Derek

[marie44]

Originally Posted by chaley

Do all of the books have a series and series index? Those that do not will sort by title.

Please look at a series that is sorted incorrectly and check that it appears in a different order than you get if you sort calibre's library by the new column. You can make this easier by searching for the series, then clicking on the sort column header to sort it. It the orders are the same but not what you want, then we need to play with the column. If the orders are different, then I have a problem somewhere.

Series & index information are all correct - that was the first thing I checked back when my first adjustments didn't work.

When I sort by the new column, #sony_sort, everything appears how it should appear - i.e. how I want it to appear on the 650. (With one exception of the prequel marked 0.50 - it shows up last rather than first, and yes, it does sort correctly when I sort by Calibre's default series column - but that's not directly related to the problem at hand). Stated another way - the #sony_sort order and how series display on the 650 do not match for the series I'm having problems with.

Debug log, as requested:

Starting up...
Started up in 1.75
Job:
1 Get device information finished
No details available.
DEBUG: 0.0 PRS505: starting fetching books for card None
DEBUG: 0.0 USBMS: Fetching list of books from device. oncard= None
DEBUG: 0.0 USBMS: dirs are: E:\ database/media/books
DEBUG: 1.1 USBMS: count found in cache: 137, count of files in metadata: 137, need_sync: False
DEBUG: 1.1 USBMS: Finished fetching list of books from device. oncard= None
DEBUG: 1.2 Building XMLCache... {0: 'E:\\database\\cache\\media.xml'}
DEBUG: 1.5 Done building XMLCache...
DEBUG: 1.5 Updating JSON cache: 0
DEBUG: 1.5 Start build_id_playlist_map
DEBUG: 1.5 Removing playlist id: 321 Purchased
DEBUG: 1.5 after cleaning playlists
DEBUG: 1.5 Finish build_id_playlist_map. Found 133
DEBUG: 1.6 Finished updating JSON cache: 0
DEBUG: 1.6 PRS505: finished fetching books for card None
DEBUG: 1.6 PRS505: starting fetching books for card carda
DEBUG: 1.6 USBMS: Fetching list of books from device. oncard= carda
DEBUG: 1.7 Building XMLCache... {0: 'E:\\database\\cache\\media.xml'}
DEBUG: 1.8 Done building XMLCache...
DEBUG: 1.8 PRS505: finished fetching books for card carda
DEBUG: 1.8 PRS505: starting fetching books for card cardb
DEBUG: 1.8 USBMS: Fetching list of books from device. oncard= cardb
DEBUG: 1.8 Building XMLCache... {0: 'E:\\database\\cache\\media.xml'}
DEBUG: 1.9 Done building XMLCache...
DEBUG: 1.9 PRS505: finished fetching books for card cardb
Job: 2 Get list of books on device finished
No details available.
DEBUG: 4.0 PRS505: started sync_booklists
DEBUG: 4.1 Building XMLCache... {0: 'E:\\database\\cache\\media.xml'}
DEBUG: 4.2 Done building XMLCache...
DEBUG: 4.3 PRS505: collection fields: [u'#sony']
Device using plugboard device_db PRS505 [[u'{title}{series:re(([^\\s])[^\\s]+(\\s|$),\\1)| [|}{series_index:| |]}', u'title']]
DEBUG: 4.3 PRS505: use plugboards [[u'{title}{series:re(([^\\s])[^\\s]+(\\s|$),\\1)| [|}{series_index:| |]}', u'title']]
DEBUG: 4.3 Starting update [u'#sony']
DEBUG: 4.3 Start build_id_playlist_map
DEBUG: 4.4 Removing playlist id: 321 Purchased
DEBUG: 4.4 after cleaning playlists
DEBUG: 4.4 Finish build_id_playlist_map. Found 133
DEBUG: 4.4 Updating XML Cache: 0
DEBUG: 4.8 Timezone votes: 0 GMT, 43 LTZ, use_tz_var=True
DEBUG: 4.8 Starting update_playlists [u'#sony'] 0
DEBUG: 4.8 Starting get_collections: on_connect
DEBUG: 4.8 Renaming rules: {'#sony': ''}
DEBUG: 4.8 Sorting rules: [(['*'], '#sony_sort')]
DEBUG: 4.8 update_playlists: finished building maps
DEBUG: 4.8 Finishing update_playlists
DEBUG: 4.8 In update/ Starting refresh of device_collections
DEBUG: 4.8 Start build_id_playlist_map
DEBUG: 4.8 after cleaning playlists
DEBUG: 4.8 Finish build_id_playlist_map. Found 133
DEBUG: 4.8 Running fix_ids()
DEBUG: 4.9 Finished running fix_ids()
DEBUG: 4.9 Finished update
DEBUG: 5.0 USBMS: starting sync_booklists
DEBUG: 5.7 USBMS: finished sync_booklists
DEBUG: 5.7 PRS505: finished sync_booklists
Job: 3 Send metadata to device finished
No details available.
get_metadata cache: 875 calls, 74.74% misses

And I hugely appreciate the amount of time you're putting in to try and determine the problem, btw. Thank you!

[chaley]

Originally Posted by Glubbish

Hi Chaley,

Sorry but I am a complete neophyte as far as this goes.
Setting the column to
[u'{series}{series_index:0>2s| - | - }{title}', u'title']
ended up with the literals, I am not sure what the u is supposed to do.
I tried also without the [] but it looks like
u'<series> - <index> - <title>',u'title'
where the names in <> are the actual book values, the rest are literal.

Thanks Derek

My apologies for not being clear. What I was trying to say was to create another line in the device_db/PRS505 plugboard. The new line's template should be identical to the existing one for title. The destination of the new line should be set to title_sort.

[chaley]

Originally Posted by marie44

Series & index information are all correct - that was the first thing I checked back when my first adjustments didn't work.

When I sort by the new column, #sony_sort, everything appears how it should appear - i.e. how I want it to appear on the 650. (With one exception of the prequel marked 0.50 - it shows up last rather than first, and yes, it does sort correctly when I sort by Calibre's default series column - but that's not directly related to the problem at hand). Stated another way - the #sony_sort order and how series display on the 650 do not match for the series I'm having problems with.

The first (unrelated) problem is my fault. I gave you the wrong template string for #sony_sort. It should be

{series}{series_index:0>2s| | - }{title}

Note the new zero in front of the > character and the space between the two vertical bars. As it was, series with indices more than 10 would not sort correctly. This change will fix some of the problems, but ...

Your unrelated example points out a much more serious issue, one that I am surprised hasn't come up before as the problem has been there for months (at least). The problem: series_index formatting does not work correctly for series numbers that contain decimal points. For example, using {series_index:0>2s}, we get values:

Series Index     Formatted value
  1                   01
  1.1                 1.10
  2                   02
  10                  10

With these formatted values, 1.1 will sort after 2 but before 10. The results get worse in some cases. There is no fix for this problem in the current release of calibre other than possibly some very complicated regexp that would reformat the answer.

What we need to do is to produce a formatted value for a series index that takes into account that it is really a number, so that 1 < 1.1 < 2. I am changing the template processing to accomplish this.

Once the changes are available (should be next release), then the template would use {series_index:0>5.2f} (note the 'f' instead of 's' at the end). With this template we will get the values:

Series Index     Formatted value
  1                   01.00
  1.1                 01.10
  2                   02.00
  10                  10.00

These will sort correctly because the decimal point is aligned at the third character.

For those of you who are technically minded, the template processor now accepts any formatting string that python's format function accepts. If the type specifier wants an integer or floating point number, calibre attempts to convert the input string (from any field) to the correct type, tossing an exception if the conversion fails. If conversion succeeds, then calibre applies the format and returns the string result.

[haiyeekayakee]

I hope this isn't cluttering up this thread, but I just want to express my gratitude for the info that has been provided here, in fact I can't thank you enough

I have loads of tags on my books in Calibre, but these were getting made into collections on my Sony which were an awful sprawling mess as a result, plus I couldn't easily see what books were in what order in a series.

But thanks to you guys and gals I finally have it set up just the way I like it, woohoo