Sony Collections: Manual Management

ilovejedd · 02-20-2012, 06:54 PM

I believe if you have Automatic Management enabled, it's possible to customize collection names to something like "Series: Harry Potter", "Genre: Fantasy" by using the following tweak:

Code:

sony_collection_renaming_rules={'series':'Series', 'tags':'Genre'}
sony_collection_name_template='{category:||: }{value}'

Is there a way to get similar behavior when using Manual Management? Might this be doable by customizing the Device Interface Plugin?

If that's not possible, would Automatic Management work with a multi-library, multiple calibre install scenario?

For example, what would happen if the same reader was connected/synced to the following computers/libraries.

Quote:

Computer A Library 1 {'series': 'Series', 'tags': 'Genre', '#type': ''}
(title, series [series_index], tags, #type)

"Harry Potter and the Philosopher's Stone", "Harry Potter [1.0]", "Fantasy", "Fiction"
"Twilight", "Twilight [1.0]", "Paranormal", "Fiction"

Computer A Library 2 {'series': 'Series', 'tags': 'Category', '#type': ''}
(title, series [series_index], tags, #type)

"Fanfic Title #1", "", "Harry Potter", "Fanfiction"
"Fanfic Title #2", "", "Harry Potter", "Fanfiction"

Computer B Library 3 {'series': 'Series', 'tags': 'Genre', '#type': ''}
(title, series [series_index], tags, #type)

"Harry Potter and the Philosopher's Stone", "Harry Potter [1.0]", "Children, Fantasy", "Fiction"

Computer B Library 4 {'series': 'Series', 'tags': 'Category', '#type': ''}
(title, series [series_index], tags, #type)

"Fanfic Title #1", "", "Harry Potter", "Fanfiction"
"Fanfic Title #2", "", "Harry Potter, Twilight", "Fanfiction"

Say I sync my reader with Library 1, then 2, then 3, then 4. Would it give me the following collections:

Quote:

Fiction
Harry Potter and the Philosopher's Stone
Twilight

Fanfiction
Fanfic Title #1
Fanfic Title #2

Category: Harry Potter
Fanfic Title #1
Fanfic Title #2

Category: Twilight
Fanfic Title #2

Genre: Children
Harry Potter and the Philosopher's Stone

Genre: Fantasy
Harry Potter and the Philosopher's Stone

Genre: Paranormal
Twilight

Series: Harry Potter
Harry Potter and the Philosopher's Stone

Series: Twilight
Twilight

Thanks!

chaley · 02-21-2012, 06:23 AM

Quote:

Originally Posted by ilovejedd

Is there a way to get similar behavior when using Manual Management? Might this be doable by customizing the Device Interface Plugin?

The answer is a definite maybe.

In manual mode, books are added to collections only when they are sent to the device. Metadata changes made after that point are ignored. In addition, books are only added to collections, they are never deleted. Finally, in manual mode no renaming rules or ordering rules are used, so the tweaks you mention will be ignored.

Last point: the interactions in manual mode between calibre, the device, and (if used) Sony Reader are complicated. There is a good chance that you will get some unexpected results.

Quote:

If that's not possible, would Automatic Management work with a multi-library, multiple calibre install scenario?

No. The sony database will always reflect the metadata for books matched (ondevice is True) in the last library connected.

There is another dragon here: ondevice identification. When you send a book to the sony or match a book to the library using metadata, the book's unique ID (UUID) is stored in the metadata.calibre file on the device. This UUID is used to map a book back to the source book in the calibre library. If a book appears in more than one library, it will have more than one UUID, and the UUID of the last matching library is what will appear in metadata.calibre. Because of this, there is a good chance that connections between a book and a library will be lost, especially when plugboards modify the metadata in the book.

It is worth noting that you can get an approximation of your desired collections when using manual mode only if you send the book from each library.

ilovejedd · 02-21-2012, 10:58 AM

Quote:

Originally Posted by chaley

It is worth noting that you can get an approximation of your desired collections when using manual mode only if you send the book from each library.

Hmm, so if doing manual management, what template should I use for collections? I'm fine with resending ebooks to the reader whenever I want to update metadata/collection information. In fact, that's already what I do. One issue, as can be seen from the above example, is it's a pain navigating a large number of unsorted collections some of which may have the same name but different uses altogether (e.g. Harry Potter as series and Harry Potter as tag/category).

I went into Preferences » Plugins » Customize SONY Device Interface and the default for collections is: series, tags

If I want series to show up as Series: Series Name and the tags to show up as Category: Tag Name or Genre: Tag Name (would be using different Calibre Portable installs so I don't ever forget to change settings when sending ebooks), what should I need to change the collections option to?

If that's not possible, do you know what I might be able to do to quickly creating custom columns using existing metadata? I figure if I can't customize the device interface plugin to automatically generate the custom collection names, I should be able to use custom columns instead and use those to generate collections.

e.g.

Code:

series                         #myseries (
Harry Potter             ->    Series: Harry Potter
Twilight                 ->    Series: Twilight

tags                           #category
Harry Potter, Twilight   ->    Category: Harry Potter, Category: Twilight

Also wondering, is possible to do automatically generate values for custom columns like this:

Code:

tags                               #type          #genre
Fiction, Children, Fantasy    ->   Fiction        Genre: Children, Genre: Fantasy
Fiction, Supernatural         ->   Fiction        Genre: Supernatural
FanFiction, Humor             ->   FanFiction     Genre: Humor
Magazine                      ->   Magazine

Basically, if the tag matches some predefined values (in this case: Fiction, FanFiction & Magazine), it goes to #type, otherwise, it goes to #genre as Genre: Tag Name.

Thanks so much!

Terisa de morgan · 02-21-2012, 11:14 AM

I have manual management and diferent collections according to tags and custom colums, and no problem.

ilovejedd · 02-21-2012, 11:57 AM

Quote:

Originally Posted by Terisa de morgan

I have manual management and diferent collections according to tags and custom colums, and no problem.

That's not exactly the issue. I already get that, too. For example, with the above computers/libraries, I would get the following collections on my reader:

Quote:

Children
Harry Potter and the Philosopher's Stone

Fanfiction
Fanfic Title #1
Fanfic Title #2

Fantasy
Harry Potter and the Philosopher's Stone

Fiction
Harry Potter and the Philosopher's Stone
Twilight

Harry Potter
Fanfic Title #1
Fanfic Title #2
Harry Potter and the Philosopher's Stone

Paranormal
Twilight

Twilight
Fanfic Title #2
Twilight

Granted, the above doesn't look too bad but I've already got 65 collections with only small subsections of my libraries (~15%). I've got over 3,000 titles worth of fanfics so you can see why I'd want to get organized when I transfer them to my reader. What I want to accomplish is add some extra sorting so instead of the above mish-mash, I would get something like this:

Quote:

Fiction
Harry Potter and the Philosopher's Stone
Twilight

Fanfiction
Fanfic Title #1
Fanfic Title #2

Category: Harry Potter
Fanfic Title #1
Fanfic Title #2

Category: Twilight
Fanfic Title #2

Genre: Children
Harry Potter and the Philosopher's Stone

Genre: Fantasy
Harry Potter and the Philosopher's Stone

Genre: Paranormal
Twilight

Series: Harry Potter
Harry Potter and the Philosopher's Stone

Series: Twilight
Twilight

chaley · 02-21-2012, 12:35 PM

@ilovejedd: If you use manual management then sony_collection_name_template is ignored. If you want to have the category name (as in "Genre") in front of the category value(s) you must put it there yourself.

You can indeed do what you want with an is_multiple composite custom column (built from other cols, like tags). The template for the column would create a set of comma-separated values of the form "Genre: value, Category:value, value" or whatever you want. The hard part is the template.

It needs to check several fields for specific values and construct the appropriate list for the custom column.

What I would do is write a custom template function that does the job. Going this route gives you the full power and performance of python, of course at the cost of requiring knowledge of python. The function would get the values of tags, series, and what-have-you (I am not following all the details of the transformations you describe) and would construct the value for the composite column. This post shows some of what you would do in the template function. What is missing is that you can get arbitrary metadata items for a book from the kwargs parameter. For example, kwargs.get('series') will return the value of the series or None if there isn't one. Similarly, kwargs.get('tags') will return a list of tags on the book. You use the lookup name as the parameter for 'get'. From what I understand, you would want a different function in each library, but I am not sure of that.

You could do this in a general program mode template using in_list, list_union, and the like, but it would most likely be very slow.

ilovejedd · 02-21-2012, 02:49 PM

Quote:

Originally Posted by chaley

What I would do is write a custom template function that does the job. Going this route gives you the full power and performance of python, of course at the cost of requiring knowledge of python. The function would get the values of tags, series, and what-have-you (I am not following all the details of the transformations you describe) and would construct the value for the composite column. This post shows some of what you would do in the template function. What is missing is that you can get arbitrary metadata items for a book from the kwargs parameter. For example, kwargs.get('series') will return the value of the series or None if there isn't one. Similarly, kwargs.get('tags') will return a list of tags on the book. You use the lookup name as the parameter for 'get'. From what I understand, you would want a different function in each library, but I am not sure of that.

You could do this in a general program mode template using in_list, list_union, and the like, but it would most likely be very slow.

Okay, that kinda flew over my head (as did the example you linked to), sorry. Yes, I plan on using a different function for the different libraries but if I have an example for how to do it for one library, I think I'll be able figure out the rest.

For example, I have the following:

Code:

Title                                        Series                   Tags
Harry Potter and the Philosopher's Stone     Harry Potter [1.0]       Fiction, Children, Fantasy
Twilight                                     Twilight [1.0]           Fiction, Paranormal

How do I create the following custom columns from the above?

Code:

Title                                        #MySeries                     #Type     #Genre
Harry Potter and the Philosopher's Stone     Series: Harry Potter [1.0]    Fiction   Genre: Children, Genre: Fantasy
Twilight                                     Series: Twilight [1.0]        Fiction   Genre: Paranormal

#MySeries: Basically exactly the same as series except with "Series: " added to the beginning of the string.

#Type: If certain pre-defined values are in tags (Fiction, Non-Fiction, Fanfiction, Magazine), it goes to this column.

#Genre: Basically all tags (excluding Fiction, Non-Fiction, Fanfiction & Magazine) with "Genre: " added to the beginning of string.

I'm trying to understand your sample code but as I'm a total noob to Python, I'm finding it quite difficult to comprehend.

chaley · 02-22-2012, 02:50 AM

Here is a general program mode template that does what I think you want. It puts all values into one custom column, not three

Code:

program:
	categories = 'Fiction, Non-Fiction, Fanfiction, Magazine';
	types = list_intersection(field('tags'), categories, ',');
	genres = list_re(list_difference(field('tags'), categories, ','), ',', '^', 'Genre: ') ;
	series = test(field('series'), strcat('Series: ' , field('series'), ' [', field('series_index'), ']'), '');
	list_union(types, list_union(genres, series, ','), ',')

Here is a custom template function that does the same thing, except that the series index is formatted n.n.

Code:

Function: myFunc or whatever you want
Arg count: 0
Documentation: whatever you want
Program code:
def evaluate(self, formatter, kwargs, mi, locals):
    # create a set of categories
    categories = {'fiction', 'non-fiction', 'fanfiction', 'magazine'}
    # get a list of non-empty tags
    tags = [v.strip() for v in kwargs.get('tags') if v.strip()]
    res = []
    for v in tags:
        if v.lower() in categories:
            # tag is in categories. Add it 'as is' to the result list
            res.append(v)
        else:
            # tag is not in categories. Add it as a genre to the result list
            res.append('Genre: ' + v)
    # add the series if there is one
    series = kwargs.get('series')
    if series:
        res.append('Series: ' + series + (' [%3.1f]'%kwargs.get('series_index')))
    return ', '.join(res)

You call it using the template

Code:

{:'myFunc()'}

Note that if the only thing that varies are the categories, you could pass these in as an argument instead of building them into the function. The function would look something like:

Code:

Function: myFunc or whatever you want
Arg count: 0
Documentation: whatever you want
Program code:
def evaluate(self, formatter, kwargs, mi, locals, cats):
    categories = set([v.strip() for v in cats.split(',') if v.strip()])
    tags = [v.strip() for v in kwargs.get('tags') if v.strip()]
    res = []
    for v in tags:
        if v.lower() in categories:
            res.append(v)
        else:
            res.append('Genre: ' + v)
    series = kwargs.get('series')
    if series:
        res.append('Series: ' + series + (' [%3.1f]'%kwargs.get('series_index')))
    return ', '.join(res)

You call it using the template

Code:

{:'myFunc('fiction, non-fiction, fanfiction, magazine')'}

ilovejedd · 02-22-2012, 10:15 AM

Thanks! I used the last one and it seems to be working great (just had a small typo v,lower() instead of v.lower()). Curious about the series information (Series: My Series [n.n]), wouldn't having the series index in there create a separate collection for each book in a series?

Also, I wonder how do you run the following?

Quote:

Originally Posted by chaley

Here is a general program mode template that does what I think you want. It puts all values into one custom column, not three

Code:

program:
	categories = 'Fiction, Non-Fiction, Fanfiction, Magazine';
	types = list_intersection(field('tags'), categories, ',');
	genres = list_re(list_difference(field('tags'), categories, ','), ',', '^', 'Genre: ') ;
	series = test(field('series'), strcat('Series: ' , field('series'), ' [', field('series_index'), ']'), '');
	list_union(types, list_union(genres, series, ','), ',')

Does this go into templates or do you enter it somewhere else?

chaley · 02-22-2012, 10:25 AM

Quote:

Originally Posted by ilovejedd

Thanks! I used the last one and it seems to be working great (just had a small typo v,lower() instead of v.lower()). Curious about the series information (Series: My Series [n.n]), wouldn't having the series index in there create a separate collection for each book in a series?

Yes it will. That is what I thought you wanted from your #MySeries example.

Quote:

Does this go into templates or do you enter it somewhere else?

It is a template, written using template general program mode.

theducks · 02-22-2012, 10:26 AM

That goes in place of a simple template (anywhere one is allowed)

ilovejedd · 02-22-2012, 11:02 AM

Quote:

Originally Posted by chaley

Yes it will. That is what I thought you wanted from your #MySeries example.

I wanted #MySeries to also be of type series. Now that I've played with custom columns more, I'm guessing that's not allowed for composite columns? In that case, I can just use the metadata plugboards or change the send to device settings to add the series index to the title or filename for sorting purposes and just create a single collection (Series: Name) for each series.

Quote:

Originally Posted by chaley

It is a template, written using template general program mode.

Ah, so that's what general program mode means. I read the manual but couldn't really understand it all that well.

Really, a very, very, very big

for all the help!

ilovejedd · 02-23-2012, 10:54 AM

Woohoo. Just installed a nightly build of PRS+ (thanks Analogus!) and it now supports sub-collections. Granted, it's just one level but that's good enough for me. Here's a revised template to get sub-collections. Basically the same as chaley's script above except it uses Series| and Genre| as the prefix instead of Series: and Genre:

Code:

'''
Function: myFunc
Arg count: 1

Documentation:
myFunc(categories) -- Format tags and series so they show up as sub-collections on the PRS+ custom firmware for Sony Readers.

example:
{:'myFunc('fiction, non-fiction, fanfiction, magazine, news')'} - Creates a top-level collection for any tag named fiction, non-fiction, fanfiction, magazine or news. All other tags will be created as sub-collections under Genre.
'''

def evaluate(self, formatter, kwargs, mi, locals, cats):
    categories = set([v.strip() for v in cats.split(',') if v.strip()])
    tags = [v.strip() for v in kwargs.get('tags') if v.strip()]
    res = []
    for v in tags:
        if v.lower() in categories:
            res.append(v)
        else:
            res.append('Genre|' + v)
    series = kwargs.get('series')
    if series:
        res.append('Series|' + series)
    return ', '.join(res)

I've also created a modified version of the script which would create the sub-collections under the main categories.

e.g.
Fiction, Children, Fantasy -> Fiction, Fiction|Children, Fiction|Fantasy
FanFiction, Harry Potter, Twilight -> FanFiction, FanFiction|Harry Potter, FanFiction|Twilight
News, New York Times -> News, News|New York Times

Code:

'''
Function: myFunc
Arg count: 1

Documentation:
myFunc(categories) -- Format tags and series so they show up as sub-collections on the PRS+ custom firmware for Sony Readers.

example:
{:'myFunc('fiction, non-fiction, fanfiction, magazine, news')'} - Creates a top-level collection for any tag named fiction, non-fiction, fanfiction, magazine or news. All other tags will be created as sub-collections under one of the top-level collections. If a main category isn't found, the book is assigned to Fiction.
'''

def evaluate(self, formatter, kwargs, mi, locals, cats):
    categories = set([v.strip() for v in cats.split(',') if v.strip()])
    tags = [v.strip() for v in kwargs.get('tags') if v.strip()]
    res = []
    booktype = ''
    for v in tags:
        if v.lower() in categories:
            booktype = v
    if booktype == '':
        booktype = 'Fiction'
    res.append(booktype)
    for v in tags:
        if v.lower() not in categories:
            res.append(booktype + '|' + v)
    series = kwargs.get('series')
    if series:
        res.append('Series|' + series)
    return ', '.join(res)

Help optimizing would be much appreciated. I'm sure there's a much better way to get the booktype without having to iterate through all the tags.

Screenshots of how sub-collections work can be found here:
https://www.mobileread.com/forums/sho...postcount=2964

ilovejedd · 02-24-2012, 09:31 PM

Question on manual/only on send metadata management behavior:
If the books already exist on the device and I just want to update the metadata (e.g. to update collections, etc), would selecting "Send to Device" also re-send the actual epubs? Reason I'm asking is because PRS+ sub-collections are still kind of a moving target. Currently, I'm just using subsets of my libraries to test sub-collections (while also trying to figure out what would be the best way to organize my ebooks). Wouldn't want to send 3000 books to my reader only to have to do it over and over again whenever I need to update collections.

chaley · 02-24-2012, 11:17 PM

Quote:

Originally Posted by ilovejedd

Question on manual/only on send metadata management behavior:
If the books already exist on the device and I just want to update the metadata (e.g. to update collections, etc), would selecting "Send to Device" also re-send the actual epubs?

Yes. In manual mode, collections are updated only when the book is actually sent.

Also note that collections are merged when you resend. Removing a tag from the book then resending will not remove the book from that tag's collection.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Sony PRS-T1] seeking advice on library management and collections	Ozric	Library Management	16	11-06-2011 10:51 AM
Sony - 2 questions about automatic management	ThR	Devices	0	03-18-2011 02:57 AM
Sony collection management changes in calibre 0.7.9	chaley	Calibre	30	07-18-2010 08:29 PM
Manual Creation / Management of Collections	ahi	Sony Reader	11	06-25-2009 08:50 AM
sony reader manual	bugsbunny14	Sony Reader	5	09-26-2006 07:06 PM

02-21-2012, 11:14 AM	#4
Terisa de morgan Grand Sorcerer Posts: 6,233 Karma: 11768331 Join Date: Jun 2009 Location: Madrid, Spain Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2	I have manual management and diferent collections according to tags and custom colums, and no problem.

02-21-2012, 12:35 PM	#6
chaley Grand Sorcerer Posts: 11,740 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	@ilovejedd: If you use manual management then sony_collection_name_template is ignored. If you want to have the category name (as in "Genre") in front of the category value(s) you must put it there yourself. You can indeed do what you want with an is_multiple composite custom column (built from other cols, like tags). The template for the column would create a set of comma-separated values of the form "Genre: value, Category:value, value" or whatever you want. The hard part is the template. It needs to check several fields for specific values and construct the appropriate list for the custom column. What I would do is write a custom template function that does the job. Going this route gives you the full power and performance of python, of course at the cost of requiring knowledge of python. The function would get the values of tags, series, and what-have-you (I am not following all the details of the transformations you describe) and would construct the value for the composite column. This post shows some of what you would do in the template function. What is missing is that you can get arbitrary metadata items for a book from the kwargs parameter. For example, kwargs.get('series') will return the value of the series or None if there isn't one. Similarly, kwargs.get('tags') will return a list of tags on the book. You use the lookup name as the parameter for 'get'. From what I understand, you would want a different function in each library, but I am not sure of that. You could do this in a general program mode template using in_list, list_union, and the like, but it would most likely be very slow.

02-22-2012, 10:26 AM	#11
theducks Well trained by Cats Posts: 29,792 Karma: 54830978 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	That goes in place of a simple template (anywhere one is allowed)

02-24-2012, 09:31 PM	#14
ilovejedd hopeless n00b Posts: 5,111 Karma: 19597086 Join Date: Jan 2009 Location: in the middle of nowhere Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9	Question on manual/only on send metadata management behavior: If the books already exist on the device and I just want to update the metadata (e.g. to update collections, etc), would selecting "Send to Device" also re-send the actual epubs? Reason I'm asking is because PRS+ sub-collections are still kind of a moving target. Currently, I'm just using subsets of my libraries to test sub-collections (while also trying to figure out what would be the best way to organize my ebooks). Wouldn't want to send 3000 books to my reader only to have to do it over and over again whenever I need to update collections.

Advert

Advert