Plugboard, template, and custom composite column recipes

[eschwartz]

You'd need to have them all have some uniform style, or writing the regex could take nearly as long as doing it manually.

We could do it with multiple passes. For instance,

Find:

(^[^-]+)( - .*)?(: A Novel|other junk endings)?

Replace:

\1

Should chop off everything including and after a "-" and removes pesky ": A Novel" and other standardized junk endings.

Find:

(^[^:]+):.*(\[\d\]|, Book \d+)

Replace:

\1

Removes anything after a ":" but only if it also matches a "[#]" or the string ", Book #" where # is an actual number of any length. Presumably those are always series' of some sort.

These two should cover every example you gave.

EDIT: Combine the two together with (I think)

(^[^-:]+)(: (A Novel|other junk endings|(.*(\[\d\]|, Book \d+))))?( - .*)?

I hope you don't have any more styles to filter, though, or this may end up becoming monstrous.

[crissman]

Excellent.. This is exactly what I needed.. works perfectly.

as for other formats.. Umm Blush.. just noticed when trying this out that there is also a:

Horizon Storms: The Saga of Seven Suns - Book 3

there is a - instead of a , like the others where.

[eschwartz]

Try:

(^[^-:]+)(: (A Novel|other junk endings|(.*(\[\d\]|(,|-) Book \d+))))?( - .*)?

[crissman]

Originally Posted by eschwartz

Try:

(^[^-:]+)(: (A Novel|other junk endings|(.*(\[\d\]|(,|-) Book \d+))))?( - .*)?

Yup!! that did it, I think I need to study programming :P I am too OCD not to.

Thanks again eschwartz

[MiniMouse]

Can anyone tell me how to make a re(pattern, replacement) template for multible pattern, replacements in a custom column?

[chaley]

Originally Posted by MiniMouse

Can anyone tell me how to make a re(pattern, replacement) template for multible pattern, replacements in a custom column?

A bit more information will be helpful. Is the custom column a tags-like column? What sort of patterns do you want to replace with what sort of replacements? Will the pattern appear more than once in the string? Examples are good.

[MiniMouse]

Originally Posted by chaley

A bit more information will be helpful. Is the custom column a tags-like column? What sort of patterns do you want to replace with what sort of replacements? Will the pattern appear more than once in the string? Examples are good.

I'm trying to organize my fanfiction columns, more specific my relationsship column (I think tags-like, I'm not home at the moment, so I can't look). I want the pairings look the same what ever site taken from and I thought that might be easier over the custom column than the personal.ini of FFDL.

For example:

Sherlock Holmes/John Watson, Sebastian Moran/Jim Moriarty

or

S. Holmes/J. Watson, S. Moran/J. Moriarty

would be

Holmes/Watson, Moran/Moriarty

I know I can do that (when written like my first example) for one pairing like

{#relationship:re(Sherlock Holmes/John Watson, Holmes/Watson)}

So how do I do that for a whole list of pairings?

Mini

[chaley]

Originally Posted by MiniMouse

I'm trying to organize my fanfiction columns, more specific my relationsship column (I think tags-like, I'm not home at the moment, so I can't look). I want the pairings look the same what ever site taken from and I thought that might be easier over the custom column than the personal.ini of FFDL.

...

So how do I do that for a whole list of pairings?

Mini

What you want to do is very hard with regular expressions. Basically you would need to apply a separate regexp for each possible pairing, because there is no way to express A->B, C->D in a regular expression.

If your relationship names all have the form ("some characters" space "desired name") (all of your examples do have this form) then you can easily do what you want by stripping all the leading characters from the name to leave the last word. This GPM template shows how to do that.

program:
	sample = 'S. Holmes/J. Watson, S. Moran/J. Moriarty';
	list_re(sample, ',',  '^.* ([^ ]*)/.* ([^ ]*$)', '\1/\2')

"Sample" would become "field('#relationship')"

This scheme will not work if the last word must change based on the original. For example, it won't work if you want 'S. Holmes/J. Watson' to become 'Holmes/Whatnot" because whether or not to change from Watson to Whatnot depends on the seeing "Watson" and not "Moran". It also will not work if last names can be two words, such as von Trapp.

If in fact you really want a series of expressions of the form A becomes B then you would need to write each of them as a separate regular expression. If there are more than a few of them, the performance would be terrible.

An alternate approach would be to write a custom template function that uses python dictionaries to make the changes. This would have acceptable performance (I think) but requires you to know how to code in python. The maintenance would also be problematic because all changes would need to be made in python.

Yet another alternate approach would be to write a calibre python script that processes the relationship column to create another tags-like column containing the values you want. This would have great performance and be somewhat easier to maintain, but still suffers from having to do it in python. An example of what this might look like is under the spoiler.

Spoiler:

def init_cache(library_path):
	from calibre.db.backend import DB
	from calibre.db.cache import Cache
	backend = DB(library_path)
	cache = Cache(backend)
	cache.init()
	return cache

from collections import defaultdict
	
cache = init_cache(library_path = sys.argv[1])

transformation_dict = {
	'Sherlock Holmes': 'Holmes',
	'S. Holmes': 'Holmes',
	'John Watson': 'Watson',
	'J. Watson': 'Watson',
	'Sebastian Moran': 'Moran',
	'S. Moran': 'Moran',
	'Jim Moriarty': 'Moriarty',
	'J. Moriarty': 'Moriarty'
	}

values_to_write = {}
for id_ in cache.all_book_ids():
	relationships = cache.field_for('#mytextmult', id_)
	new_relationships = []
	if relationships:
		for relationship in relationships:
			pair = relationship.split('/')
			if len(pair) == 2:
				left, right = pair
				newleft = transformation_dict.get(left, left)
				newright = transformation_dict.get(right, right)
				new_relationships.append(newleft + '/' + newright)
			else:
				new_relationships.append(relationship)
		values_to_write[id_] = new_relationships
print values_to_write

cache.set_field('#mytextmult', values_to_write)

[MiniMouse]

Originally Posted by chaley

Yet another alternate approach would be to write a calibre python script that processes the relationship column to create another tags-like column containing the values you want. This would have great performance and be somewhat easier to maintain, but still suffers from having to do it in python. An example of what this might look like is under the spoiler.

Spoiler:

def init_cache(library_path):
	from calibre.db.backend import DB
	from calibre.db.cache import Cache
	backend = DB(library_path)
	cache = Cache(backend)
	cache.init()
	return cache

from collections import defaultdict
	
cache = init_cache(library_path = sys.argv[1])

transformation_dict = {
	'Sherlock Holmes': 'Holmes',
	'S. Holmes': 'Holmes',
	'John Watson': 'Watson',
	'J. Watson': 'Watson',
	'Sebastian Moran': 'Moran',
	'S. Moran': 'Moran',
	'Jim Moriarty': 'Moriarty',
	'J. Moriarty': 'Moriarty'
	}

values_to_write = {}
for id_ in cache.all_book_ids():
	relationships = cache.field_for('#mytextmult', id_)
	new_relationships = []
	if relationships:
		for relationship in relationships:
			pair = relationship.split('/')
			if len(pair) == 2:
				left, right = pair
				newleft = transformation_dict.get(left, left)
				newright = transformation_dict.get(right, right)
				new_relationships.append(newleft + '/' + newright)
			else:
				new_relationships.append(relationship)
		values_to_write[id_] = new_relationships
print values_to_write

cache.set_field('#mytextmult', values_to_write)

Dealing with templates and regex I learned by now.
Now it seems I have to learn something new... again. I think the last option could be more my solution BUT where do I put something like this? Never have done that before. Now I need a bit more information

Mini

[chaley]

Originally Posted by MiniMouse

Dealing with templates and regex I learned by now.
Now it seems I have to learn something new... again. I think the last option could be more my solution BUT where do I put something like this? Never have done that before. Now I need a bit more information

Mini

Scripts like my last alternative do not go anywhere in calibre. Instead they are "command line" utilities that you run whenever you wish. If the script is in the file "foo" then to run it you would first ensure that calibre is not running then use

calibre-debug -e foo /path/to/your/library

One reason that these scripts can be very fast is that they can process the entire library at once instead of processing it book-by-book. For example, the script I provided computes the value of the target column for all the books then uses the calibre API to write all the books at once.

These scripts are not the easiest thing to understand, especially if you are not a programmer. If you want to try to use it or something like it, feel free to ask questions.

Do note that it might be easier in the long run to change the ini file of the fanfic downloader.

[MiniMouse]

Originally Posted by chaley

Scripts like my last alternative do not go anywhere in calibre. Instead they are "command line" utilities that you run whenever you wish. If the script is in the file "foo" then to run it you would first ensure that calibre is not running then use

calibre-debug -e foo /path/to/your/library

One reason that these scripts can be very fast is that they can process the entire library at once instead of processing it book-by-book. For example, the script I provided computes the value of the target column for all the books then uses the calibre API to write all the books at once.

These scripts are not the easiest thing to understand, especially if you are not a programmer. If you want to try to use it or something like it, feel free to ask questions.

Do note that it might be easier in the long run to change the ini file of the fanfic downloader.

Thanks @chaley

I will think about what to do and be sure, I come back when I have questions

[S3NTYN3L]

Can anyone please explain why this recipe isn't working for me?

Metadata Plugboard:
format:mobi
device:kindle2
source template1: {series_index:0>4.1f|| - {title}}
dest field1: title
source template2: {author}{#subseries_index:>2s| - {#subseries}| - }
dest field2:authors


OK, now the first template works just fine.
Heck, even the second template works fine when incorporated into the first, (minus the {author} bit of course).

My expected output for the second template is:
AuthorName - SubSeriesTitle - 1

What I end up with no matter where I put the {author} tag is:
AuthorFirstName - SubSeriesTitle - 1 AuthorLastName


Looking at it on the KT the books get sorted by SubSeriesTitle alphabetically.
Then by subseries_index, albeit correctly, then correctly by the title template.


So, an example of what's displayed on the KT if the above confused you:

25.0 - A Kingdom Besieged
Raymond E. - Chaoswar Saga - 1 Feist

26.0 - A Crown Imperiled
Raymond E. - Chaoswar Saga - 2 Feist

27.0 - Magician's End
Raymond E. - Chaoswar Saga - 3 Feist

17.0 - Talon of the Silverhawk
Raymond E. - Conclave of Shadows - 1 Feist

And so on and so forth...

For what it's worth, any books NOT having a subseries are displayed as expected via the above templates.

I mean like;
Robot Dreams
Isaac Asimov

and

01.0 - The Early Asimov - Volume 1
Isaac Asimov
Also,my Calibre library displays everything just as you would expect. I mean, Series then SubSeries in the correct order.


Can someone please assist?

[eschwartz]

That is because the template is sending the following author name, just as you told it to:

Feist, Raymond E. - Chaoswar Saga - 1

Now take a look at what happens when you transform that from LN, FN <--> FN LN.

The Kindle automatically switches LN, FN to FN LN in the display field, and uses the LN section for the sort values. (MOBI/AZW3 does not have a separate author_sort and author field.)

Some people have gotten around this by using {author_sort:'re(",", "_")'} but you will sacrifice the ability to sort based on author names.

[eschwartz]

Cross-posting an interesting idea from elsewhere, putting the series into the author field on a Kindle and displaying properly:

Originally Posted by PRS-T2

{author_sort:||,}{series:| - | }{series_index:0>3s}

Target = author_sort and authors.

I was happy a bit too early: for books where there is no series information, the Kindle still displays "firstname lastname", but the sorting is all right across all books, so...

Maybe that can be tricked with a blank after the comma. Will try it tonight.

Also, if a series has more than one author, like authors Preston/Child with the Agent Pendergast series, the series is grouped, however the index is wrong. Books will then be ordered by first mentioned author (Preston), then title. But at least all information is visible incl. index numbers, and the series is displayed in one block. I got used to this behaviour with my Sony already.

Attention: the plug will NOT work for series written by several independent authors, in my case there's only one: Justifiers. It's written so badly however, that I don't think I'll read it again
In such case I just use the search field which works like a charm and suggests the whole series after typing in a few letters...

Originally Posted by eschwartz

That is interesting. I don't think anyone thought of adding a second comma to author_sort.

Originally Posted by PRS-T2

Me neither until I started thinking what the hell the kindle does to the author_sort field after sorting. The comma idea came after you explained the kindle way of handling that field as far as sorting and display goes.


Without that comma, it displays the line like this:
<Greg - Amboss 001 Bear>
With comma, either added to the series or author sort, or just put between them:
<Bear, Greg, - Amboss 001>
Which I prefer for obvious reasons

[PRS-T2]

Here's what it looks like on the Paperwhite2 with FW 5.4.5.1:

(image attached)