Plugboard, template, and custom composite column recipes

[nynaevelan]

I think it might be a combination of all the parts of my plugboard, here is what I have in there:

program:start=list_item(field('tags'),-2, ' - ');end=list_item(field('tags'),-1, ' - ');part3=list_item(end,-2,'/');part4=list_item(end,-1,'/');part3alt=list_item(part4,-2,' ');part4=list_item(part4,-1,' ');strcat(shorten(start,1,'',0),test(start,' - ',''),shorten(part3,1,'',0),test(part3,'/',''),shorten(part3alt,1,'',0),test(part3alt,' ',''),shorten(part4,1,'',0),) - {title} - {#stripped_series:lookup(.\s,#initials,.,#shortened,series)}{series_index:0>2s| [|] } - {#purchased1:.3||}

The different parts include the tags, title, series, series index and custom purchased column. Other than the title I use all these pieces to ease my collections creation and of course to know the series info, so I need all the pieces to show up. I like the tags first because that is the main collections piece. Other than that I do not care what order they show up in as long as the one line is short enough I can read most of the title and the series info. Thus what started this journey.

[eschwartz]

Originally Posted by nynaevelan

I think it might be a combination of all the parts of my plugboard, here is what I have in there:

program:start=list_item(field('tags'),-2, ' - ');end=list_item(field('tags'),-1, ' - ');part3=list_item(end,-2,'/');part4=list_item(end,-1,'/');part3alt=list_item(part4,-2,' ');part4=list_item(part4,-1,' ');strcat(shorten(start,1,'',0),test(start,' - ',''),shorten(part3,1,'',0),test(part3,'/',''),shorten(part3alt,1,'',0),test(part3alt,' ',''),shorten(part4,1,'',0),) - {title} - {#stripped_series:lookup(.\s,#initials,.,#shortened,series)}{series_index:0>2s| [|] } - {#purchased1:.3||}

The different parts include the tags, title, series, series index and custom purchased column. Other than the title I use all these pieces to ease my collections creation and of course to know the series info, so I need all the pieces to show up. I like the tags first because that is the main collections piece. Other than that I do not care what order they show up in as long as the one line is short enough I can read most of the title and the series info. Thus what started this journey.

I know what the problem is, we are mixing program and template language!!!

Since I'm the one who knows what he's doing in the land of program:, I should've anticipated this and warned you, sorry.

This will literally be the easiest step, it's really easy to convert from template to program mode. Give me a few minutes...

[eschwartz]

Latest plugboard -- this is the biggest monster of all. It took longer than I expected, but ONLY because,... I have taken the liberty of combining the famous series-stripping plugboard into this, so you can just get rid of those custom columns.

(If you would look at the template guide on the calibre manual, it actually shows that set of templates, but includes "The following program produces the same results as the original recipe, using only one custom column to hold the results of a program that computes the special title value:" and then gives the second part of my program.)

EDIT: In red is the series-stripping plugboard, which replaces the custom columns. In bold underlined green is the part of it that actually has the same text as the custom columns.

Here goes:

program:
#	save stuff before " - "
start=list_item(field('tags'),-2, ' - ');
#	save stuff after it
end=list_item(field('tags'),-1, ' - ');
#	divides 'end' into two, at '/'
part3=list_item(end,-2,'/');
part4=list_item(end,-1,'/');
#	divides last element for multi-word
part3alt=list_item(part4,-2,' ');
part4=list_item(part4,-1,' ');

#		return a value
thetags=strcat(
#		add first initial
	shorten(start,1,'',0),
#		add first separator if first initial not empty
	test(start,' - ',''),
#		add second initial
	shorten(part3,1,'',0),
#		add second separator if third not empty
	test(part3,'/',''),
#		if last section had ' ' not '/', use these two lines instead
##			add third initial part 1
	shorten(part3alt,1,'',0),
##			if third initial part 1 exists, add ' ' separator
	test(part3alt,' ',''),
#		add third initial
	shorten(part4,1,'',0),
);


#		compute the equivalent of the composite fields
#		and store them in local variables
	stripped = re(field('series'), '^(A|The|An)\s+', '');
	shortened = shorten(stripped, 4, '-' ,4);
	initials = re(stripped, '[^\w]*(\w?)[^\s]+(\s|$)', '\1');

#		Format the series index. Ends up as empty if there is no
#		series index. Note that leading and trailing spaces will be
#		removed by the formatter, so we cannot add them here.
#		We will do that in the strcat below. Also note that because
#		we are in 'program' mode, we can freely use curly brackets
#		in strings, something we cannot do in template mode.
	s_index = template('{series_index:0>2.0f}');

#		print(stripped, shortened, initials, s_index);

#		Now concatenate all the bits together. The switch picks
#		between initials and shortened, depending on whether
#		there is a space in stripped. We then add the brackets
#		around s_index if it is not empty. My version this is not
#		the last bit, but we save this as "theseries" for later use.
	theseries=strcat(
		switch( stripped,
			'.\s', initials,
			'.', shortened,
			field('series')),
		test(s_index, strcat(' [', s_index, '] '), ''));

#		put all the bits together, and return a monster value
	strcat(
		thetags,
		' - ',
		theseries,
		' - ',
		finish_formatting(
			field('series_index'),
			'0>2s',
			' [',
			'] '
		),
		' - ',
		substr(field('#purchased1'),0,3)
	)

Simplified into one line, to go straight into the plugboard box:

program:start=list_item(field('tags'),-2, ' - ');end=list_item(field('tags'),-1, ' - ');part3=list_item(end,-2,'/');part4=list_item(end,-1,'/');part3alt=list_item(part4,-2,' ');part4=list_item(part4,-1,' ');thetags=strcat(shorten(start,1,'',0),test(start,' - ',''),shorten(part3,1,'',0),test(part3,'/',''),shorten(part3alt,1,'',0),test(part3alt,' ',''),shorten(part4,1,'',0),);stripped = re(field('series'), '^(A|The|An)\s+', '');shortened = shorten(stripped, 4, '-' ,4);initials = re(stripped, '[^\w]*(\w?)[^\s]+(\s|$)', '\1');s_index = template('{series_index:0>2.0f}');theseries=strcat(switch( stripped,'.\s', initials,'.', shortened,field('series')),test(s_index, strcat(' [', s_index, '] '), ''));strcat(thetags,' - ',theseries,' - ',field('title'),' - ',finish_formatting(field('series_index'),'0>2s',' [','] '),' - ',substr(field('#purchased1'),0,3))

[nynaevelan]

YAY!!! You did it!!! You are my Plugboard king!!

Oh and to the template stuff, went zipping right past my head.

[eschwartz]

Originally Posted by nynaevelan

YAY!!! You did it!!! You are my Plugboard king!!

Oh and to the template stuff, went zipping right past my head.

You probably created three custom columns as per the first post:

Originally Posted by chaley

General Metadata

Metadata: Put series into the title, using either initials or a shortened form. Strip leading articles from the series name (any) (inspired by Gary_M_Mugford)

Spoiler:

The solution requires creating three composite columns. The first column is used to remove the leading articles. The second is used to compute the 'shorten' form. The third is to compute the 'initials' form. Once you have these columns, the plugboard selects between them. You can hide any or all of the three columns on the library view.

First column:

Name: #stripped_series. Template: {series:re(^(A|The|An)\s+,)||}

Second column (the shortened form):

Name: #shortened. Template: {#stripped_series:shorten(4,-,4)}

Third column (the initials form):

Name: #initials. Template: {#stripped_series:re(([^\s])[^\s]+(\s|$),\1)}

Plugboard expression:

Template:{#stripped_series:lookup(.\s,#initials,.,#shortened,series)}{series_index:0>2s| [|] }{title}
Destination field: title

This set of fields and plugboard produces:
Series: The Lord of the Rings
Series index: 2
Title: The Two Towers
Output: LotR [02] The Two Towers

Series: Dahak
Series index: 1
Title: Mutineers Moon
Output: Dahak [01] Mutineers Moon

Series: Berserkers
Series Index: 4
Title: Berserker Throne
Output: Bers-kers [04] Berserker Throne

Series: Meg Langslow Mysteries
Series Index: 3
Title: Revenge of the Wrought-Iron Flamingos
Output: MLM [03] Revenge of the Wrought-Iron Flamingos

You can delete those now, as I incorporated programming into my plugboard that handles it all. I edited the latest plugboard to highlight that part, as you can see, the green stuff actually has the same name=value of your custom columns.

That's all, really.

[chaley]

@eschwartz: it is nice to see someone using the more powerful bits of template language.

For something as complicated as this template you might consider using a custom template function. This is especially true if the template is for a composite column. For a save template, performance is not such a concern.

Re the one line templates: I think you can right click on the save box and open a template editor. Also, there is no problem pasting in a template containing newlines. The same template parser is used for all program mode templates, no matter where they come from.

[eschwartz]

Originally Posted by chaley

@eschwartz: it is nice to see someone using the more powerful bits of template language.

For something as complicated as this template you might consider using a custom template function. This is especially true if the template is for a composite column. For a save template, performance is not such a concern.

Re the one line templates: I think you can right click on the save box and open a template editor. Also, there is no problem pasting in a template containing newlines. The same template parser is used for all program mode templates, no matter where they come from.

It says in the manual about the plugboard itself not liking them.

It would be possible to do the above with no custom columns by putting the program into the template box of the plugboard. However, to do so, all comments must be removed because the plugboard text box does not support multi-line editing. It is debatable whether the gain of not having the custom column is worth the vast increase in difficulty caused by the program being one giant line.

I never thought to right-click and check for that, or that it might have changed since then. I will have to keep this in mind for the future.

As for custom functions -- they don't suffer the performance impact of templates? I figured it's the same calculating whether in the function or out of it.... Oh, but I remember that general program mode is supposed to run faster than template mode since it is compiled to python. This is the next level up of that? If so, cool and I will have to remember that too.

Helping people is so much fun, for totally selfish reasons as well -- I learn so much more this way!

And thank you for creating this wonderful tool in the first place!!

[ACGAuthor]

My apologies if these have already been asked. I've searched the forums but I can't seem to find anything that answers my question.

A few questions (probably very obvious answers that I'm missing):

First, I am using the "General Metadata" 3-part method to add series tagging (I only discovered this new method eschwartz posted today, and I'm not entirely certain what all it does so I haven't tried it yet.) The only thing I changed on the General Metadata method was that I made the hyphenated version 6,-,6 instead of 4,-,4 because nothing is more annoying than seeing a hyphen "Mountains" abbreviated to "Moun-ains". 12 letters seems to be a good cut-off with the one-word series I have.

Anyway. When I first installed the plugboard, it worked great. Exactly as it was meant to. And when I changed the 4s to 6s, it did what I wanted it to with the titles (though I'm not entirely sure how to get it to update those titles on my PW (v1) except for removing the books and re-sideloading them.)

However. When I cashed in my gift cards and bought a new batch of ebooks and tried to add them, it would not send them to the PW with the series in the titles. So I have some books on the PW with (for example) "[ABCD] Book in ABCD series" and then more recent ones from that same series with just "Book in ABCD series". I don't know why it's not doing the changes for the most recent additions (likely I'm missing something, but I can't find what.)

Second: When I have a series with an ampersand in the title (ex. "Fine & Dandy") the abbreviated form comes out "F& D." Is there some way to get rid of that space between the ampersand and the second initial?

Third: Has anyone figured out how to do a plugboard to strip the "personal" banner from mobi docs with PDOC the personal doc field rather than having to re-convert the files? I saw it mentioned as a possibility in another thread about getting rid of the personal banner but the person discussing it hadn't had any success. Has someone cracked this problem, yet?

Thank you so much and happy holidays!

ETA: One more thing, though it's a little out of the scope of this particular thread. When setting up user categories (which are then made into Kindle collections), I would like to create user categories to shorten my lists of authors (Authors A-D, E-H, I-M etc etc.) Right now what I'm doing is I'm adding a tag (Author A-D, or whatever) to the tags, and then adding that tag to the custom user category.

But that is a bit clunky because it means each time I have to add a batch of books, I have to go through and add the appropriate tag, unlike, for example, a user category by series or author name where it gets updated automatically (assuming the series/author name is correct when the book is added, of course).

Would there be some way to make this more of an automated process, if the author's last name begins with a certain letter?

And to make it even more complex, say the I want to sort by author groups withinin a particular genre. If something has a tag of, say "Urban Fantasy" or "Romance" or whatever, could I sort it by author then? I'm not good enough with composing searches to come with with that specific of a search, and then I don't know how to translate those search results into something I can add to a custom user category.

Thank you again!

[eschwartz]

Originally Posted by ACGAuthor

My apologies if these have already been asked. I've searched the forums but I can't seem to find anything that answers my question.

A few questions (probably very obvious answers that I'm missing):

First, I am using the "General Metadata" 3-part method to add series tagging (I only discovered this new method eschwartz posted today, and I'm not entirely certain what all it does so I haven't tried it yet.) The only thing I changed on the General Metadata method was that I made the hyphenated version 6,-,6 instead of 4,-,4 because nothing is more annoying than seeing a hyphen "Mountains" abbreviated to "Moun-ains". 12 letters seems to be a good cut-off with the one-word series I have.

Anyway. When I first installed the plugboard, it worked great. Exactly as it was meant to. And when I changed the 4s to 6s, it did what I wanted it to with the titles (though I'm not entirely sure how to get it to update those titles on my PW (v1) except for removing the books and re-sideloading them.)

However. When I cashed in my gift cards and bought a new batch of ebooks and tried to add them, it would not send them to the PW with the series in the titles. So I have some books on the PW with (for example) "[ABCD] Book in ABCD series" and then more recent ones from that same series with just "Book in ABCD series". I don't know why it's not doing the changes for the most recent additions (likely I'm missing something, but I can't find what.)

Second: When I have a series with an ampersand in the title (ex. "Fine & Dandy") the abbreviated form comes out "F& D." Is there some way to get rid of that space between the ampersand and the second initial?

Third: Has anyone figured out how to do a plugboard to strip the "personal" banner from mobi docs with PDOC the personal doc field rather than having to re-convert the files? I saw it mentioned as a possibility in another thread about getting rid of the personal banner but the person discussing it hadn't had any success. Has someone cracked this problem, yet?

Thank you so much and happy holidays!

The one I just posted was a customized plugboard that used a specific custom column to meet an esoteric need. Chances are you don't need exactly the same thing as nynaevelan.

But we can help you figure out what YOU need.

Firstly, you are correct. The only way to get the updated metadata onto the Kindle is to update the book file. (Though other readers can be different. I believe the Kobo handles this with a database calibre can directly inform of the changes.)

Can you post the exact set of plugboards you are using and an example book title you wish to see?

And I'm afraid there's no getting around the PDOC problem. All ebooks are considered PDOCs unless they have an ASIN (amazon identifier metadata) that matches a book in the Kindle Store. If a book comes from Amazon and has not been converted, it will keep that ASIN. Otherwise, Amazon cannot match it to anything on their servers and will instead give it the PDOC banner.

[eschwartz]

Originally Posted by ACGAuthor

ETA: One more thing, though it's a little out of the scope of this particular thread. When setting up user categories (which are then made into Kindle collections), I would like to create user categories to shorten my lists of authors (Authors A-D, E-H, I-M etc etc.) Right now what I'm doing is I'm adding a tag (Author A-D, or whatever) to the tags, and then adding that tag to the custom user category.

But that is a bit clunky because it means each time I have to add a batch of books, I have to go through and add the appropriate tag, unlike, for example, a user category by series or author name where it gets updated automatically (assuming the series/author name is correct when the book is added, of course).

Would there be some way to make this more of an automated process, if the author's last name begins with a certain letter?

And to make it even more complex, say the I want to sort by author groups withinin a particular genre. If something has a tag of, say "Urban Fantasy" or "Romance" or whatever, could I sort it by author then? I'm not good enough with composing searches to come with with that specific of a search, and then I don't know how to translate those search results into something I can add to a custom user category.

Thank you again!

All searches are boolean -- you can type in the search field

tags:"=NameOfMyTag" and authors:"=The Author"

The boolean operators "and" "not" "and not" & "or" can be used to group your search terms however you like.

You can also click any element in the tag browser on the right, and by holding down Ctrl select another element as part of the same search. The button "alter tag browser" will give you the option to choose "search type when selecting multiple items" whether to use "and" or "or".

Wherever you are adding the authors' initial, you can use the template "{author_sort[0]}" to get the author's sort value "LN, FN" with only the [0] character, where the first character in the string is "0" away from the beginning. You will need to use it in a custom column built from other columns.

EDIT: Or rather, the column should be equal to

Authors {author_sort[0]:switch([A-D],A-D,[E-H],E-H,[I-M],I-M,none)}

Continue filling out the middle with comma-separated pattern,value pairs, then end with "none" for unmatched values. Even though there should never be unmatched values, the program requires you to give a fallback.

[ACGAuthor]

Originally Posted by eschwartz

The one I just posted was a customized plugboard that used a specific custom column to meet an esoteric need. Chances are you don't need exactly the same thing as nynaevelan.

But we can help you figure out what YOU need.

You're awesome, thanks!

Firstly, you are correct. The only way to get the updated metadata onto the Kindle is to update the book file. (Though other readers can be different. I believe the Kobo handles this with a database calibre can directly inform of the changes.)

Okay, I thought maybe there was some way to use the "Match with Library" feature or something to get it to update metadata. But nope, looks like I need to delete the book in question, then re-sideload. Gotcha.

Can you post the exact set of plugboards you are using and an example book title you wish to see?

Okay, this is the plugboard I started with:

General Metadata

Metadata: Put series into the title, using either initials or a shortened form. Strip leading articles from the series name (any) (inspired by Gary_M_Mugford)

Spoiler:

The solution requires creating three composite columns. The first column is used to remove the leading articles. The second is used to compute the 'shorten' form. The third is to compute the 'initials' form. Once you have these columns, the plugboard selects between them. You can hide any or all of the three columns on the library view.

First column:

Name: #stripped_series. Template: {series:re(^(A|The|An)\s+,)||}

Second column (the shortened form):

Name: #shortened. Template: {#stripped_series:shorten(4,-,4)}

Third column (the initials form):

Name: #initials. Template: {#stripped_series:re(([^\s])[^\s]+(\s|$),\1)}

Plugboard expression:

Template:{#stripped_series:lookup(.\s,#initials,.,#shortened,series)}{series_index:0>2s| [|] }{title}
Destination field: title

This set of fields and plugboard produces:
Series: The Lord of the Rings
Series index: 2
Title: The Two Towers
Output: LotR [02] The Two Towers

Series: Dahak
Series index: 1
Title: Mutineers Moon
Output: Dahak [01] Mutineers Moon

Series: Berserkers
Series Index: 4
Title: Berserker Throne
Output: Bers-kers [04] Berserker Throne

Series: Meg Langslow Mysteries
Series Index: 3
Title: Revenge of the Wrought-Iron Flamingos
Output: MLM [03] Revenge of the Wrought-Iron Flamingos

But like I said, I customized the second column part to make the limit of a word that is shortened and hyphenated 12 letters instead of 8, like such:

[CODE]Second column (the shortened form):

Name: #shortened. Template: {#stripped_series:shorten(6,-,6)}

So this is the book title I'd like to see:

[Cadeleonian] Lord of the White Hell, Part One

ETA: Of course, that doesn't address the ampersand issue. For example, right now a book in the series "Cut & Run" shows up as "[C& R] Title" when I would rather have it "[C&R] Title" (or "[C & R] Title" will do, but that defeats the purpose of trying to make things as compact as possible.

Would it also be possible to change the word "and" in a series title to an ampersand when abbreviating? So if the series was actually "Cut and Run" it would be abbreviated "C&R" instead of "CR"?

And I'm afraid there's no getting around the PDOC problem. All ebooks are considered PDOCs unless they have an ASIN (amazon identifier metadata) that matches a book in the Kindle Store. If a book comes from Amazon and has not been converted, it will keep that ASIN. Otherwise, Amazon cannot match it to anything on their servers and will instead give it the PDOC banner.

Okay. I tried to use the QualityControl plugin to add the ASIN as it recommended in another thread, but it didn't seem to work on everything I added. Particularly since some of the books I have aren't available on Amazon, lol. So I guess I will just keep re-converting. Thank you!

[ACGAuthor]

Originally Posted by eschwartz

All searches are boolean -- you can type in the search field
The boolean operators "and" "not" "and not" & "or" can be used to group your search terms however you like.

You can also click any element in the tag browser on the right, and by holding down Ctrl select another element as part of the same search. The button "alter tag browser" will give you the option to choose "search type when selecting multiple items" whether to use "and" or "or".

Wherever you are adding the authors' initial, you can use the template "{author_sort[0]}" to get the author's sort value "LN, FN" with only the [0] character, where the first character in the string is "0" away from the beginning. You will need to use it in a custom column built from other columns.

Okay, excellent. Creating that last initial column would be exactly what I need. Then I can add those to my user category.

BUT I'm still not entirely clear on how to add those to my user category ONLY IF the tag is for a specific genre or whatever.

Would it be possible to do a series of custom columns to make that happen? For Example, A custom column with the last initial, and then a custom column with the last initial if it has a tag for a specific genre?

ETA: Wait, saw you edited while I was replying. Will get back to you.

ETA2: Ohh excellent. Now I just need to figure how to make it genre-specific and I'm golden!

[eschwartz]

[Cadeleonian] Lord of the White Hell, Part One

I am assuming this is based on

series:Cadelonian
series #:1
title:Lord of the White Hell

You will need the plugboard

{#stripped_series:lookup(.\s,#initials,.,#shortened,series)|[|] }{title}{series_index:|, Part |}

And that will get you

[Cadeleonian] Lord of the White Hell, Part 1

(The series number is stored as an integer,... not much I can do)

[ACGAuthor]

Originally Posted by eschwartz

I am assuming this is based on

series:Cadelonian
series #:1
title:Lord of the White Hell

You will need the plugboard

{#stripped_series:lookup(.\s,#initials,.,#shortened,series)|[|] }{title}{series_index:|, Part |}

And that will get you
(The series number is stored as an integer,... not much I can do)

Actually, I am being stupid. Sorry. I meant:

Cadelonian [01] Lord of the White Hell, Part One

I got myself confused. I blame the massive quantity of food I'm digesting after the holiday meal.

[eschwartz]

series:Cadelonian
series #:1
title:Lord of the White Hell, Part One


plugboard:

{#stripped_series:lookup(.\s,#initials,.,#shortened,series)}{series_index:0>2s| [|] }{title}

should do you just fine, then. What are you getting instead?