Possible bugs with new "Languages" field.

[arspr]

Hi, I do really like this new feature.

Moreover, I did have in my Calibre database two custom fields for language: the first one for the current book language (#language) and the second one for the original language of the book (#original_language).

But, although I'm not sure I think there are some bugs with this feature.

1st one. Select "Greek, Ancient (to 1453)"
Voila you get an error. I think the problem is that it should be "Greek; Ancient (to 1453)" like "Spanish; Castilian".

2nd one. Comparison with the new Languages field doesn't work
Because of the new Languages field, I intend to erase my previous #language column, while keeping #original_language one.

In addition to #language and #original_language, I have got a calculated "Original Language?" field (with a "#original" lookup name).

The template for this field is

{#original:'strcmp(field('#original_language'), field('#language'), 'No', 'Yes', 'No')'}

With this code, it perfectly works (see Old-Right.jpg screenshot)

But if I update the #original field template, using Languages instead of #language, it stops working.

So this code:

{#original:'strcmp(field('#original_language'), field('Languages'), 'No', 'Yes', 'No')'}

doesn't work, (see New-Wrong.jpg).

Am I doing something wrong? or is it a bug?

[kovidgoyal]

1) There is indeed a problem with language names that have commas in them. Fix for that will be in next release.

2) The field name is languages with a lowerwcase l. Also, behind the scenes, the languages field does not actually store language names, rather it stores 3 letter ISO 639 language codes. So for strcmp to work your original language field would also need to store the ISO 639 codes.

[arspr]

Wow, thanks for your reply.

About 2). Confirmed. I've just used eng instead of "English" and spa instead of "Spanish; Castilian" within #original_language and it perfectly works.

But with this situation, it would be really nice if you could add a new possible "Languages"-type column in the "Add your own columns" section. I mean from a plain user point of view, it's quite odd storing eng, spa, or whatever other needed code.

[theducks]

Originally Posted by arspr

Wow, thanks for your reply.

About 2). Confirmed. I've just used eng instead of "English" and spa instead of "Spanish; Castilian" within #original_language and it perfectly works.

But with this situation, it would be really nice if you could add a new possible "Languages"-type column in the "Add your own columns" section. I mean from a plain user point of view, it's quite odd storing eng, spa, or whatever other needed code.

Authors<->Books are Numbers assigned to Authors (names)
Series<->Books are Numbers assigned to Series (names)
Calibre has dozens of tables with key links

Relational DB's use keys for linkages. What you see on the screen is a very flattened view of your data and not a Flatfile like people make with Excel

[kovidgoyal]

Originally Posted by arspr

But with this situation, it would be really nice if you could add a new possible "Languages"-type column in the "Add your own columns" section. I mean from a plain user point of view, it's quite odd storing eng, spa, or whatever other needed code.

Why? As a user you will never see those codes, expect in a highly specific situation, like comparing to another column via a template. You are of course free to create whatever custom columns you like, but language in calibre has a well defined meaning, specified by ISO 639. The languages column will never accept arbitrary strings.

[arspr]

Originally Posted by kovidgoyal

Why? As a user you will never see those codes, expect in a highly specific situation, like comparing to another column via a template. You are of course free to create whatever custom columns you like, but language in calibre has a well defined meaning, specified by ISO 639. The languages column will never accept arbitrary strings.

I need this kind of feature in order to keep my current data management. I not only store the book language but also the language it was written in, its original language. So I need two language columns. In addition I also perform a comparison between them which tells me if each book has been translated or not.

Till now, Calibre didn't have a specific language type of column, so I just used two custom Tag-like columns.

Now that Calibre offers a real and "official" language column, I do want to use this feature but:

• I still need another extra language field.
• If this second column is not available as a Language-type one, I'll have to use a general Tag-like column for it. But then, my current comparison between languages stops working, so:
  • Either I just don't use this new Language feature at all...
  • Or I have to store "spa", "eng", etc. in this custom second language column instead of their "true" names, which is completely awkward .
  • Or you provide some function within the template coding language in order to make a strcmp (My second language column (remember it's Tag-Like) vs. FunctionToConvertInternalCodesToDisplayedStrings(New Official Languages Column) )
  • Or there's another way to make this comparison that I don't know about. But this one was the solution that advanced users gave me (see this old thread about it)

[kovidgoyal]

Feel free to open a bug report requesting a template function to translate from ISO codes to human readable strings.

[arspr]

Thanks for your attention.

I've posted two feature requests, in case that some coder think interesting adding just one of them. (I DO really hope some coder think these possible enhancements are worthy ):

• My desired solution: Language-like custom columns. https://bugs.launchpad.net/calibre/+bug/832074
• The other possible solution: a template function to convert the ISO codes in their GUI strings: https://bugs.launchpad.net/calibre/+bug/832084

Keep working on this great software.

[arspr]

Solved with 0.8.20

I've just tested the new language_strings(field(Languages), 0) function and it works like a charm.

I've just deleted my obsoltete #language column. Now I use the "official" Languages column, plus an extra #original_language.

[arspr]

Sorry for the triple post...

Something is not working properly or probably I'm doing something wrong...

Look at the attached screenshot (at the selected book). As a test I've set languages as Latin (lat) and Spanish (esp) because I wanted to make a stress test (ISO alphabetical sorting is different from English names).

The trouble is that the "Original Language? (#original)" doesn't work fine. It should give Yes, but it returns No. I post its modified and updated code:

{#original:'strcmp(field('#original_language'),language_strings(field('languages'),0), 'No', 'Yes', 'No')'}

I firstly thought it was because of the different sorting (look at Languages and Original Languages columns). Because of it I made an auxiliary column "aux (#aux)" with just:

{#aux:'language_strings(field('languages'),0)'}

And apparently, the sorting is automatically fixed inside the language_strings function. Apparently #aux and #original_language are the same.

So, why does strcmp evaluate them as different? What am I doing wrong?

[chaley]

The problem is that the order of the languages in the two lists are not the same. Both are in database order, which is roughly the order that they were added to the library. The language_strings function does not sort the list. You are seeing it sorted because the GUI sorts it when it is displayed.

Try

{:'strcmp(list_sort(field('#original_language'), 0,','),language_strings(list_sort(field('languages'), 0, ','),0), 'No', 'Yes', 'No')'}

[arspr]

Perfect, sorting BOTH fields solves the issue.

(Before posting, I had tried sorting but only languages field. And I needed sorting #original_language too, because the database sorting thing you've just explained).

I've used a slight variant to your code:

{#original:'strcmp(list_sort(field('#original_language'),0,','),list_sort(language_strings(field('languages'),0),0,','), 'No', 'Yes', 'No')'}

I suppose that if I sorted before calling language_strings, I would get "Spanish, Latin" because of "esp, lat", which wouldn't match the alphabetically sorted "Latin, Spanish".

[chaley]

Originally Posted by arspr

I've used a slight variant to your code:

{#original:'strcmp(list_sort(field('#original_language'),0,','),list_sort(language_strings(field('languages'),0),0,','), 'No', 'Yes', 'No')'}

I suppose that if I sorted before calling language_strings, I would get "Spanish, Latin" because of "esp, lat", which wouldn't match the alphabetically sorted "Latin, Spanish".

You are correct. The code I supplied would not work for that reason.

Why do you start with "{#original:"? That field is never used in the functions (the token $ does not appear). You could use "{:" and make it clear that there is no field being passed into the functions.

I am considering adding a "list_equal" function, where two lists are considered equal if they contain the same items in any order (case insensitive compares). That would get rid of the list_sorts, which are superfluous in this case. One question: I would assume that both lists have the same separator, making the call something like

list_equal(list1, list2, ',', yes_value, no_value).

What do you think?

[arspr]

Originally Posted by chaley

Why do you start with "{#original:"? That field is never used in the functions (the token $ does not appear). You could use "{:" and make it clear that there is no field being passed into the functions.

A very simple explanation, I didn't know I could use that syntax...

I'm quite noob using templates and in the Calibre Manual all the examples start with {whatever_field: ...}

I am considering adding a "list_equal" function, where two lists are considered equal if they contain the same items in any order (case insensitive compares). That would get rid of the list_sorts, which are superfluous in this case. One question: I would assume that both lists have the same separator, making the call something like

list_equal(list1, list2, ',', yes_value, no_value).

What do you think?

I'm no expert, but this straight-forward comparison seems less hack-like than using strcmp.

Although, in order to prevent further work I would code (if easy):

list_equal(list1, list2, ','(for list1), ','(for list2), yes_value, no_value).

In this way you would have a fully robust comparison function fully independent from the separator.

Just one doubt. How will it behave with "Greek, Modern (1453-)"?
I feel that you have one serious inconsistency here. If languages_string gives the values separated with commas, commas shouldn't be used within the names.

[chaley]

Originally Posted by arspr

Although, in order to prevent further work I would code (if easy):

list_equal(list1, list2, ','(for list1), ','(for list2), yes_value, no_value).

In this way you would have a fully robust comparison function fully independent from the separator.

That is what I was thinking. I don't want to introduce parameters that have no purpose, but your argument below solves that.

Just one doubt. How will it behave with "Greek, Modern (1453-)"?
I feel that you have one serious inconsistency here. If languages_string gives the values separated with commas, commas shouldn't be used within the names.

This is well-and-truly a problem. Not only will the proposed list_equal not work -- none of the current list functions work either.

My opinion is that the language names should not have commas in them. I would instead use semicolons. Unfortunately the language names come from elsewhere, and we have no control over them. For example, the German translation uses semicolons where all the other languages use commas!

I am not sure what to do. One possibility to add an argument (or change the current second argument) to language_strings, telling it to convert commas to semicolons. This will make the comma separator work correctly, but will have the side effect of making the results of language_string be different from that shown in the language column. Your 'system' would work only if you used semicolons instead of commas in your #original_language field. I would also need to make the language_codes function accept either style.

Another solution would be to preprocess the language code tables to convert all commas to semicolons. My guess is that Kovid would not be in favor of this approach, but I haven't asked him.

It might be that the right solution is the one that I don't want to do: a custom column that knows the contents are language codes. My problem is that creating a new column type has *many* ramifications throughout calibre, with a high probability that I will seriously break something.

More thought is required.