Author Sort is messing things up

[DrJanet]

Originally Posted by theducks

You mis-understood me.
Sort the Displayed list to show side by side by side.
This changes nothing except the display order on the screen.
Now you can see the similars. I fix Publisher info this way
Baen Books, Baen Pub., Baen Publishers, Baen. All get Highlighted and set to my standard: Baen Books. in a few clicks.

I'm sorry, but I don't understand what you mean by "Sort the Displayed list to show side by side by side."

[DrJanet]

Originally Posted by Starson17

You have several options:

First, there is manual changing, but that's not viable.

Second, you can rewrite the open code.

Third, you can bulk preprocess your books to rename them to the way that calibre prefers (firstname lastname). That's probably the easiest for you. I suggest Booksorter. I've also used batch files to do this job. I made a temporary copy of the books, bulk renamed and then imported them.

Fourth, you can use calibredb to output the opf file metadata, process it with SED to fix your author names, and use calibredb again to bring it back in. I did this on thousands of books, and it's slow, but it works.

BTW, you wrote"Of course I would love to be able to set - as a delimiter and direct Caliber to put the series, and series order into other fields, but that's for the future wishlist."

I'm not sure if you realize it, but it's quite easy to automatically put the series and series_index into other fields using the regex expression in Add/Save. A search on "regex" here will give you expressions to match your current filename format and bring in those fields correctly.

Good Luck.

These are excellent suggestions, thank you! I'll try Booksorter for the FName-LName issue, since I can also run Windows.

I did take a quick look at the regex link from the prefs pane the 1 day, and my eyes glazed over. However, when I searched on the term in this forum, I found a wealth of information to help me make sense of it. So, now I feel comfortable trying to parse the filenames out as you suggested.

I'll let you know how it all goes.

[alexxxm]

I just discovered this thread, since I'm having the same problems DrJanet describes.
If I may ask, could it be possible to nuke the author sort field in the bulk edit window? I could select all CTRL+A, edit bulk, and set an empty autor sort value

alessandro

[chaley]

Introduction: I prefer to use 'lastname, firstname' for my authors. I suppose that I am too visually challenged to have my eyes bounce around the page looking for a lastname, even if the list is sorted correctly. What I have been doing is setting the author field as I want. Unfortunately, this produces an author_sort value that is totally wrong, so I copy by hand the author field to the author_sort field. This process was marginally acceptable until I needed to import several hundred books with the author already set the way I want. I would have been required to change 100s of books manually. Not fun.

Fix: I fixed the problem by making a small change to calibre. This change, submitted to Kovid & co. for consideration as a supported calibre feature (don't know if it will be accepted), changes the author -> author_sort processing to do no name munging at all. The author field is copied directly to author_sort field without modification. This makes my import much easier, and in fact makes calibre noticeably easier to use (for me). Of course I still need to deal with the occasional honorific (de, de la, van, von, etc), but that is OK.

I am happy to share my change with those who want it. I will continue to maintain it until the change or something equivalent is incorporated into calibre.

Installing the change is fairly easy. It requires adding one file into the calibre program folders, and setting one environment variable. Nothing will change in calibre's behavior until both of these things are done.

To add the file:

1. Download the one file. To get it, go to http://charles.the-haleys.org/calibre.
2. Look for a folder with a name containing your version of calibre. If you don't see your version, you shouldn't try to use this feature. Currently only calibre version 0.6.37 is supported. Folder names of n.n.n--m.m.m indicate that the file in that folder is good for all versions between the numbers, inclusive.
3. Click on the folder for your version.
4. Download the file __init__.py, putting it somewhere on your computer that will easy to find. For windows, your desktop is a good choice. Alternatively, you can download it directly into the folder described below. Note: if you have Windows set up to hide extensions (the default), the file name might appear to be __init__. You won't see the ".py". This is OK.
5. On your computer, navigate to the calibre installation folder. On windows, this is probably \Program Files\Calibre2, but it could be anywhere. If you can't find it, check the properties of the shortcut you use to start calibre for where calibre.exe is placed.
6. Starting from the calibre folder found above, navigate to the subfolder "Lib\site-packages\calibre\ebooks\metadata".
7. Move (or copy) the file you downloaded (__init__.py) to this folder. Note that there will be a file already in the folder named __init__.pyo (__init__ if you hide extensions). This is OK.

Now test to be sure that calibre still runs. It might not if you put the above file in the wrong folder. Note that the new feature won't work yet.

To enable the new feature, you must set the environment variable CALIBRE_AUTHOR_SORT_METHOD=none. The page http://support.tabs3.com/main/R10463.htm describes how to set environment variables on windows. The variable name is CALIBRE_AUTHOR_SORT_METHOD and the variable value is "none" (without the quotes).

Now test the new feature. Be sure you stop and restart the calibre GUI, then pick some convenient book and edit its metadata. Put something odd into the author_sort box, then click the button that copies the author field to the author sort field. The author_sort box should now be equal to the author box. If it is not, then you have either put the __init__.py file into the wrong folder or not correctly set the environment variable.

To uninstall the feature, remove the __init__.py file you copied earlier. To turn it off temporarily, change the value of the environment variable to anything but "none".

Note that if you update calibre, you might be required to copy and install a new version of the __init__.py file. As new versions of calibre come out, I will prepare a new __init__.py (if necessary). You might want to check if I have prepared a file for the new version before you upgrade calibre.

WARNING: if you upgrade calibre but do not either remove or upgrade the __init__.py file, calibre may stop working or not work as expected.

Best of luck,
Charles

[itimpi]

I would have though a more useful version of this enhancement would be to examine the author field for a comma, and if there is one present copy it across unchanged to the author-sort field when requested to set the author-sort field automatically. If there is no comma behave as now.

That would mean for the vast majority of users who have author as firstname lastname there would be no change, but those who store author as lastname, firstname would get the author sort field set to the same value. Also there would be no need for the environment variable you mentioned.

Also I would ahve thought that Kovid would then be happier with this enhancement as it is all automated and not subject to user error.

[Starson17]

Originally Posted by itimpi

I would have though a more useful version of this enhancement would be to examine the author field for a comma, and if there is one present copy it across unchanged to the author-sort field when requested to set the author-sort field automatically. If there is no comma behave as now.

I thought about the comma issue, when I proposed some code (included as of 0.6.35) that splits lastname, firstname middlename format titles at the comma instead of the space between firstname and middlename . I have found a few author names that have commas preceding Jr., Sr. or the IIIrd, so I'd hesitate to use the presence of a comma as a guaranteed marker for the desire to copy author unchanged into author_sort. I'd think an option box would be necessary to avoid changing current behavior too much.

Although I've used lastname, firstname format in my directories for 20+ years, I'm now comfortable with the calibre default. However, when I first started using it, I did consider trying to force it to do it "my" way. AFAICT, a global copy of the name field of the authors table into the author_sort field of the books table in the metadata.db SQL file should not cause any problems. You can't do that sort of change with author name or book title without screwing up the directory structure and database, but I think it's OK to change author_sort.

It wouldn't be too hard to provide that kind of global update function, regardless of how you deal with each individual entry.

[chaley]

@itimpi: I thought of that, but I decided against it, both for the reasons that Starson17 gave and because it would be more 'magic' behavior that would get me when I least expect it. As for the environment variable, I don't generally use them in this way (as options), but I didn't find a different straightforward way of globally affecting both the calibre CLI programs and the GUI.

@Starson17: I first tried to do this in SQL directly against the database, but failed because some of the triggers refer to python functions. In addition, that process would break any 'non-standard' author_sort settings one has made. The way I did it permits one to select the books to change, open the bulk metadata editor, check the "Automatically set author sort" while leaving the author field empty, then 'run'. To run it globally, select everything. In fact I did just that after I found that slight variations in the author sort showed up as variations in the new catalog e-book.

[Starson17]

Originally Posted by chaley

@Starson17: I first tried to do this in SQL directly against the database, but failed because some of the triggers refer to python functions.

Yes, I've run into the triggers. too. I was thinking it would be done through Python/Calibre, not directly.

In addition, that process would break any 'non-standard' author_sort settings one has made.

As a matter of interest, what 'non-standard' author_sort settings do you use?

The way I did it permits one to select the books to change, open the bulk metadata editor, check the "Automatically set author sort" while leaving the author field empty, then 'run'.

It just seems to me that using an environment variable for this feature is different from the way most of the other optional user functionality is implemented.

[chaley]

Originally Posted by Starson17

As a matter of interest, what 'non-standard' author_sort settings do you use?

Almost all of them are related to 'noble' names containing what the French call 'particules'. Names like this are supposed to sort in the order of the word following the particules, but generally they appear with the particules where they belong. Some examples:

• "Joe de Blogs" appears as "de Blogs, Joe", but sorts as "Blogs de, Joe" (or as "Blogs, Joe de", preferred by some)
• "A van L" appears as "van L, A", sorted as "Lam van, A"
• Names like "Baron George von Q" add complexity. I usually make these "von Q (Baron), George" and sort it as "Q von Baron, G", to distinguish the name from "George Baron von Q". Fortunately I don't currently have any books that exhibit this complexity.

However, there are complications (as usual). For example, the name
"Charles De Gaulle" appears as "De Gaulle, Charles", and *sorts* as "De Gaulle, Charles", because in this case the De is not a particule. In addition, some people prefer one form or the other. I have a friend with a name of the form "X de Y" who insists that the name sort as "de Y" because (he says) they haven't had a noble patent for hundreds of years.

[edit] Some Asian names also present complications. For example, when I lived in Malaysia, my Chinese-ancestry students had names like "Lim Cheong", where Lim is the surname. However, the Malay and Indian-ancestry students had names in the other order (fn ln). Some (mostly Malays) had only one name sometimes adorned with parentage, other Malays have the equivalent of particules, and some had both. I ended up entering Chinese-ancestry names as they appeared without any commas, but entered the other names with the order reversed (with commas). Thus I would see a list looking like
"Geetha" (a single name)
"Gupta, Deepak" (a 'normal' name)
"Nai binti Ab (Nik)" (comes from "Nik Nai binti Ab", or "Dame Nai daughter of Ab".)
"Lim Cheong" (a Chinese name)
[end edit]


For a while I ignored the problem. However, people I have cited in my papers have contacted me to tell me that their name is sorted into the wrong place in the references section. I now try to try to get it right to avoid possible offense, and this has carried into how I maintain my library.

A rule that recognizes de, du, de la, van, von, di, and a host of others as special would catch most of these cases, but not all. even with the rules, a global replace would (and in my case did) break something.

Originally Posted by Starson17

It just seems to me that using an environment variable for this feature is different from the way most of the other optional user functionality is implemented.

No argument from me.

[itimpi]

Originally Posted by chaley

@itimpi: I thought of that, but I decided against it, both for the reasons that Starson17 gave and because it would be more 'magic' behavior that would get me when I least expect it. As for the environment variable, I don't generally use them in this way (as options), but I didn't find a different straightforward way of globally affecting both the calibre CLI programs and the GUI.

I might raise this as a change request then! If this was the documented behavior in the User Guide, then I do not see why it should be that any cases are "magic".

I am quite happy to live with the few exceptions where there is a comma in the name. As far as I am concerned this is really an error in the metadata so in these cases I will simply correct the metadata. In these cases if I really want something unusual I still have the manual entry option.

[Starson17]

Originally Posted by chaley

Almost all of them are related to 'noble' names containing what the French call 'particules'.

Thanks for the detailed response. If you have time, one more question. I saw your bug track conversation with Kovid where he suggested that you revise the code to display the author in your preferred format, while keeping the calibre default for author. Why was that not a good approach?

You started your post stating that you didn't want to have your eyes "bounce around the page looking for a lastname, even if the list is sorted correctly." Given that you can use the author_sort field for output in Save to Disk and Send to Device, what's wrong with merely displaying it in your preferred format, while keeping both the author and author_sort fields as Kovid designed them? Your method seems to make the author field essentially the same as the author_sort field, aside from the particules issue and that could probably be addressed the same way that leading articles like 'A' and 'The' are handled when sorting titles.

What I'm asking is - What use (other than the GUI display) do you make of your reverse stored author field that can't be handled by using the author_sort field?

[chaley]

Originally Posted by Starson17

What I'm asking is - What use (other than the GUI display) do you make of your reverse stored author field that can't be handled by using the author_sort field?

The problem is that the two fields do not the same meaning. One contains the string to be displayed, and the other contains the order specification. This is, of course, the philosophical position that calibre currently takes, and it is the right one. The fact that I want the initial default of author_sort to equal author does not change the underlying semantics of the fields. Just like everyone, I want to be able to change the order without changing the content. It is only an accident that the two fields contain the same data. I confess that in my case it is a frequent accident, but it is an accident all the same.

To more specifically answer your question, I use author and not author_sort when writing to my reading devices and when generating reports with calibredb. I could use author_sort, but the output would be wrong when author_sort != author. In the new catalog feature, author_sort determines order (as it should) , the author field is displayed in the new catalog feature (as it should), and there is no option to display author_sort (which would be incorrect for the reasons given above). Book metadata would contain the incorrect string as author. Author, not author_sort, is used in the calibre library folder structure (as it should). Bulk import produces the 'wrong' answer. And so on.

Summary: I want the two fields to continue to have the meaning they have today: author is what I see and author_sort is how it is ordered. However, I want to use a different method when calibre is copying author to author_sort, should such a copy be necessary or requested. From a philosophical point of view, I don't see how this position can be controversial.

Asking someone who does not care about the different method to take time to develop and to support it is arguably controversial, and I recognize the full validity of that point of view.

[chaley]

Originally Posted by itimpi

I might raise this as a change request then! If this was the documented behavior in the User Guide, then I do not see why it should be that any cases are "magic".

I added your request to my patch, but still using the evil environment variable.

If you set CALIBRE_AUTHOR_SORT_METHOD=none-if-comma, it has the behavior your described. If there is a comma in the author string, author is copied unchanged to author_sort, otherwise the author string is processed as before to convert 'FN LN' to 'LN, FN'.

The __init__.py in the download folder referred to above contains the change.

[Starson17]

Originally Posted by chaley

I added your request to my patch, but still using the evil environment variable.

If you don't want to use an environment variable, it's pretty easy to add a new config checkbox to the GUI, and store the config option. I did that with the new overwrite_author_title_metadata option seen in the general config screen and on the single book fetch metadata screen.
I don't see any reason why you can't add another option, settable by GUI, that applies to both GUI and commandline.

calibre\gui2\dialogs\config\config.ui creates the new checkbox widget in the main config screen.

calibre\gui2\__init__.py adds the new configuration flag and sets its true/false default.

You can see an example of checking the status of the saved configuration flag in calibre\gui2\metadata.py to control program flow.

[chaley]

Starson17, thank you. Having relevant pointers into a code base as large as calibre's is extremely helpful. I will look it over this weekend.