New to Calibre, having some difficulties

[Adoby]

Originally Posted by speakingtohe

Perhaps a stupid question but there is no chance that you are importing books of a DVD or CD and they are read only attribute?

Yes, I am beginning to suspect this also. But it seems that it was possible to move the library to another folder.

[Starson17]

Originally Posted by speakingtohe

Perhaps a stupid question but there is no chance that you are importing books of a DVD or CD and they are read only attribute?

It's not this. The problem is in the metadata.db file, not in the books being imported. The metadata.db file can't be on a DVD or CD, as it couldn't have been created.

To solve this, we need to know if what is entered in the metadata edit screens (bulk and single) is being saved.

[capidamonte]

I've just read this entire thread, and I'm wondering about the "save to disk" action you keep taking. Is it possible that you've saved into the Calibre library storage folder? My reading of your early explanations suggest this.

When I first started using Calibre I found that option tremendously confusing. It's poorly named -- it's actually an export (copy), for use with non-recognized devices or for other sharing. It seems like it should be a "save my changes" or "update my library" function, but it's not. It can be used to export your entire library filtered to, say, a single format, then reimported back into Calibre after deleting the old library, thus shrinking and simplifying it by removing other conversions or sources.

If you did save into the Calibre library folder, it would explain why you had "quadruples" of folders in the library, all with different folder names.

I think you've moved your library already, but it's also possible that there were naming collisions that corrupted your sources and that permissions were mangled -- and that some of this was carried over to the new location.

Just a thought.

At this stage, I'd do an uninstall and a fresh install with a different library location on a local disk. Then import maybe six or eight books by three or four authors to experiment with so you're not juggling too many variables.

[Merischino]

Thank you guys, all of you, for your input.

Deep Breath.

I spent the last few hours completely re-editing the metadata for my existing calibre file, with the view that clearly, what I was doing wasn't working. So, knowing that there is a tweak (that I don't have) I decided to do something pretty simple, spend the time to be thorough, and then come back here with the "fixed" database and the "fixed" i hope metadata.db file which i hope will acurrately reflect what I describe below.

It seems that there is a problem, at least in my installation of Calibre, where having your own rules for what data you enter in to the "author sort" fields and what order you put the names in the "author" field creates havoc.

It seems that I have cured at least a portion of the problem by systematically goign through me Calibre database (through the GUI!) and bulk-editing where possible and single-editing where only a single item existed for that author, such that for ALL entries, the following is true:

Field Entry
Author = Author's name in first_name last_name order
Author sort = left blank
Author sort automatically? check box = checked

Where the result is that the list in all cases does display as firstname lastname, and does sort in all cases uplon lastname

Which -- is definitely an improvement over what I had before. STill not what I want, but I can definitely work with it this way.

So.. that being the case (it LOOKS like things are working), I went ahead and.... again... updated the rest of the metadata including cover art and ISBN numbers and summaries and series index with numbers where appropriate.

So.. the metadata.db file I am about to share SHOULD display in all cases that the author name is in first_name last_name order and the list if you recreate it and sort on author SHOULD sort by last name (where "Anne McCaffrey" appears in a list not at the top but before Herman Melville and after (not immediately) Arthur C. Clarke.

To Answer the earlier question about what happens if I add a new author with bad metadata, edit that metadata, exit otu of Calibre, and re-enter calbire:

a. If I add Piers Anthony's 3-part series Battle Circle, use my old methodology of Anthony, Pier in both author fields and enter series index info for the files, what happens is the same gobbledygook I originally posted with, where the 3 different books appear in strangely different places when I sort on my whole library.

but!

b if I add Piers Anthony's Battle Circle series and use the new methodology, with firstname lastname order in the Author field and let Calibre stick whatever the hell it wants in the Author-Sort field, I get Piers Anthony in the list of authors in an "A" placement in the list in all 3 cases for that series.

and (deep breath held while doing this one)

c. if I then quit out of Calibre and restart the application, click on "author" to sort alphabetically, the order of authors goes as one would expect. meaning records go like this:

Aesop
Louisa May Alcott
Piers Anthony
Isaac Asimov
Jane Austen
Mark Bould
Dan Brown
etc etc

Where there is one and only one place in the list where a person can find books by Jane Austen. Not 2, 3, 5 or if I had 10 books by her (which would be difficult!) 10 different places in the list.

Jane Austen is now happily grouping with Jane Austen and she is not pretending to Clique with Jack London nor does she mysteriously follow Anne McCaffrey.

And Sir Arthur is also staying in a single and appropriate place.

So... I hope your research into the metadata.db below will corroborate my story.

Next is the question, what tweak do I need and where do I get it, so that I can display authors my way and not get shown the highway?

And... I'm glad to know that I shouldn't have to re-convert or save to disk in order for my metadata.db to get updated with my changes. But... I have lost my metadata changes so many times now I am afraid to finish this post without asking, how do ensure that I don't have to go do this metadata edit process over again?

Bulk changing author names is not as bad of a task as going and collecting ISBNs again, re-downloading cover art again, and ensuring again that every entry in my database has both an epub and a lit file associated with that entry. I would dearly love never to have to do this work again for these files (currently, i put off the conversion portion of this work until after we figure out if there's more wrong with my installation than just this.)

Ok here's the updated metadata.db link. And, in here the author should always be firstname lastname and the authorsort should be what Calibre makes it, meaning lastname, firstname.

The rest of the metadata for each record may or may not have been updated, but for Author name and Author Sort fields, the above statement should be true without fail.

http://www.mediafire.com/file/277k9f...17/metadata.db

[Merischino]

Originally Posted by capidamonte

I've just read this entire thread, and I'm wondering about the "save to disk" action you keep taking. Is it possible that you've saved into the Calibre library storage folder? My reading of your early explanations suggest this.

When I first started using Calibre I found that option tremendously confusing. It's poorly named -- it's actually an export (copy), for use with non-recognized devices or for other sharing. It seems like it should be a "save my changes" or "update my library" function, but it's not. It can be used to export your entire library filtered to, say, a single format, then reimported back into Calibre after deleting the old library, thus shrinking and simplifying it by removing other conversions or sources.

If you did save into the Calibre library folder, it would explain why you had "quadruples" of folders in the library, all with different folder names.

I think you've moved your library already, but it's also possible that there were naming collisions that corrupted your sources and that permissions were mangled -- and that some of this was carried over to the new location.

Just a thought.

At this stage, I'd do an uninstall and a fresh install with a different library location on a local disk. Then import maybe six or eight books by three or four authors to experiment with so you're not juggling too many variables.

I experimented with the Save to Disk only twice. Both times, I treated it like an export function, specifying a separate, external hard drive location for what I believed would be an external backup of my existing Calibre file. I did this the first time to that external drive with the simple "save to disk" menu item, and the second time I did it to the external disk using the "Sate to disk in a single directory" version because I did not like having a bajiliion unnecessary subdirectories.

I do not know what created so many separate folders in the calibre file structure for a single author beyond the initial import of books into the database. As several have posted here, when you change the authorname data in the Gui, it SHOULD change the folder structure to reflect those changes, leaving no extraneous folders in its wake. But, in practice, what I had when I began this thread was a folder for every iteration of authorname that had ever existed including odd versions which would never occur in a proper databases record (e.g., corrupted metadata on import remains in the database even after being corrected.)

Not sure how to fix that one. I am hoping that having moved my calibre database to a new location on my hard drive, the old corrupt database can now be eliminated safely and I need not (hopefully) even look into this one because it should be * clean.

...and, my "save to disk" and "convert file" behaviors I hope will only need to be continued in the rarest of cases because if I understand everyone here correctly, I should not have to do either function for my metadata.db to be correct. It should correct itself when I click "ok" after changing metadata in the metadata edit dialog window.

[Starson17]

Originally Posted by Merischino

I spent the last few hours completely re-editing the metadata for my existing calibre file, with the view that clearly, what I was doing wasn't working. So, knowing that there is a tweak (that I don't have) I decided to do something pretty simple, spend the time to be thorough, and then come back here with the "fixed" database and the "fixed" i hope metadata.db file which i hope will acurrately reflect what I describe below.

This metadata.db looks correct to me.

It seems that there is a problem, at least in my installation of Calibre, where having your own rules for what data you enter in to the "author sort" fields and what order you put the names in the "author" field creates havoc.

I don't know why you say this. You can put anything you want into either, but you have to turn off the option that tells Calibre to automatically set author_sort.

Field Entry
Author = Author's name in first_name last_name order
Author sort = left blank
Author sort automatically? check box = checked

Just so you understand - if you have the option to set Author sort automatically, that's what it will do, based on what is in author.

Where the result is that the list in all cases does display as firstname lastname, and does sort in all cases uplon lastname

As it should.

Which -- is definitely an improvement over what I had before. STill not what I want, but I can definitely work with it this way.

This is Calibre's default. It was originally designed this way, but tweaks have been added for people who want last name first.

Next is the question, what tweak do I need and where do I get it, so that I can display authors my way and not get shown the highway?

Preferences|Advanced|"Open Calibre configuration directory"|edit tweaks.py

Info on the options is in that file. Use notepad to edit it or another ascii editor.

[Starson17]

Originally Posted by Merischino

I should not have to do either function for my metadata.db to be correct. It should correct itself when I click "ok" after changing metadata in the metadata edit dialog window.

Correct

[Merischino]

Quote:
This metadata.db looks correct to me.

Yay!

Quote:
It seems that there is a problem, at least in my installation of Calibre, where having your own rules for what data you enter in to the "author sort" fields and what order you put the names in the "author" field creates havoc.
I don't know why you say this. You can put anything you want into either, but you have to turn off the option that tells Calibre to automatically set author_sort.

I say this because what I did in the previous iterations of my calibre was NOT to select "automatically sort author" and TO enter exactly what I wanted it to look like in "author" and exacltly what I wanted it to sort like in "author sort". What i got, was not anything at all like what I entered. What you saw when you looked at the db, was not what I entered -- either because I moved the library and thereby lost all my edits up until that point or for some other reason like maybe a glitch when author and authorsort fields are not flippable by the program even when you aren't asking the program to flip them. eg, a "tweak" i didn't have was not correcting for a problem that exists with the software.

What I want, and what I can't get calibre to do, is to let me identify that the author should display as authorlastname comma authorfirstname and it should sort based on authorlastname and not on authorfirtname, no matter where authorfirstname appears in the display. This problem persists whenever i edit the metadata to reflect what I want, and never do I specify my own data for authorsort at the same time telling the program to sort automatically.

Quote:
Field Entry
Author = Author's name in first_name last_name order
Author sort = left blank
Author sort automatically? check box = checked
Just so you understand - if you have the option to set Author sort automatically, that's what it will do, based on what is in author.

yes. This is the first time I have ever told the program to sort for me. I didn't want the program to sort for me, I wanted it sorted a different way than it defaults to, and therefore I never asked it to do that before. Asking it, and letting it, do that is the only way I can get my installation to sort in any way that can be characterized by the word "consistent"

Quote:
Where the result is that the list in all cases does display as firstname lastname, and does sort in all cases uplon lastname
As it should.

correct. and, I reiterate, as it never does when I tell it to sort by user-entered data. It only operates as it should when author-sort is populated by the program and not by me.

Quote:
Which -- is definitely an improvement over what I had before. STill not what I want, but I can definitely work with it this way.
This is Calibre's default. It was originally designed this way, but tweaks have been added for people who want last name first.

I need those tweaks.

Quote:
Next is the question, what tweak do I need and where do I get it, so that I can display authors my way and not get shown the highway?
Preferences|Advanced|"Open Calibre configuration directory"|edit tweaks.py

Info on the options is in that file. Use notepad to edit it or another ascii editor.

This is where I need to be led by the hand. I can get to that file, open it with text edit or what have you, but knowing what to enter to get it to behave the way I want is another question entirely. What are the magic words (or coded instructions?) to tell Calibre to let me enter my own author display and author sort information pretty please?

[jackie_w]

@Merischino... on a precautionary note...

Another caveat for you so you don't get your carefully edited Author field accidentally overwritten...

If you use the [Fetch metadata from server] button in the [Edit Metadata] window - to get book summary etc.
Make sure the box (bottom left) labelled

"Overwrite author and title with author and title of selected book"

is unchecked.

[jackie_w]

Originally Posted by Merischino

Quote:
This is where I need to be led by the hand. I can get to that file, open it with text edit or what have you, but knowing what to enter to get it to behave the way I want is another question entirely. What are the magic words (or coded instructions?) to tell Calibre to let me enter my own author display and author sort information pretty please?

The relevant section of my tweaks.py is

# The algorithm used to copy author to author_sort
# Possible values are:
#  invert: use "fn ln" -> "ln, fn" (the original algorithm)
#  copy  : copy author to author_sort without modification
#  comma : use 'copy' if there is a ',' in the name, otherwise use 'invert'
author_sort_copy_method = 'invert'

My guess is that you want yours changed to:

# The algorithm used to copy author to author_sort
# Possible values are:
#  invert: use "fn ln" -> "ln, fn" (the original algorithm)
#  copy  : copy author to author_sort without modification
#  comma : use 'copy' if there is a ',' in the name, otherwise use 'invert'
author_sort_copy_method = 'copy'

[Starson17]

Originally Posted by Merischino

I say this because what I did in the previous iterations of my calibre was NOT to select "automatically sort author" and TO enter exactly what I wanted it to look like in "author" and exacltly what I wanted it to sort like in "author sort".

That's what's confusing us both. What would you have seen if you immediately looked at the record using "E" and looked directly at the author and author_sort fields? When I do that, I see what I entered.

What I want, and what I can't get calibre to do, is to let me identify that the author should display as authorlastname comma authorfirstname

That's not the way Calibre is designed to work. It will work that way, if you force it to with a tweak, but it's not really designed to do that.

and it should sort based on authorlastname and not on authorfirtname,

It will always sort based on whatever is in author_sort. All that the tweak does is control what Calibre puts into that field when you first enter a book (or edit metadata and check the option that tells Calibre to change author_sort).

no matter where authorfirstname appears in the display.

Calibre really doesn't know which part of a name is the first name.

This is the first time I have ever told the program to sort for me.

Calibre always enters something in the author_sort field when it adds a book. It assumes you have entered the author's name "firstname lastname", and it puts "lastname, firstname" into author_sort. It never leaves it blank. The option setting only applies when changing (editing) metadata.

I didn't want the program to sort for me,

It has to sort.

I wanted it sorted a different way than it defaults to,

Then you have to manually set author_sort for each record, or you have to set the tweak so it fills it in automatically the way you want.

and therefore I never asked it to do that before.

It has to put something into author_sort. That's the way it works.

Asking it, and letting it, do that is the only way I can get my installation to sort in any way that can be characterized by the word "consistent"

It is consistent. A big part of your problem is that when you add a book, Calibre will determine the author name from either the filename or from internal metadata. That's controlled in Preference|Add/Save. If you have it set to get metadata from inside the book, and your author name is not correct, or is reversed, Calibre will fill in the author and author_sort differently.

It only operates as it should when author-sort is populated by the program and not by me.

It may seem that way, but you're wrong. Just go into any record, change the author_sort to "zzzzz" and sort the authors - that record will sort to the "z's". Calibre won't touch your author_sort after it's set the first time (unless you set the option checkbox during editing). You are free to put anything you want in there, and whatever you put in there will control the sort order.

This is where I need to be led by the hand. I can get to that file, open it with text edit or what have you, but knowing what to enter to get it to behave the way I want is another question entirely. What are the magic words (or coded instructions?) to tell Calibre to let me enter my own author display and author sort information pretty please?

You can always enter your own data in those two. Whatever you want to put in there is OK. All that the tweak does is control what Calibre does when it first adds the book or when you set the option and tell it to change author_sort based on what you've put into author. You don't need the tweak, but it makes it easier if you are going to use lastname, firstname. If you want it simple, don't use reverse name order or any tweaks - Calibre is designed to use normal name order.

[Starson17]

Originally Posted by jackie_w

My guess is that you want yours changed to:

# The algorithm used to copy author to author_sort
# Possible values are:
#  invert: use "fn ln" -> "ln, fn" (the original algorithm)
#  copy  : copy author to author_sort without modification
#  comma : use 'copy' if there is a ',' in the name, otherwise use 'invert'
author_sort_copy_method = 'copy'

Most people who like Lastname, Firstname use author_sort_copy_method = 'comma'

Using "comma", if metadata has the author as "Smith, John" then it sorts under Smith. If it has author "John Smith", it still sorts under Smith. Using "copy" it sorts under John in the latter case.

[jackie_w]

Originally Posted by Starson17

Most people who like Lastname, Firstname use author_sort_copy_method = 'comma'

Fair enough, my mistake. I don't actually use the tweak myself.
Does the comma option not disguise the fact that one's metadata may be in an inconsistent mess?

[Merischino]

Before I get into the nitty gritty, I think I should preface this by telling you that I believe you are misunderstanding me.

In all cases, I know that Calibre has a default behavior and that as a software program in the absence of information from the user there are a set of rules it follows. Such that, importing books means it populates fields with whatever metadata (however correct or not, however complete or not) it finds in the source file. and, prior to any user entry, calibre has a set of rules for how it will treat each record in its database absent correction or update from a user.

It is also my understanding that when I hit "e" and get a metadata correction form with fields which I can change or edit data, or enter data where there is none, that my user-entered changes should overrite or supercede whatever the default behavior was that Calibre used absent my corrections/changes/entries.

That said, when up til and including now, I go into Calibre and I add a book from Anne McCaffrey that has incorrect metadata (for example, the title is a string of useless information like "data saved 02-4-12 v.x200" when what should be in that field is the title of a book. My expectation is that when I enter the title of the book into that field, erasing that meaningless string, Calibre should from that moment forward display the "title" in the "title field" in all database results tables, and that it should no longer display that meaningless string.

now, speaking for the author or author-sort field. Let's say "author" was entered as the title incorrectly upon import. I would like the display in lists to show the author in a Lastname, Firstname format e.g. McCaffrey, Anne. So, in the Author (not the author sort) field, I enter exactly what I want to see in the list. Whatever calibre displayed before I made that change, whatever rules it followed, it should now as a result of my entry always list this particular record's authoer as McCaffrey, Anne.

I understand that if I do nothing to correct the author-sort field, Calibre will perform a default procedure on what is in the "author" field, taking what's after the space and putting iti n the front, adding a comma, and putting whatever was before the space at the end. This is Calibre's default. Absent information from me, this is what Calibre will do.

WITH information from me, where I leave the author-sort field blank and I check the "automatically sort" checkbox, this is also what calibre will do.

WITH information from me, where I do NOT check that box, and I DO enter specific data which is different from what calibre's default behavior is, namely if I enter "McCaffrey, Anne" into the author-sort box, what I expect is that then it will Sort this record in a list of authors such that it will appear in the Middle of the list in a "m" location while displaying the author's name as "McCaffrey, Anne".

Wat I have been explaining is that this last statement is false in all cases in my installation of calibre. I cannot make Calibre behave in this manner unless or until I install some tweak which I understand exists and which I understand is not included in default installs of Calibre.


[QUOTE]

Originally Posted by Starson17

That's what's confusing us both. What would you have seen if you immediately looked at the record using "E" and looked directly at the author and author_sort fields? When I do that, I see what I entered.

When I make metadata changes in my GUI and click "ok" and then go back into that record, I see my changes reflected in that record. When I quit out of calibre and come bak some time later, I do not find that my changes have in all cases remained. Speaking only for this afternoon, I can say unequivocally that the changes made to my calibre database TODAY but BEFORE i moved my calibre database to a new location per suggestions in this thread, when I shared the after-move metadata.db with you all of my changes were NOT reflected in the metadata.db that you looked at, whereas in the case of Miguel de Cervantes Saavedra I can confirm that the db you looked at did NOT have my changes whereas the db i Was still using DID have that change. I cannot explain this. It makes no logical sense for this to be true, but that is the case.

That's not the way Calibre is designed to work. It will work that way, if you force it to with a tweak, but it's not really designed to do that.

If this statement is true, that sorting and displaying based on a lastname-first rule for both fields, is not possible without a tweak, then I am confused why we have been goign in circles all day? It seems I need a tweak. However, elsewhere I'm told that anything I enter into those two fields should work. In practice, I believe the statement that "calibre is not designed to do that" and not the statement that "calibre will do whatever you want to do with those two fields". Please understand, I don't want to fight about it... I just want to resolve whatever I can and whatever I can't, I want to learn what calibre WILL do. not what it won't do.

It will always sort based on whatever is in author_sort. All that the tweak does is control what Calibre puts into that field when you first enter a book/

actually, I find that this last doesn't prove true in my instance. It will sort based on something that will ultimately, somehow, prove to be somehow logical, because computers operate on rules. However, when I put lastnamefirst in display and lastnamefirst in sort never checking "autosort", it seems to corrupt the data that is in "autosort" and output some kind of mess into the metadata.db file that is not what anyone would ever intend. Example: i bulk-edit all Anne McCaffrey entries such that they follow my rules. I send you my metada.db. you look at it, and tell me that you have Anne McCaffrey all over the place in everywhich way but loose in that database. You tell me that Miguel de Cervantes Saavedra in my database has an author sort which neither any human being would choose to enter, nor would Calibre's rule structure have assigned had I left it to autosort.

End conclusion? "Calibre isn't designed to do that" without a tweak.

Calibre really doesn't know which part of a name is the first name.

I would imagine that Calibre is capable only of breaking the field into "before the space" and "after the space" strings of data, or "before the comma" and "after the comma" strings of data, so that strings of data with commas in it or with more than one space in it, get broken/flipped in strange manners. E.g., files with multiple authors like a fictitions "Bould, Mark, Adams, Robert & Lackey, Mercedes" would get parsed into something like "Mark, Bould, Adams, Robert & Lackey, Mercedes"

Calibre always enters something in the author_sort field when it adds a book. It assumes you have entered the author's name firstname last, and it puts lastname, firstname into author_sort. It never leaves it blank. The option setting only applies when changing (editing) metadata.

This agrees 100% with what I entered above as my assumption. Unless I'm missing something.

It has to sort.

I must have mis-spoke. I imagine we are talking about if I do not check the "automatically sort" box that it will still sort in some fashion. This does not conflict with my view of how software, in general, works.

Then you have to manually set author_sort for each record, or you have to set the tweak so it fills it in automatically the way you want.

What I was doing that led to this thread was manually entering data for the author-sort field, expecting that it would then sort based on that information. It wasn't. It didn't. Hence, this thread.

It has to put something into author_sort. That's the way it works.

Understood.

It is consistent. A big part of your problem is that when you add a book, Calibre will determine the author name from either the filename or from internal metadata. That's controlled in Preference|Add/Save. If you have it set to get metadata from inside the book, and your author name is not correct, or is reversed, Calibre will fill in the author and author_sort differently.

Ok. Here is where I disagree, at least with your first statement. When I bulk edit 105 records for Anne McCaffrey and enter in my own data the way I want it in the Author and the Authorsort fields, not checking the autosort box, I do not ever get a consistent result. What I get is some records sorting up at the top of the list along with the "a"s. Some records sorting in the middle with the "M's and some records being willy nilly all over the place not relating to either an "a" or an "m" placement, irrespective of what is displayed in the list. I am not able to consistently get "McCaffrey, Anne" to be what is DISPLAYED in the list, nor am I able to consistently get any combination of McCaffrey, Anne to sort. Meaning, they do not ALL sort on "a". they do not ALL sort on "M". they do not ALL display as McCaffrey, Anne, and they do not ALL display as Anne McCaffrey.

Hence, this thread.

It may seem that way, but you're wrong. Just go into any record, change the author_sort to "zzzzz" and sort the authors - that record will sort to the "z's". Calibre won't touch your author_sort after it's set the first time (unless you set the option checkbox during editing). You are free to put anything you want in there, and whatever you put in there will control the sort order.

Ok. Here you are contradicting yourself again. Either it WILL behave according to what the user-entered data is, or it WON'T. I don't think I follow both statements being true. Is Calibre designed that way? Or is Calibre not designed that way? Do I need a tweak to do this? or don't I need a tweak to do this?

Again, I am not trying to be disagreeable. I am just trying to understand.

And I do appreciate you taking the time and effort to read through this thread and provide feedback.

[Merischino]

Jackie - thank you!
Starson - thank you!

I think I'm going to forgo the editing/tweaking that file until at least I better understand what is happening overall with my installation. There may as yet be something else wrong that is affecting what I see happening. It seems to be operating outside expected behavior and maybe something is corrupted, in which case I should probably leave well enough alone until either I do a clean install and make the weak and THEN add books... or maybe something else. i dont know.