New to Calibre, having some difficulties

[Starson17]

Originally Posted by Merischino

That said, when up til and including now, I go into Calibre and I add a book from Anne McCaffrey that has incorrect metadata (for example, the title is a string of useless information like "data saved 02-4-12 v.x200" when what should be in that field is the title of a book. My expectation is that when I enter the title of the book into that field, erasing that meaningless string, Calibre should from that moment forward display the "title" in the "title field" in all database results tables, and that it should no longer display that meaningless string.

Agreed. That's what it does. The only proviso is that the meaningless data in the book is still there. Calibre doesn't overwrite it. It only sets the data in its own database.

now, speaking for the author or author-sort field. Let's say "author" was entered as the title incorrectly upon import. I would like the display in lists to show the author in a Lastname, Firstname format e.g. McCaffrey, Anne. So, in the Author (not the author sort) field, I enter exactly what I want to see in the list. Whatever calibre displayed before I made that change, whatever rules it followed, it should now as a result of my entry always list this particular record's authoer as McCaffrey, Anne.

Correct

I understand that if I do nothing to correct the author-sort field, Calibre will perform a default procedure on what is in the "author" field, taking what's after the space and putting iti n the front, adding a comma, and putting whatever was before the space at the end.

No. Calibre already entered data in author_sort. Now you're editing metadata, and it won't touch author_sort unless you tell it to by checking the update author_sort box. If you check that box, Calibre does what you described (unless a tweak is set)

WITH information from me, where I do NOT check that box, and I DO enter specific data which is different from what calibre's default behavior is, namely if I enter "McCaffrey, Anne" into the author-sort box, what I expect is that then it will Sort this record in a list of authors such that it will appear in the Middle of the list in a "m" location while displaying the author's name as "McCaffrey, Anne".

Agreed. That's what mine does. I recognize you are saying yours does not. That's why this thread is very long. You are seeing something that I don't see. I just tested it.

Wat I have been explaining is that this last statement is false in all cases in my installation of calibre. I cannot make Calibre behave in this manner unless or until I install some tweak which I understand exists and which I understand is not included in default installs of Calibre.

The tweak does not affect this. Setting the tweak is not going to make it behave differently on this issue. All it will do is control how the author is field is copied into the author_sort field. That happens when the book is first added, and when you change author, and ALSO SET THE OPTION telling Calibre to reset author_sort according to what's now in author.

When I make metadata changes in my GUI and click "ok" and then go back into that record, I see my changes reflected in that record. When I quit out of calibre and come bak some time later, I do not find that my changes have in all cases remained. Speaking only for this afternoon, I can say unequivocally that the changes made to my calibre database TODAY but BEFORE i moved my calibre database to a new location per suggestions in this thread, when I shared the after-move metadata.db with you all of my changes were NOT reflected in the metadata.db that you looked at, whereas in the case of Miguel de Cervantes Saavedra I can confirm that the db you looked at did NOT have my changes whereas the db i Was still using DID have that change. I cannot explain this. It makes no logical sense for this to be true, but that is the case.

I don't disbelieve you, I just have no explanation.

If this statement is true, that sorting and displaying based on a lastname-first rule for both fields, is not possible without a tweak

It's not true. Simply set the author sort to whatever you want and set the author to whatever you want. If you want lastname, firstname in both, just set them that way and don't turn on the option that tells Calibre to change author_sort. Yo will have to do this for every book you add, however.

, then I am confused why we have been goign in circles all day? It seems I need a tweak.

It makes it easier for you, as there are fewer edits needed of author_sort. It won't solve any other problems you've described.

However, elsewhere I'm told that anything I enter into those two fields should work.

It will.

In practice, I believe the statement that "calibre is not designed to do that" and not the statement that "calibre will do whatever you want to do with those two fields".

Try to understand. All that the tweak does is control what gets copied from author into author_sort. That only happens once when the book is first added, and any time you set the option during a metadata edit. That's all that it does. Calibre doesn't know which part of John Smith is the first name. It doesn't care if you put Smith, John into the author.

Please understand, I don't want to fight about it... I just want to resolve whatever I can and whatever I can't, I want to learn what calibre WILL do. not what it won't do.

I wouldn't have spent this much time with you if I didn't want you to understand what it will/won't do.

I find that this last doesn't prove true in my instance. It will sort based on something that will ultimately, somehow, prove to be somehow logical, because computers operate on rules. However, when I put lastnamefirst in display and lastnamefirst in sort never checking "autosort", it seems to corrupt the data that is in "autosort" and output some kind of mess into the metadata.db file that is not what anyone would ever intend.

There is no "autosort." That option merely controls whether Calibre will change author_sort (based on author) or not when you are editing metadata.

Example: i bulk-edit all Anne McCaffrey entries such that they follow my rules. I send you my metada.db. you look at it, and tell me that you have Anne McCaffrey all over the place in everywhich way but loose in that database. You tell me that Miguel de Cervantes Saavedra in my database has an author sort which neither any human being would choose to enter, nor would Calibre's rule structure have assigned had I left it to autosort.

I did test your metadata.db with several bulk edits. They all worked as they should.

End conclusion? "Calibre isn't designed to do that" without a tweak.

Sorry, but the tweak does not affect this. It's just there so when you add a book, and Calibre first fills in the author_sort field, it will be closer to what you put into the author field - IOW, it won't take the last word in the author field and put it at the front, followed by a comma. If the tweak is set to "copy" it will put what is in the author field into author_sort, without change. If it's set to "comma" it will put what is in the author field into author_sort, without change if the name has a comma, and otherwise will do what Calibre normally does. That's it. Nothing magic. You can manually do what the tweak does, or override it and put something else in there.

Ok. Here is where I disagree, at least with your first statement. When I bulk edit 105 records for Anne McCaffrey and enter in my own data the way I want it in the Author and the Authorsort fields, not checking the autosort box, I do not ever get a consistent result.

I can't explain this. I tried it in mine - it works. I tried it in your file - it works.

What I get is some records sorting up at the top of the list along with the "a"s. Some records sorting in the middle with the "M's and some records being willy nilly all over the place not relating to either an "a" or an "m" placement, irrespective of what is displayed in the list. I am not able to consistently get "McCaffrey, Anne" to be what is DISPLAYED in the list, nor am I able to consistently get any combination of McCaffrey, Anne to sort. Meaning, they do not ALL sort on "a". they do not ALL sort on "M". they do not ALL display as McCaffrey, Anne, and they do not ALL display as Anne McCaffrey.

Again, I can't explain it.

Ok. Here you are contradicting yourself again. Either it WILL behave according to what the user-entered data is, or it WON'T. I don't think I follow both statements being true. Is Calibre designed that way? Or is Calibre not designed that way? Do I need a tweak to do this? or don't I need a tweak to do this?

I don't know what you think is a contradiction. It sorts by what you put in the author_sort field. I almost never put anything in that field. I enter the author and set the checkbox, letting Calibre fill in author_sort for me.

[Merischino]

Ok. I believe we now understand eachother. And I believe we are still at the impasse of "what should happen" vs "what is happening" being different on my machine.

I understand that whatever calibre does with respect to authorsort fields, it does once. And that after I manually enter a change (as i have done for every single record in my database), it should behave a certain way. It isn't. I don't have an explanation for why. And I understand your poitns about how the tweak is basically just a setting for what calibre defaults to that one time that it puts something into the authorsort field (upon adding a file to the database). That tweak has no real relevance for me because I don't nmind manually entering what I want, and have already done so. Adding the tweak now won't have any affect whatsoever on what is or is not currently working in the records which are already in my database.

What steps should I take in order to achieve a hopefully clean and uncorrupted installation of calibre? The fact that I just upgraded to the latest version probably is immaterial at this point, as the problems I was having before that upgrade I am still having now.

[Merischino]

...and... what is the optimal way for me to "backup" the currently working (although not in the manner i ultimately desire) database?

[DoctorOhh]

Originally Posted by Merischino

So, your statement:

Is actually false. Hate to break it to you.

I'm glad you're starting to see the light. But understand that the folder/file structure is created directly from the information you enter in the gui/metadata and if you change the information the folder/file name will change on the fly.

Understand also that the sort will take into account spaces you may have entered either before or after the names. So if you are manually entering data in the author_sort area the sort is based on exactly what you entered.

[Adoby]

Originally Posted by Merischino

...and... what is the optimal way for me to "backup" the currently working (although not in the manner i ultimately desire) database?

Exit Calibre.

Copy the WHOLE Calibre Library folder to backup media.

This includes all the folders with books and other files and the metadata.db file.

When you restore, first delete (or rename) the Calibre Library in use, before you copy back the WHOLE backup Calibre Library from backup media.

Treat the whole Calibre Library as a single item, don't backup or restore parts of it.

[capidamonte]

Is your Calibre library on a network or USB drive?

Is whatever drive you're using old enough to be suspect?

Are you running Windows? It's sounding more and more like a system problem, like you're getting bad writes to the drive.

[Adoby]

Originally Posted by capidamonte

It's sounding more and more like a system problem, like you're getting bad writes to the drive.

I doubt that. I am afraid that I have started to suspect repeated operator errors. It is the error source that is the hardest to protect against ...

[Starson17]

Originally Posted by Merischino

I believe we are still at the impasse of "what should happen" vs "what is happening" being different on my machine.

If something is still happening that shouldn't happen, it's either a bug or a hardware problem or an OS permissions issue, etc. It can be found if it can be repeated. Do you have a problem that can be repeated?

I understand that whatever calibre does with respect to authorsort fields, it does once. And that after I manually enter a change (as i have done for every single record in my database), it should behave a certain way. It isn't. I don't have an explanation for why. And I understand your poitns about how the tweak is basically just a setting for what calibre defaults to that one time that it puts something into the authorsort field (upon adding a file to the database).

You're basically right, but the tweak does affect what happens when you edit metadata if you have the option checked that tells Calibre to change author_sort when you change author. It's not totally a "one-time only" tweak.

Adding the tweak now won't have any affect whatsoever on what is or is not currently working in the records which are already in my database.

Correct.

What steps should I take in order to achieve a hopefully clean and uncorrupted installation of calibre? The fact that I just upgraded to the latest version probably is immaterial at this point, as the problems I was having before that upgrade I am still having now.

Can you describe the problems and repeat them, or are they intermittent?

[Merischino]

Originally Posted by dwanthny

I'm glad you're starting to see the light. But understand that the folder/file structure is created directly from the information you enter in the gui/metadata and if you change the information the folder/file name will change on the fly.

Understand also that the sort will take into account spaces you may have entered either before or after the names. So if you are manually entering data in the author_sort area the sort is based on exactly what you entered.

If I understand you correctly, then the fact that I have manually corrected the metadata for every single instance of Anne McCaffrey in my calibre database (through the gui) such that they are standardized aka the same in every instance (using a bulk convert), there should only be one folder for Anne McCaffrey in my calibre database?

[Merischino]

Originally Posted by capidamonte

Is your Calibre library on a network or USB drive?

Is whatever drive you're using old enough to be suspect?

Are you running Windows? It's sounding more and more like a system problem, like you're getting bad writes to the drive.

My computer is a late 2009 model Apple Mac Mini running Apple OS 10.6.4 (Snow Leopard).

I use calibre with the default setup, where the calibre library/database file structure occurs in my user folder on my hard drive. aka /user/calibre_library would be the appropriate path.

When I export/"Save to Disk", I do it for the purpose of creating a backup elsewhere in my system, which happens to be to an external terabyte hard drive made by LaCie, manufactured I don't know when but purchased in March of this year. I have yet to do anything with those exports/backups/saves, so do not know if anything gets corrupted or not when it is saved to external storage.

All that said, I agree that it sounds like something is getting corrupted in some manner. But that's a guess.

[Starson17]

Originally Posted by Merischino

If I understand you correctly, then the fact that I have manually corrected the metadata for every single instance of Anne McCaffrey in my calibre database (through the gui) such that they are standardized aka the same in every instance (using a bulk convert), there should only be one folder for Anne McCaffrey in my calibre database?

Correct. I often look through my Calibre Library (I know I'm not supposed to peek "under her skirts", but I do) to see if I have any author names that are "similar" but not correct, like an author with a middle name and the same author with only a middle initial. I then fix it in the GUI and watch the two merge together, leaving only a single folder for that author.

BTW, ignore any posts here that imply you made some kind of mistake. It really doesn't make any difference one way or the other. All that matters is whether you understand now how Calibre is supposed to work and whether it's actually working that way now.

[Merischino]

[QUOTE=Starson17;1030828]If something is still happening that shouldn't happen, it's either a bug or a hardware problem or an OS permissions issue, etc. It can be found if it can be repeated. Do you have a problem that can be repeated?

yes. I can bulk-edit the metadata for author and author-sort fields, and in the instances where I put the same exact entry into both of those fields, not checking the box for calibre to do anything related to the authorsort fields, the result is that I get multiple sort methods and multiple display methods which do not reflect what has been entered, and can still be viewed as having been entered, into the author and authorsort fields. e.g., Anne McCaffrey entered into both fields always as McCaffrey, Anne, will sort on the a sometimes, on the m sometimes, and on God only knows what sometimes. This problem is repeatable and consistent. The only way I have been able to make anything sort in a "Correct" and "Consistent" manner,s uch that authors sort in a uniform manner, is to ensure that when actually entering the metadata I do ti in a way that calibre likes: author name is firstname space lastname, authorsort is left blank, and the checkbox is checked so that calibre populates the authorsort field in a lastname comma firstname manner.

Can you describe the problems and repeat them, or are they intermittent?

the problems as described throughout this thread, and above, are not intermittent, they are consistent. They are repeatable.

[Merischino]

Originally Posted by Starson17

Correct. I often look through my Calibre Library (I know I'm not supposed to peek "under her skirts", but I do) to see if I have any author names that are "similar" but not correct, like an author with a middle name and the same author with only a middle initial. I then fix it in the GUI and watch the two merge together, leaving only a single folder for that author.

BTW, ignore any posts here that imply you made some kind of mistake. It really doesn't make any difference one way or the other. All that matters is whether you understand now how Calibre is supposed to work and whether it's actually working that way now.

Thanks. And, this is why I was so confused when I posted what did about what I see when I peek: I get multiple folders for authors in in my calibre database, with multiple books stored inside those folders, despite the fact that my calibre library had been changed/updated/standardized using the GUI such that there should only have been one folder which held two copies (one epub, one lit) of each work within that single author folder.

When I make changes using the GUI, what is reflected inside my calibre file structure is not consistent with the changes I have made. And, even if the changes to the metadata within the GUI can still, after closign out that record, be seen by opening up that record within the GUI again, it does not necessarily reflect in how things sort.

To reiterate: something is amiss in denmark?

So I'm guessing what I need to do next is backup my calibre library as it stands now, then completely delete everything calibre but the backup. and then perform a clean install. and then delete the newly created calibre file structure and replace it with the backedup one. And hope for the best?

[Starson17]

Originally Posted by Merischino

this is why I was so confused when I posted what did about what I see when I peek: I get multiple folders for authors in in my calibre database, with multiple books stored inside those folders, despite the fact that my calibre library had been changed/updated/standardized using the GUI such that there should only have been one folder which held two copies (one epub, one lit) of each work within that single author folder.

This can be cleaned up. Obviously, you need to find out if it's working incorrectly. Some of your posts indicate to me that you may have a permissions issue, but other things can be going on.

For example, if you have a folder open in File Explorer, Calibre can't remove it when it needs to. It leaves orphan folders. They don't hurt anything, but are confusing. Permissions issue can cause these orphan folders, too.

When I make changes using the GUI, what is reflected inside my calibre file structure is not consistent with the changes I have made.

In what way?

And, even if the changes to the metadata within the GUI can still, after closign out that record, be seen by opening up that record within the GUI again, it does not necessarily reflect in how things sort.

More info?

To reiterate: something is amiss in denmark?

It sounds like it, but I'm still unable to finger your problem.

So I'm guessing what I need to do next is backup my calibre library as it stands now, then completely delete everything calibre but the backup. and then perform a clean install. and then delete the newly created calibre file structure and replace it with the backedup one. And hope for the best?

If you want, to, but personally I don't see that as being the answer. I'd track down what's actually happening first.

Some questions I'd want answered first:

Are my books all accessible in the GUI (confirms that at worst, we have orphan folders problem)

If I change an author name, does the folder in the Library disappear under the old author name and reappear under the new name? (confirms Calibre can delete an old folder and create a new one)

Are my bulk edits working correctly to change author - check each record individually with an individual metadata edit.

Sorting - also check each record's metadata individually (select and press "E" in GUI) and look at author_sort. Are they all correct? Look for spaces, extra commas at the end, odd non-ascii chars, etc.

If you can see all your books, then what are these extra odd(possible orphan) folders? Note that each folder under an author has a record id at the end of the folder name. That id is also in the displayed info in GUI when that record is selected and you can see the comments. Track down if that id exists in the database or is it an orphan by checking the author or title and searching for it, then look at the id and cross-match.

Something will appear wonkified.

[itimpi]

Originally Posted by Starson17

Note that each folder under an author has a record id at the end of the folder name. That id is also in the displayed info in GUI when that record is selected and you can see the comments.

I know that USED to be the case, but I am no longer seeing it since the latest change to the GUI design. I think that is because at one time the path was displayed, while now you get the "Click to Open" with the id displayed as part of the Tooltip. This makes the information a bit less obvious (which may of course be a good thing).