Identified "losing books" behavior with Save To Disk

[unboggling]

A couple months ago I mentioned I was losing some books occasionally during Save/Add operations to populate new empty libraries, with Automerge off, and Yes to add duplicates. I finally pinned down this behavior. It happens in Save To Disk from Library 1, not in Add Books to Library 2. Add Books can't find what Save To Disk didn't put there.

In two separate records, if Authors matches Authors AND Title matches Title, one of those records is dropped during the Save To Disk. This happens when the Authors and Title combinations are identical, independently of any other fields, including IDs. For example, 2 books are both Saved when they have identical ISBN13s and unique Authors or unique Titles; but 2 books are not Saved when Authors/Title combination keys aren't unique even when their ISBN13s are unique.

Copy To Library does NOT have this behavior, it Copies both records when their Authors/Title match.

This isn't necessarily a bug with Save To Disk, but it's something to keep in mind. The work-around for Save To Disk is: within any particular Authors, append something different to identical Titles to make each Title unique, such as "(from Amazon)" on one and "(from Gutenberg)" on the other. Or, use Copy To Library from Library 1 to Library 2, rather than a Save To Disk from Library 1 followed by an Add Books to Library 2.

[Starson17]

Originally Posted by unboggling

A couple months ago I mentioned I was losing some books occasionally during Save/Add operations to populate new empty libraries, with Automerge off, and Yes to add duplicates.

As an aside (that you probably already know), at about that time there was a similar issue with Automerge on when combined with Copy to Library with deletion from the source library. Changes were made to the CTL code at that time.

I finally pinned down this behavior. It happens in Save To Disk from Library 1, not in Add Books to Library 2. Add Books can't find what Save To Disk didn't put there.

In two separate records, if Authors matches Authors AND Title matches Title, one of those records is dropped during the Save To Disk. This happens when the Authors and Title combinations are identical, independently of any other fields, including IDs. For example, 2 books are both Saved when they have identical ISBN13s and unique Authors or unique Title; but 2 books are not Saved when Authors/Title combination keys aren't unique even when their ISBN13s are unique.

I was able to replicate this when the Save to Disk template includes only the {authors} and {title} fields. In that case, it's trying to create two books with identical names. I think it should probably be fixed to produce an error message, and/or offer to modify the names to make them unique. I'd suggest adding a bug report to at least make a record of the issue.

Copy To Library does NOT have this behavior, it Copies both records when their Authors/Title match.

Correct (mostly). If Automerge is off, it will see this as a Duplicate and ask if you want to add the duplicate book. If Automerge is on, it will obey the options for Automerge (one of which is to silently ignore the duplicate).

This isn't necessarily a bug with Save To Disk, but it's something to keep in mind.

It should probably at least give a warning.

The work-around for Save To Disk is: within any particular Authors, append something different to identical Titles to make each Title unique, such as "(from Amazon)" on one and "(from Gutenberg)" on the other. Or, use Copy To Library from Library 1 to Library 2, rather than a Save To Disk from Library 1 followed by an Add Books to Library 2.

Another workaround would be to make sure your save template has a unique field for the name or folder. You could add a column for version number, only populate that field when you have multiple versions, and use a save template that ignores that field unless it's populated.

[chaley]

Originally Posted by Starson17

Another workaround would be to make sure your save template has a unique field for the name or folder. You could add a column for version number, only populate that field when you have multiple versions, and use a save template that ignores that field unless it's populated.

You could also add {id} to the template. This adds calibre's internal ID to the path (a number), which is guaranteed to be unique for a given library.

[unboggling]

Originally Posted by Starson17

As an aside (that you probably already know), at about that time there was a similar issue with Automerge on when combined with Copy to Library with deletion from the source library. Changes were made to the CTL code at that time.

I didn't know that.

Originally Posted by Starson17

I was able to replicate this when the Save to Disk template includes only the {authors} and {title} fields. In that case, it's trying to create two books with identical names. I think it should probably be fixed to produce an error message, and/or offer to modify the names to make them unique. I'd suggest adding a bug report to at least make a record of the issue.

I'll do a bug report when I have time and figure out how to do one, since I've never done one before. Oops, I should have specified my Save Template in original post, but you're right, that is the sort of template I'd been using. I hadn't thought of it as the culprit, but it makes sense in hindsight now that you pointed it out. Here's the exact template I was using:

{author_sort}/{title}/{title} - {authors}

Originally Posted by Starson17

[re: Copy To Library not having this behavior] Correct (mostly). If Automerge is off, it will see this as a Duplicate and ask if you want to add the duplicate book. If Automerge is on, it will obey the options for Automerge (one of which is to silently ignore the duplicate).

Good to know, though normally I don't use Automerge at all.

Originally Posted by Starson17

[re: not necessarily a bug] It should probably at least give a warning.

I agree, since I believe that is the default Save template after install, which most users are using.

Originally Posted by Starson17

Another workaround would be to make sure your save template has a unique field for the name or folder. You could add a column for version number, only populate that field when you have multiple versions, and use a save template that ignores that field unless it's populated.

Good idea, except I prefer to put version and edition info in parentheses after title, to minimize the number of custom columns.

Originally Posted by chaley

You could also add {id} to the template. This adds calibre's internal ID to the path (a number), which is guaranteed to be unique for a given library.

Great idea, since that doesn't need an extra column. So I just did that, using double parentheses around the {id} to enable easy handling later with a new Add from filename regex if necessary. That should completely solve the problem. Anyway, my new save template is:

{author_sort}/{title}/{title} - {authors} - (({id}))

Thank you both.

[Starson17]

Originally Posted by chaley

You could also add {id} to the template. This adds calibre's internal ID to the path (a number), which is guaranteed to be unique for a given library.

This is a good idea, and a lot easier than maintaining a custom column, if he doesn't mind adding ID to all saved books. I was trying to suggest something that wouldn't change his default filename except when there was a duplicate.

[unboggling]

Originally Posted by Starson17

This is a good idea, and a lot easier than maintaining a custom column, if he doesn't mind adding ID to all saved books. I was trying to suggest something that wouldn't change his default filename except when there was a duplicate.

I'd rather have to strip the id off during a later Add Books than maintain an extra column.

btw, anybody, off topic but since I have your attention at the moment:
Which do you think is a better title between the following two, for what I still think of as "KISS posts" in my KISS thread?

Workflow with Examples for New calibre Users

or

KISS for New calibre Users

Edit: (The number of hits dropped significantly when I changed the title from KISS.)

[unboggling]

How would I change my Add Books regex to strip off the id in the title? Sorry, I don't understand regex well enough to figure it out by myself yet… Here's the new save template:

{author_sort}/{title}/{title} - {authors} - (({id}))

And this is the Add by filename regex I've been using:

(?P<author>[^_-]+) -?\s*(?P<series>[^_0-9-]*)(?P<series_index>[0-9]*)\s*-\s*(?P<title>[^_].+) ?

Edit: Never mind, I just tested and that Add by filename regex handles it automatically as is. Also, normally I Add (checkbox Checked) using Read metadata from file contents — where it doesn't matter anyway.

[Starson17]

Originally Posted by unboggling

How would I change my Add Books regex to strip off the id in the title? Sorry, I don't understand regex well enough to figure it out by myself yet… Here's the new save template:

{author_sort}/{title}/{title} - {authors} - (({id}))

And this is the Add by filename regex I've been using:

(?P<author>[^_-]+) -?\s*(?P<series>[^_0-9-]*)(?P<series_index>[0-9]*)\s*-\s*(?P<title>[^_].+) ?

Edit: Never mind, I just tested and that Add by filename regex handles it automatically as is. Also, normally I Add (checkbox Checked) using Read metadata from file contents — where it doesn't matter anyway.

Odd, the posted regex doesn't seem to work for the Save to Disk template above.

{author_sort}/{title}/{title} - {authors} - (({id}))

If it's working for you, that's fine, but if not, add this to the end :

 - \(\(\d+\)\)

to strip off " - ((id#))" at the end of your filename.

[unboggling]

Originally Posted by Starson17

Odd, the posted regex doesn't seem to work for the Save to Disk template above.

Preferences/SaveToDisk, Save metadata in OPF file, was checked. When I turned it off, you're right, that previously posted regex didn't work.

Originally Posted by Starson17

{author_sort}/{title}/{title} - {authors} - (({id}))

If it's working for you, that's fine, but if not, add this to the end :

 - \(\(\d+\)\)

to strip off " - ((id#))" at the end of your filename.

Thanks. I played with that and various permutations of Save templates and Add regexes for awhile. Recently I switched to using FN LN instead of LN, FN for Authors column, and I wanted to make all of these relatively congruent: my Save template, Add regex, FN LN convention in calibre, and usual browsing-order/sorting-convention in calibre (Authors, Series, Title).

For Save template, I went with single parentheses instead of double parentheses around the "{id}". I like that the {id} makes any saved book-record's filename unique so solves any OS filename conflict problems for identical Authors/Titles. This is what I decided to use as Save Template for now:

{author_sort}/{title}/{authors} - {series}{series_index:0>2s| | - }{title} - ({id})

For Add regex, I had been using this for awhile but it wasn't doing enough of what I wanted, and I didn't understand it well enough to mess with (though I tried…):

(?P<author>[^_-]+) -?\s*(?P<series>[^_0-9-]*)(?P<series_index>[0-9]*)\s*-\s*(?P<title>[^_].+) ?

So I also changed to a slightly simpler Add regex, that does mostly what I wanted, that I understand a little better:

(?P<author>[^_]+?) - ((?P<series>.*) (?P<series_index>[0-9]*) - )?(?P<title>.+)

I usually Save To Disk with checkboxes checked to Save cover separately, Update metadata in saved copies, and Save metadata in OPF file. I usually do Adds with checkbox checked to Read from file contents, only using that checkbox unchecked to Read from filename as a last resort when Reading from file contents didn't work well. So for my purposes, for Adding I don't usually need to use Reading by filename or include the phrase Starson suggested for the ID digits that were appended to filename in the Save template.

For Add reading from filename, trying various permutations of "( - \(\d+\))" on various Add regexs, I couldn't get any to work in all cases of has ID versus doesn't have ID, only for has but not doesn't have, and vice versa. I was probably doing something wrong in my regex noobness, but couldn't make any of these work well in enough cases (listing for illustration purposes just a few attempts with just the simpler Add regex):

(?P<author>[^_]+?) - ((?P<series>.*) (?P<series_index>[0-9]*) - )?(?P<title>.+)(| - \(\d+\)|)??

(?P<author>[^_]+?) - ((?P<series>.*) (?P<series_index>[0-9]*) - )?(?P<title>.+)(| - \(\d+\)|)?

(?P<author>[^_]+?) - ((?P<series>.*) (?P<series_index>[0-9]*) - )?(?P<title>.+)( - \(\d+\))?

(?P<author>[^_]+?) - ((?P<series>.*) (?P<series_index>[0-9]*) - )?(?P<title>.+)( - \(\d+\))

I wonder if the greedy ".+" after <title> was interfering with my trying to make optional the "\d+" phrase. If I'm interpreting it correctly, the ".+" after <title> was saying match any single character except LF or CR (that was the dot), repeat that once or more (that was the plus), and include all of that in the <title> capture (that was the entire title phrase). That seemed to be the cause of including the "\d+" element in the title capture rather than ignoring it. But removing the ".+" after <title> made things worse rather than better. Sigh. I don't want the ID included in title capture, whether it exists in the filename or not.

I'd appreciate it if anyone can explain what I was doing wrong with the "\d+" phrasing or anything else.

[unboggling]

Originally Posted by Starson17

Another workaround would be to make sure your save template has a unique field for the name or folder. You could add a column for version number, only populate that field when you have multiple versions, and use a save template that ignores that field unless it's populated.

Re-reading back through this thread, I noticed something I missed earlier in Starson's first post: the possibility of appending the {id} to a folder rather than filename. So I tested this and it works to solve filename conflicts of identical Authors/Titles in Saves, with the benefit of not needing to later strip ID out of filename with Add-using-filename regex. Because the Add regex reads from the filename itself rather than the folder it's in.

So, my newest and better Save template is now:

{author_sort}/{title} - ({id})/{authors} - {series}{series_index:0>2s| | - }{title}

Thanks, Starson. And thanks chaley for seconding the notion with the {id} idea. Sorry I missed seeing the folder option earlier.

[Starson17]

Originally Posted by unboggling

Preferences/SaveToDisk, Save metadata in OPF file, was checked. When I turned it off, you're right, that previously posted regex didn't work.

<grin> - It's always the details that trip you up.

I know you've settled on using ID in the folder, but I wondered why the regexes you posted didn't work, so I tested this one from your list:

(?P<author>[^_]+?) - ((?P<series>.*) (?P<series_index>[0-9]*) - )?(?P<title>.+)( - \(\d+\))

It parsed this filename just fine for Title, Series, Series Index and Author, and dropped the parenthetical ID:

Author is me - Series is me 6 - Title is me - (34).txt

[unboggling]

That's one of the exact same add regexs I tried. It worked for cases where filename had the appended ID, as in

unboggling iBoggle - Regex 01 - Regex Difficulties - (123)

But it didn't work in cases where filename didn't have an appended ID in that ID format. For me, yesterday. That's why I started trying pipe alternatives and optionals. So if you're getting different results I don't know what I did differently or wrong. I'll have to try it again soon, changing just one variable at a time, when I have some more time for head scratching, hair pulling, and frustration … but not today.