Calibre 1.0 - Duplicate formats

[icallaci]

I have my Calibre prefs set to overwrite an existing format if I add the same book to my library. I noticed tonight that I have a bunch of duplicate formats. It looks like they have all appeared since I upgraded to version 1.0. The book titles, authors, etc. are all exactly the same, because I copy the .epub file to a flash drive, fix the typos and then re-add the book to Calibre by clicking on the Edit Metadata button and then clicking on the Add Books icon. I ran Library Maintenance tonight, and the duplicates showed up, but since I have my prefs set to overwrite, should this be happening? It has not ever happened before, so...

[kovidgoyal]

To be clear, you're saying that if you do the following steps:

0) copy out the .epub file from a book somewhere else
1) Click edit metadata for the book
2) Click the add book button and choose the file copied out in step 0
3) Click OK

You end up with a duplicate epub file? What exactly is duplicated? You have two epub files in the book record, with different names? Or something else?

[icallaci]

Originally Posted by kovidgoyal

To be clear, you're saying that if you do the following steps:

0) copy out the .epub file from a book somewhere else
1) Click edit metadata for the book
2) Click the add book button and choose the file copied out in step 0
3) Click OK

You end up with a duplicate epub file? What exactly is duplicated? You have two epub files in the book record, with different names? Or something else?

Yes, I end up with two epubs in the folder (not sure about any difference in the naming, sorry--I ran Library Maintenance and deleted the dups). I just tried it again without making any changes to the file and did not get a duplicate, so it appears that either the file size must be different or something else must be different in order to generate a duplicate. I'm sorry I don't have a lot of info for you at this point. I will keep an eye on it, and do some more testing to see if I can narrow down what is happening. Thanks.

Edit: Not sure if it matters, but the OS is Windows 7, 64-bit.

[icallaci]

Originally Posted by kovidgoyal

To be clear, you're saying that if you do the following steps:

0) copy out the .epub file from a book somewhere else
1) Click edit metadata for the book
2) Click the add book button and choose the file copied out in step 0
3) Click OK

You end up with a duplicate epub file? What exactly is duplicated? You have two epub files in the book record, with different names? Or something else?

Sorry, it was late last night, and I misread your post. I do not show two epub files in the book record. I show only one epub in Calibre's book record, but I have two epub files in the file system folder. Calibre's book record points to the correct file, but I am left with an additional file that is orphaned in the file system. I will try to provide additional information later today, after I do some testing.

[icallaci]

Well, of course I cannot make this happen again (after seeing approximately 30 duplicates over the course of several days). The only thing I can think of that might have some bearing on the problem, is that the duplicate files had their names truncated from 36 characters to 31 characters, as noted in this thread: https://www.mobileread.com/forums/sho...d.php?t=220936, which seems to be new behavior. I will continue to watch for this problem, and if I can make it happen again, I will write the steps down exactly to see if anyone else can replicate this problem.

[tilia]

I have noticed something similar.

When I add books to my kindle, I also make a new version of the catalog I keep on the device. With the new database, the new version of the catalog shows up as a new entry in the book list in calibre.

Both entries in the list points to the same files in the library-folder. And removing the catalogue from the device, removes the Ondevice-tick from both entries.

The duplicate entry disappears with a restart of calibre.

[kovidgoyal]

@tilia: That is indeed a bug and will be fixed in the next release, it does not however have anything to do with icallaci's problem. https://github.com/kovidgoyal/calibr...e7fa93e6c8921c

[icallaci]

OK, I think I know what is going on. The duplicate format problem is definitely related to the length of the book title and/or filename. I just added new versions of several books and only got duplicates of the ones with long file names. Books with short titles don't show the problem. Here are steps to replicate the problem:

1. Pick an epub book with a long title (not sure how long it has to be, so I suggest 36+ characters). The epub must have been added to Calibre before Calibre 1.0.

2. Copy the .epub file at the OS level to a flash drive and make whatever changes you want to it. (I use Sigil, but I have no idea if this contributes to the problem or not. I doubt it.) I also am not sure if file size has anything to do with the problem, so I suggest making enough changes to affect the size of the file.

3. Open Calibre (upgraded to 1.0), select the epub, and click on the Edit Metadata button. (Make sure your preferences for adding books are set to "overwrite existing duplicate formats.")

4. Add the new version of the edited epub by clicking on the red rectangle with the white plus sign, finding the file on the flash drive, and double-clicking it.

5. Go to the directory where Calibre stores books and find the book you just added. You should (I hope) find two epubs with similar names. Only one of these (the correct one, thank goodness) shows up in Calibre. The old version is still in the directory, orphaned.

I hope this helps. (Using Windows 7, 64-bit.)

[DoctorOhh]

Originally Posted by icallaci

5. Go to the directory where Calibre stores books and find the book you just added. You should (I hope) find two epubs with similar names. Only one of these (the correct one, thank goodness) shows up in Calibre. The old version is still in the directory, orphaned.

It looks like you have solved the mystery. Everything is working as planned except you would expect the old file to be deleted. I suspect previously this wasn't an issue since it would have over-written the old file with the exact same name.

If you run Check library under Library Maintenance does it give you the option to delete these orphaned books?

[kovidgoyal]

@icallaci: Ah, yes that makes sense. The problem is caused by a performance optimization, where when replacing a format the original is not deleted first, but just directly replaced. This causes the orphan problem when the filename for the new format is different from the filename for the old format, as in the case of the changed path length calculation. Fix will be in the next release.

You can just ignore the orphans or remove then via Library Maintenance.

[kovidgoyal]

And here is the fix: https://github.com/kovidgoyal/calibr...3d31b51196c4fb

[icallaci]

Originally Posted by kovidgoyal

And here is the fix: https://github.com/kovidgoyal/calibr...3d31b51196c4fb

Thank you so much!

[icallaci]

Originally Posted by DoctorOhh

If you run Check library under Library Maintenance does it give you the option to delete these orphaned books?

Yes, Library Maintenance finds the orphaned books and lets me delete them. I do wish there was a "Select All" for the procedure, though.