Duplicates found!

Moejoe · 02-22-2009, 04:08 PM

I'm encountering a problem and hope it's just my own stupidity that's causing this.

Every time I try to add a new book (tried with .lit, .epub and others) I receive this warning.

The books are fresh and have never been added to the library. If I choose OK 2 entries are created. One contains the original ebook, the other a ZIP file. This only started happening today, and I already have some 100+ books in the library that imported without any such errors.

I'm running Mac OS X 10.5.6 and Calibre 0.4.139.

Thanks in advance for any advice.

pilotbob · 02-22-2009, 04:11 PM

Do you by any chance have the book and a zip containing the book in the same folder and are selecting the "folder" that those books are in? Or perhaps you have two formats of the same book in the folder?

I think the default add books will add all books in the folder if you use the default add books button.

How "exactly" are you adding the book?

BOb

Moejoe · 02-22-2009, 04:18 PM

Quote:

Originally Posted by pilotbob

Do you by any chance have the book and a zip containing the book in the same folder and are selecting the "folder" that those books are in? Or perhaps you have two formats of the same book in the folder?

I think the default add books will add all books in the folder if you use the default add books button.

How "exactly" are you adding the book?

BOb

Adding the books by clicking on the Add Books icon (not drilling down to the other options), but it seems like the problem has ceased after I used an old Windows trick -- I restarted

Everything I've imported after the restart works fine, just as it did earlier.

I have no idea why this odd quirk occurred, but I'll keep an eye open to see if it comes back.

idragon · 02-23-2009, 09:25 AM

Welcome to the club ...

There are a few of us having this problem,
https://www.mobileread.com/forums/showthread.php?t=39427

I believe Kovid is looking into this, but I suspect its a fairly obscure bug. I had a quick look at some of the source code, but havent been able to spot anything out of the ordinary.

ilovejedd · 02-23-2009, 11:46 AM

A bit OT, but close...

How does Calibre check for duplicates when adding new books? By filename? By title? I've added some books that aren't duplicates (same title but different authors) and Calibre seems to be recognizing them as duplicates.

Thanks in advance!

kovidgoyal · 02-23-2009, 12:58 PM

I've found the bug, will be fixed in the next release. calibre uses the title to check for duplicates. It does not compare authors, since the case of books having the same title by different authors is rare and you can simply manually add them in that case.

idragon · 02-23-2009, 04:18 PM

I dont think this specific error/bug is related to the check for duplicates with the Title.

I have definitely added books that didnt exist before (different titles) and it still popped up with the error.
There is also the problem that it makes copies of previous books and adds them under the "new" book (If you click on Ok to add duplicates)
The OP mentiones that, and in the other thread the OP also had that problem, and I can attest to that exact same behaviour.
I just noticed this user in Ticket #1853 got the error with a new clean database after the 2nd add.

Its almost like an array/variable somewhere doesnt get cleaned up propely (with links to previous files that were added) , and then when you click on ok to add the Duplicate. It copies that "other" file under the new book as a book format. Hope that makes sense

Mitchll · 02-23-2009, 04:21 PM

Quote:

Originally Posted by idragon

I dont think this specific error/bug is related to the check for duplicates with the Title.

I have definitely added books that didnt exist before (different titles) and it still popped up with the error.
There is also the problem that it makes copies of previous books and adds them under the "new" book (If you click on Ok to add duplicates)
The OP mentiones that, and in the other thread the OP also had that problem, and I can attest to that exact same behaviour.
I just noticed this user in Ticket #1853 got the error with a new clean database after the 2nd add.

Its almost like an array/variable somewhere doesnt get cleaned up propely (with links to previous files that were added) , and then when you click on ok to add the Duplicate. It copies that "other" file under the new book as a book format. Hope that makes sense

I have to agree. I had the error message when I added new title/book
mitch

kovidgoyal · 02-23-2009, 05:01 PM

It was a threading related bug caused by different threading behavior on the OS X platform. Basically, each time you add books a new thread ic created to carry out the work without frezing the GUI. On OSX the old threads were not being deleted so everytime you add anew book all the previous add threads would try to re-add their books as well, causing the duplicates.

ilovejedd · 02-23-2009, 05:15 PM

Quote:

Originally Posted by kovidgoyal

calibre uses the title to check for duplicates. It does not compare authors, since the case of books having the same title by different authors is rare and you can simply manually add them in that case.

Would it be possible to request this functionality? How hard would it be to implement? I read a lot of fanfiction and a number of those tend to have similar titles. It's pretty amazing how many fanfics are titled 'Untitled'.

idragon · 02-23-2009, 05:22 PM

Quote:

Originally Posted by kovidgoyal

It was a threading related bug caused by different threading behavior on the OS X platform. Basically, each time you add books a new thread ic created to carry out the work without frezing the GUI. On OSX the old threads were not being deleted so everytime you add anew book all the previous add threads would try to re-add their books as well, causing the duplicates.

Yup that would explain it. Thanks Kovid

gttexas · 02-23-2009, 05:26 PM

I've also puzzled over the problem until I noticed that the "duplicate" book was titled "Zane Grey". It seems I had a number of books to be imported which had the title and Author fields reversed!

kovidgoyal · 02-23-2009, 06:58 PM

Quote:

Originally Posted by ilovejedd

Would it be possible to request this functionality? How hard would it be to implement? I read a lot of fanfiction and a number of those tend to have similar titles. It's pretty amazing how many fanfics are titled 'Untitled'.

It's not too hard to implement but I'm not sure it should be, since there are a lot of cases where book metadata is incomplete so although the authors dont match, the books are actually the same.

Moejoe · 02-23-2009, 07:41 PM

Quote:

Originally Posted by idragon

Welcome to the club ...

There are a few of us having this problem,
https://www.mobileread.com/forums/showthread.php?t=39427

I believe Kovid is looking into this, but I suspect its a fairly obscure bug. I had a quick look at some of the source code, but havent been able to spot anything out of the ordinary.

I should have looked more closely before posting this thread. Sorry if it's incovenienced anyone.

ilovejedd · 02-23-2009, 08:53 PM

Quote:

Originally Posted by kovidgoyal

It's not too hard to implement but I'm not sure it should be, since there are a lot of cases where book metadata is incomplete so although the authors dont match, the books are actually the same.

Hmm, true. How about modifying the prompt to include more information?

For example, instead of displaying just the title of the duplicate match, you can show:

Code:

Existing Entries								Pending Additions
Foundation - Isaac Asimov (epub, 53KB)			Foundation - Mercedes Lackey (epub, 68KB)
Macbeth - William Shakespeare (epub, 92KB)			Macbeth (txt, 47KB)

Maybe an option to save the list to a txt/csv file can be added, too?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Possible Bug on Duplicates	Giuseppe Chillem	Calibre	3	05-06-2010 07:26 PM
Duplicates	pauldadams	Calibre	17	05-04-2010 11:57 PM
Duplicates...	jaxx6166	Sony Reader	5	07-09-2009 09:13 PM
duplicates in database	RJA	Calibre	3	06-22-2009 09:06 AM
Cybook not found in linux, found in win XP	fjf	Bookeen	15	01-18-2008 06:57 PM

02-22-2009, 04:08 PM	#1
Moejoe Banned Posts: 5,100 Karma: 72193 Join Date: Feb 2009 Location: South of the Border Device: Coffin	Duplicates found! I'm encountering a problem and hope it's just my own stupidity that's causing this. Every time I try to add a new book (tried with .lit, .epub and others) I receive this warning. The books are fresh and have never been added to the library. If I choose OK 2 entries are created. One contains the original ebook, the other a ZIP file. This only started happening today, and I already have some 100+ books in the library that imported without any such errors. I'm running Mac OS X 10.5.6 and Calibre 0.4.139. Thanks in advance for any advice.

02-22-2009, 04:11 PM	#2
pilotbob Grand Sorcerer Posts: 19,832 Karma: 11844413 Join Date: Jan 2007 Location: Tampa, FL USA Device: Kindle Touch	Do you by any chance have the book and a zip containing the book in the same folder and are selecting the "folder" that those books are in? Or perhaps you have two formats of the same book in the folder? I think the default add books will add all books in the folder if you use the default add books button. How "exactly" are you adding the book? BOb

02-23-2009, 09:25 AM	#4
idragon Junior Member Posts: 4 Karma: 10 Join Date: Feb 2009 Device: itouch	Welcome to the club ... There are a few of us having this problem, https://www.mobileread.com/forums/showthread.php?t=39427 I believe Kovid is looking into this, but I suspect its a fairly obscure bug. I had a quick look at some of the source code, but havent been able to spot anything out of the ordinary.

02-23-2009, 11:46 AM	#5
ilovejedd hopeless n00b Posts: 5,110 Karma: 19597086 Join Date: Jan 2009 Location: in the middle of nowhere Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9	A bit OT, but close... How does Calibre check for duplicates when adding new books? By filename? By title? I've added some books that aren't duplicates (same title but different authors) and Calibre seems to be recognizing them as duplicates. Thanks in advance!

02-23-2009, 12:58 PM	#6
kovidgoyal creator of calibre Posts: 45,229 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I've found the bug, will be fixed in the next release. calibre uses the title to check for duplicates. It does not compare authors, since the case of books having the same title by different authors is rare and you can simply manually add them in that case.

02-23-2009, 04:18 PM	#7
idragon Junior Member Posts: 4 Karma: 10 Join Date: Feb 2009 Device: itouch	I dont think this specific error/bug is related to the check for duplicates with the Title. I have definitely added books that didnt exist before (different titles) and it still popped up with the error. There is also the problem that it makes copies of previous books and adds them under the "new" book (If you click on Ok to add duplicates) The OP mentiones that, and in the other thread the OP also had that problem, and I can attest to that exact same behaviour. I just noticed this user in Ticket #1853 got the error with a new clean database after the 2nd add. Its almost like an array/variable somewhere doesnt get cleaned up propely (with links to previous files that were added) , and then when you click on ok to add the Duplicate. It copies that "other" file under the new book as a book format. Hope that makes sense

02-23-2009, 05:01 PM	#9
kovidgoyal creator of calibre Posts: 45,229 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It was a threading related bug caused by different threading behavior on the OS X platform. Basically, each time you add books a new thread ic created to carry out the work without frezing the GUI. On OSX the old threads were not being deleted so everytime you add anew book all the previous add threads would try to re-add their books as well, causing the duplicates.

02-23-2009, 05:26 PM	#12
gttexas Junior Member Posts: 3 Karma: 10 Join Date: Sep 2008 Location: Richardson, TX Device: Kindle, PRS 300	I've also puzzled over the problem until I noticed that the "duplicate" book was titled "Zane Grey". It seems I had a number of books to be imported which had the title and Author fields reversed!

Advert

Advert