Duplicates

pauldadams · 12-22-2009, 08:10 PM

I just started using Calibre. After reading in all of my ebooks and updating the meta data for each one, I copied (since I have a WinMobile device) my Calibre directory to my device. When I opened my library with Mobipocket I saw that I had 2 of every book.

When I looked through my Calibre directory on my XP notebook I found that I had at least 2 and in many cases 3 or 4 copies of the same book. I don't believe this should ever be allowed to happen, let alone for each book!

I took the time to import all books including duplicates back in to Calibre and went through each entry and deleted those copies that I don't want. Calibre warns that this will delete the file from my computer but this is exactly what I wanted.

All of the duplicates are still here. Additionally, the detail view of many books show the wrong book cover although the meta data view shows the correct cover. If I save the meta data the detail image will update to the correct cover until I move away and come back. Then the old, incorrect image appears in the detail view, although the meta data is still correct.

I now have a horrible mess on my hard disk which I will need to go through to delete one by one. I imagine a lot of correct meta data will also be deleted. If I go through to update the meta data, will I gain new duplicates?

This is definitely NOT what I expected to get!

polly · 12-24-2009, 04:35 AM

It sounds like you went mucking about in the files maintained by Calibre. That's guaranteed to make a mess because Calibre is only aware of the changes it makes there.

The trick with your Calibre library is to leave it alone. Don't even open it to look inside. Hide it someplace deep in your directory if it offends you. Just leave it alone and let Calibre deal with it.

If you want to extract books from Calibre, Save to Disk will put the desired books wherever you want. Trust the metadata that you see within Calibre. It only gives me duplicate books when I add the same book twice, and then it warns me that that book is already in Calibre. Or were you talking about the same book in different formats?

pauldadams · 12-25-2009, 11:23 AM

No, I didn't touch anything in the Calibre directory. Calibre doesn't recognize when my WinMobile device is plugged in (even though Windows does and I could save files there via Windows) so I had to do a copy of the Calibre directory to my device to move my books there. When I opened the library with Mobipocket mobile to read my books I noticed there were duplicates and then found the duplicate directories on my hard drive. I had multiple sub-directories for the same book just with a different number in parentheses at the end of the directory name.

I imported a the whole directory back in to Calibre and deleted the duplicates. Some I had to manually delete via Windows. My hard disk is now cleaned up but I continue to have issues with the wrong cover appearing in the detail view. The metedata view is correct and the jpg file in the directory is correct. So this is Calibre mis-reading data from metadata.db file, which I assure you I've never touched.

Starson17 · 12-26-2009, 09:29 AM

Quote:

Originally Posted by pauldadams

had to do a copy of the Calibre directory to my device to move my books there.

You didn't have to copy the Calibre directory. You should have done a "save to disk" to a temporary directory on your hard drive, and then copied *that* directory. As polly said, you shouldn't ever even look inside the Calibre Library directory with Windows Explorer. Calibre will keep multiple copies of the same book in the same folder, but these aren't really duplicates - they are just different formats.

Quote:

I had multiple sub-directories for the same book just with a different number in parentheses at the end of the directory name.

The different numbers indicate that you imported the same book more than once. (The number is the ID that Calibre uses to identify a book) Calibre warns you when you try to do this. These are true duplicates. You can get rid of the second copy by deleting it, or if it is a different format from the first copy, by saving the second to disk, deleting it from Calibre and bringing it back as a different format of the first book.

Edit: BTW, I have four Win Mobile Devices, and the newest one can optionally connect through either ActiveSync or as a USB drive to Windows. Calibre can only talk directly to it when it's connected as a USB drive. Windows can only sync to it when connected via ActiveSync. You may want to check the manual for your Win Mobile device to see if it has this feature. It asks me when I first connect the USB cable how I want to connect.

Darryl_Gittins · 04-29-2010, 11:52 AM

I've already done this (mucked with the files) and totally botched my entire book collection. I now have a huge mess of duplicates.
Can anyone offer suggestions on how to start from scratch and clean up the mess?

Starson17 · 04-29-2010, 04:07 PM

Quote:

Originally Posted by Darryl_Gittins

I've already done this (mucked with the files) and totally botched my entire book collection. I now have a huge mess of duplicates.
Can anyone offer suggestions on how to start from scratch and clean up the mess?

I suggest:

Get your Calibre GUI interface to the library cleaned up, so that you have one and only one entry for each book you want, with all the book formats for each and all metadata set correctly. You can stop there ... or ...

You can then Save to Disk with the .opf and .jpg options turned on (they are by default), then turn off Calibre, change the name of your Calibre library (so Calibre can't find the library) and Add Books all of the Books you Saved to Disk. That will give you a clean library, with all your metadata.

If you stopped before the last steps, you'd have some extra junk in your library, but you wouldn't see it and it wouldn't cause any harm other than eating up some space.

Edit: If you have duplicates in the database that appear in the GUI, you may want to wait until the next release of Calibre when the Merge function will be introduced. You'll be able to merge those dupes, with all the metadata.

itimpi · 04-29-2010, 06:32 PM

Quote:

Originally Posted by Starson17

Edit: If you have duplicates in the database that appear in the GUI, you may want to wait until the next release of Calibre when the Merge function will be introduced. You'll be able to merge those dupes, with all the metadata.

The merge function is already there (in the 0.6.49 release). I was using it extensively late yesterday and earlier today.

Starson17 · 04-29-2010, 08:19 PM

Quote:

Originally Posted by itimpi

The merge function is already there (in the 0.6.49 release). I was using it extensively late yesterday and earlier today.

You're running it in the source, not the binary. Kovid develops directly into the source, then does a snapshot and upgrades the version number. Anyone running the source from a few days after 0.6.49 was released has it, but anyone running the 0.6.49 binary does not, since that snapshot was taken just before the Merge code was added. I assumed the OP wasn't running source in a development environment.

(I hope you like it. I put a lot of effort into that code ... even if Kovid did have to fix it up)

itimpi · 04-30-2010, 01:53 AM

Quote:

Originally Posted by Starson17

You're running it in the source, not the binary. Kovid develops directly into the source, then does a snapshot and upgrades the version number. Anyone running the source from a few days after 0.6.49 was released has it, but anyone running the 0.6.49 binary does not, since that snapshot was taken just before the Merge code was added. I assumed the OP wasn't running source in a development environment.

OK - that will be it as I tend to download the source as well whenever I download a new release. . A shame that there is not a way of displaying in the GUI that one is running from source as a reminder. I have raised ticket #5417 to request this as it should be an easy enhancement. In fact I may look at the source myself later today to see if I can figure out the exact change reuqired - having raised the request I might as well see if I can also figure out the fix

Quote:

(I hope you like it. I put a lot of effort into that code ... even if Kovid did have to fix it up)

It is definitely a significant improvement - already saves me a lot of work as I can now simply import any book without regard to format and sort them into a single entry afterwards when required.

Starson17 · 04-30-2010, 07:59 AM

Quote:

Originally Posted by itimpi

A shame that there is not a way of displaying in the GUI that one is running from source as a reminder.

I have raised ticket #5417 to request this as it should be an easy enhancement. In fact I may look at the source myself later today to see if I can figure out the exact change reuqired - having raised the request I might as well see if I can also figure out the fix

I think that would be an excellent addition. I didn't even realize until a couple of weeks ago that the source differed from the binaries. Thanks for tackling it.

Quote:

It is definitely a significant improvement - already saves me a lot of work as I can now simply import any book without regard to format and sort them into a single entry afterwards when required.

Yes, that's why I wrote it (or as much of it as Kovid kept)

I just wish I could code well enough that he didn't have to completely rewrite my efforts. I hesitate to ask for an improvement, thinking I should do it myself, then hesitate to do it, thinking he'll just have to rewrite it anyway.

kovidgoyal · 04-30-2010, 09:32 AM

Quote:

Originally Posted by Starson17

I hesitate to ask for an improvement, thinking I should do it myself, then hesitate to do it, thinking he'll just have to rewrite it anyway.

Rewriting is always easier than writing from scratch, so don't hesitate

Jaelle · 05-01-2010, 08:44 AM

I have a related question on this that I've been wondering about. From reading the thread above it appears that I probably caused the issue with first importing all my books into calibre, and then using the save to disk button, creating duplicates.

When I look at the calibre library with Windows Explorer, there are two folders for every author. There is a folder with the authors name, first name then last name. And there is a folder with the authors name, last name comma first name.

The folders that have the first name first have each book in subfolders with a number in parentheses. These folders contain the ebook file and a cover.

The folders that have the last name first contain the ebook file, jpeg for the cover, and the .opf file.

I don't want every book stored on my hard drive twice. So I have two questions.

1) Which set of folders should be deleted without doing any harm to my calibre library?

2) How do I avoid everything being duplicated in the future? Stop using Save to Disk? (I was under the impression I guess that you should do a save after importing into calibre.)

Thanks for any help!

itimpi · 05-01-2010, 09:37 AM

The folders with a number in the name are those that belong in the Calibre library. Any without numbers have been created by some method other han importing into Calibre.

The Save to Disk facility is used for when you want to export a file from the Calibre library. No need to do it as a matter of course. Sounds as if you used it and then gave the same location for the Save to Disk as your Calibre library and thus the duplication.

Jaelle · 05-01-2010, 09:41 AM

Thanks! I thought that was the case, but I wanted to make sure before I start deleting the extra folders.

DoctorOhh · 05-01-2010, 10:21 AM

Quote:

Originally Posted by Jaelle

Thanks! I thought that was the case, but I wanted to make sure before I start deleting the extra folders.

The save to disk feature is a great way to back up your books, just make sure to pick a different folder next time.

12-22-2009, 08:10 PM	#1
pauldadams Junior Member Posts: 2 Karma: 10 Join Date: Dec 2009 Device: WinMobile	Duplicates I just started using Calibre. After reading in all of my ebooks and updating the meta data for each one, I copied (since I have a WinMobile device) my Calibre directory to my device. When I opened my library with Mobipocket I saw that I had 2 of every book. When I looked through my Calibre directory on my XP notebook I found that I had at least 2 and in many cases 3 or 4 copies of the same book. I don't believe this should ever be allowed to happen, let alone for each book! I took the time to import all books including duplicates back in to Calibre and went through each entry and deleted those copies that I don't want. Calibre warns that this will delete the file from my computer but this is exactly what I wanted. All of the duplicates are still here. Additionally, the detail view of many books show the wrong book cover although the meta data view shows the correct cover. If I save the meta data the detail image will update to the correct cover until I move away and come back. Then the old, incorrect image appears in the detail view, although the meta data is still correct. I now have a horrible mess on my hard disk which I will need to go through to delete one by one. I imagine a lot of correct meta data will also be deleted. If I go through to update the meta data, will I gain new duplicates? This is definitely NOT what I expected to get!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre Duplicates	mitch13	Calibre	5	11-13-2010 06:42 AM
Possible Bug on Duplicates	Giuseppe Chillem	Calibre	3	05-06-2010 07:26 PM
Duplicates...	jaxx6166	Sony Reader	5	07-09-2009 09:13 PM
duplicates in database	RJA	Calibre	3	06-22-2009 09:06 AM
Duplicates found!	Moejoe	Calibre	32	06-12-2009 12:20 AM

12-24-2009, 04:35 AM	#2
polly Evangelist Posts: 454 Karma: 270240 Join Date: Aug 2009 Device: Sony PRS 650, PocketBook 360, Astak PocketPro (RIP), Tungsten T3	It sounds like you went mucking about in the files maintained by Calibre. That's guaranteed to make a mess because Calibre is only aware of the changes it makes there. The trick with your Calibre library is to leave it alone. Don't even open it to look inside. Hide it someplace deep in your directory if it offends you. Just leave it alone and let Calibre deal with it. If you want to extract books from Calibre, Save to Disk will put the desired books wherever you want. Trust the metadata that you see within Calibre. It only gives me duplicate books when I add the same book twice, and then it warns me that that book is already in Calibre. Or were you talking about the same book in different formats?

12-25-2009, 11:23 AM	#3
pauldadams Junior Member Posts: 2 Karma: 10 Join Date: Dec 2009 Device: WinMobile	No, I didn't touch anything in the Calibre directory. Calibre doesn't recognize when my WinMobile device is plugged in (even though Windows does and I could save files there via Windows) so I had to do a copy of the Calibre directory to my device to move my books there. When I opened the library with Mobipocket mobile to read my books I noticed there were duplicates and then found the duplicate directories on my hard drive. I had multiple sub-directories for the same book just with a different number in parentheses at the end of the directory name. I imported a the whole directory back in to Calibre and deleted the duplicates. Some I had to manually delete via Windows. My hard disk is now cleaned up but I continue to have issues with the wrong cover appearing in the detail view. The metedata view is correct and the jpg file in the directory is correct. So this is Calibre mis-reading data from metadata.db file, which I assure you I've never touched.

04-29-2010, 11:52 AM	#5
Darryl_Gittins Junior Member Posts: 6 Karma: 10 Join Date: Jan 2010 Device: Sony Reader	I've already done this (mucked with the files) and totally botched my entire book collection. I now have a huge mess of duplicates. Can anyone offer suggestions on how to start from scratch and clean up the mess?

05-01-2010, 08:44 AM	#12
Jaelle Junior Member Posts: 9 Karma: 10 Join Date: Mar 2010 Device: Kindle 2	I have a related question on this that I've been wondering about. From reading the thread above it appears that I probably caused the issue with first importing all my books into calibre, and then using the save to disk button, creating duplicates. When I look at the calibre library with Windows Explorer, there are two folders for every author. There is a folder with the authors name, first name then last name. And there is a folder with the authors name, last name comma first name. The folders that have the first name first have each book in subfolders with a number in parentheses. These folders contain the ebook file and a cover. The folders that have the last name first contain the ebook file, jpeg for the cover, and the .opf file. I don't want every book stored on my hard drive twice. So I have two questions. 1) Which set of folders should be deleted without doing any harm to my calibre library? 2) How do I avoid everything being duplicated in the future? Stop using Save to Disk? (I was under the impression I guess that you should do a save after importing into calibre.) Thanks for any help!

05-01-2010, 09:37 AM	#13
itimpi Wizard Posts: 4,553 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	The folders with a number in the name are those that belong in the Calibre library. Any without numbers have been created by some method other han importing into Calibre. The Save to Disk facility is used for when you want to export a file from the Calibre library. No need to do it as a matter of course. Sounds as if you used it and then gave the same location for the Save to Disk as your Calibre library and thus the duplication.

05-01-2010, 09:41 AM	#14
Jaelle Junior Member Posts: 9 Karma: 10 Join Date: Mar 2010 Device: Kindle 2	Thanks! I thought that was the case, but I wanted to make sure before I start deleting the extra folders.

Advert

Advert