Have many duplicate files

[Jaejae]

I am currently changing laptop you see and I transfered all my files from my old calibre library to my new laptop. I downloaded calibre 64bit. The problem starts when my old calibre files have duplicates, I literally have like thousands of books and I have no idea how many of them have duplicates. Too many subfolder for me to check one by one. Anyway, I was unaware of the problem at first and spent like an hour importing my ebook files to my new calibre library and it chose to wait until the end of the transfer to tell me I have like 1k+ duplicate files and urges me to select action. I chose not to transfer the duplicate files so I won't end up with up with 2 ebook versions of everything. But after that, I clicked on the files in calibres and they says 'no ebook format found'.


What do i do? Anyone any idea?

[eschwartz]

http://manual.calibre-ebook.com/faq....ter-to-another

[BetterRed]

@Jaejae - FWIW: for aesthetic reasons only, I would have cleaned up the duplicates on the old computer using the Find Duplicates plugin before I migrated my libraries to the shiny new computer.

But you could use it equally well on the new computer, probably more so if it's a faster computer.

BR

[Lizj1]

Jaejae,
it sounds as though Calibre found and imported the opf files first (which I think just contain the metadata needed by Calibre), then reported the actual book files as duplicates (the detailed messages would have told you what formats were already imported - I'm guessing none - and what formats were in the duplicate)

Can you start again from the beginning? Create a new, empty library, and this time when you add books use the "One book per directory" option. Since you are adding them from a Calibre library, that should mean that the opf ad the book file get handled as one.

Of coyurse, if you DO have duplicate books, you will need the Find Duplicates plugin as BetterRed says.

The other thing you can do is go in Preferences, Adding Books, and set Automerge when adding duplicate books. This will only help when the book details (title and author) look IDENTICAL.

Hope this helps,
Liz

[BetterRed]

Originally Posted by Lizj1

Jaejae,
it sounds as though Calibre found and imported the opf files first (which I think just contain the metadata needed by Calibre), then reported the actual book files as duplicates (the detailed messages would have told you what formats were already imported - I'm guessing none - and what formats were in the duplicate)

Can you start again from the beginning? Create a new, empty library, and this time when you add books use the "One book per directory" option. Since you are adding them from a Calibre library, that should mean that the opf ad the book file get handled as one.

Of coyurse, if you DO have duplicate books, you will need the Find Duplicates plugin as BetterRed says.

The other thing you can do is go in Preferences, Adding Books, and set Automerge when adding duplicate books. This will only help when the book details (title and author) look IDENTICAL.

Hope this helps,
Liz

@Lizj1 - did you look into the link posted by eschwartz in post #2. It references a recently added feature that migrates the libraries and the configuration data to a new machine at one fell swoop. It doesn't go through the rigmarole of adding the books again, not that that was entirely necessary anyway.

BR

[Lizj1]

BetterRed, Oops, no, I did not ... I see that is a better way (if your collection is smallish, or if you use Synctoy or similar - Copying between computers usually causes me more problems than re-adding) !

[theducks]

Originally Posted by Lizj1

BetterRed, Oops, no, I did not ... I see that is a better way (if your collection is smallish, or if you use Synctoy or similar - Copying between computers usually causes me more problems than re-adding) !

Really?

If you followed the advice (before the new tool), it worked every time between computers using the same OS Family. I have done it dozens of times on good sized libraries (5K+ books) and all it took was time .

The only time I had issues was with a Library that I touched (metadata) with Linux, then tried it with Windows (mostly name case issues)

[BetterRed]

And providing you watch your peas and queues, it also works between Windows and Mac (and back again a few weeks later Ψ²)

I've helped at least half-a-dozen people migrate to new machines over the past 2-3 years, Never did more than copy library folders holus-bolus to new machine, likewise with configuration data. If the old machine was working, I de-duped and resolved library check errors first.

Assuming you want your data to be clean and tidy on the new computer, calibre is a walk in the park compared with pictures, music, etc which are invariably a dogs breakfast. Mail is the worst, it often means migrating to a new client; OE to anything, apart from the bin, can be a nightmare.

BR

[theducks]

The only other thing to be done is point Calibre at the Libraries (New) Location.

That is easily done with Switch/Create (changing Windows version, usually changes the path to the User Files , even if you kept the same name for the Library folder)

[Alpha o]

I had the same - finally straightened out. It was Kobo, not Calibre.

I normally download to the computer and stick the files on the SD card. Put the card into Kobo Glo, and it reimported quite a few of them if the date and author were different. (some books show "unknown")

I've always transferred that way - I might be using an Android tablet (which uses a card) a Windows laptop, and a Mint desktop which may see or not see the device.
Easier to bring the card to the reader since all of them see the card.

I won't use cloud storage. There is a need for some hard and fast rules if a company changes its mind about "free" (windows just did) or any other policy.

[davidfor]

Originally Posted by Alpha o

I normally download to the computer and stick the files on the SD card. Put the card into Kobo Glo, and it reimported quite a few of them if the date and author were different. (some books show "unknown")

Sorry, you are only half right. The Kobo devices work on the full file path to identify books. You can put the identical book with identical metadata in two different places, and as far as the Kobo device is concerned you have two different books. It will list them as two books in the library.

If you replace a copy of a book, using the same file name and path, then the firmware will check if it has changed. If it has changed, then the book is removed from the database and then added as a new book. If it has not changed, nothing is done. The only thing that is used to determine "changed" is the file size. This is recorded in the database when the book is first seen. If the next time the device checks the book and the file size has changed, the book is considered to be a new book. If you are using calibre to put the books onto the SD card, and then replace them later, it is highly likely they changed. Calibre updates the metadata during a save-to-disk or send-to-device, so, if any metadata changed, the file size is probably going to change.

Note: The KoboTouch driver has a workaround for this and will update the file size in the database when sending new copy of the book to the device.