[GUI Plugin] Find Duplicates - Page 9

capnm · 08-06-2011, 11:58 AM

Quote:

Originally Posted by Noughty

Also maybe it could search more by size? Calibre shows only 0,X MB. If it showed it more detail in KB it would be easier to see if it is a dupe format.

I'll repeat my comment above, the Count Pages plugin can be really useful in identifying duplicates, similar to your idea about size, but better.

As to the rest, they really sound like jobs for the Mark I eyeball

Noughty · 08-06-2011, 01:13 PM

But it only counts epub and mobi books. And I mostly have pdf as before calibre integrated reader I found it best for me. So now my library has lots of old pdfs.
I need to find dupes among pdfs (like is it a dupe or a converted version of an original)

capnm · 08-06-2011, 02:22 PM

Quote:

Originally Posted by Noughty

So now my library has lots of old pdfs.
I need to find dupes among pdfs (like is it a dupe or a converted version of an original)

Ouch.

1) Find Duplicates isn't going to be looking inside any files. It is a tool to analyze the metadata stored in the Calibre database to identify possible duplicates -- leaving content analysis up to you.

2) For the most part, Calibre developers are only minimally interested in supporting pdfs since pdf is an extraordinarily unfriendly format to work with.

Sorry, I know these are not very helpful comments

Noughty · 08-07-2011, 07:50 AM

As a solution I made a new library for Dupes. I leave one book in my main library and transfer another to Dupes library. This way there is no danger of deleting an original or a different copy and it doesn't mess up my library. Books don't take up much place so deleting dupes ain't so important as them not messing with your library.

And now I do hate pdf too. Impossible to count pages, hard to convert (author,title remains in every page for epubs etc.)

Thanks for the help

DoctorOhh · 08-07-2011, 09:46 AM

Quote:

Originally Posted by Noughty

And now I do hate pdf too. Impossible to count pages, hard to convert (author,title remains in every page for epubs etc.)

I agree, but PDF to ePub only put the author,title on every page if these things exist in the PDF, which most purchased PDFs do have.

My library currently consists of 8600 ePubs and 5 PDFs.

mbovenka · 08-09-2011, 08:02 AM

I found out that using 'Soundex' for the Title and 'Ignore' for the Author in an Author/Title duplicate search doesn't work well. With my library (~40K books) the plugin starts eating memory like mad, in the end crashing Calibre when it runs out, which happens in half a minute or less (this on a 2.4GHz Corei5 with 2GB RAM + the same VM)

It doesn't do this when using Soundex for both, or indeed any other combo I have tried (mostly Fuzzy/Fuzzy or Fuzzy/Ignore) or when using ISBN matching.

Those all work just fine, and have weeded out literally thousands of dups (probably close to 4000) from the mess that was my ebook collection

nynaevelan · 08-16-2011, 01:19 PM

Hi Kiwidude:

I ran into some files that seem to have the author name reversed in my db, such as Brockmann Suzanne and Suzanne Brockmann, is there any way to use the plugin to find these files??

Nyn

kiwidude · 08-16-2011, 03:23 PM

@Nyn - if you do an "Ignore Title, Similar Author" search that should help you find those.

nynaevelan · 08-16-2011, 06:07 PM

Quote:

Originally Posted by kiwidude

@Nyn - if you do an "Ignore Title, Similar Author" search that should help you find those.

Thanks, I will try that.

It worked great, it found 4 more authors like that. Thanks again.

Nyn

mbovenka · 08-17-2011, 05:32 AM

Quote:

Originally Posted by kiwidude

@Nyn - if you do an "Ignore Title, Similar Author" search that should help you find those.

Yep, that catches lots of variant spellings (initials with or without periods, initials vs. full first names, things like that). I spent yesterday morning going through the output of that exact search standardizing my authors...

Thanks for a great plugin!

nynaevelan · 08-17-2011, 08:51 PM

Hi Kiwidude:

Me again,

I am not sure if this should be put here or in the Quality Check plugin. But, I was wondering if it would not be too difficult to add a check that looks for series with similar titles, to ensure that the series are named correctly.

Nyn

drMerry · 08-20-2011, 07:46 PM

Quote:

Originally Posted by nynaevelan

Hi Kiwidude:

Me again,

I am not sure if this should be put here or in the Quality Check plugin. But, I was wondering if it would not be too difficult to add a check that looks for series with similar titles, to ensure that the series are named correctly.

Nyn

That would be a great option indeed, for the quality check.

Philosopher · 08-24-2011, 12:21 PM

This is certainly one of the most useful and significant plugins - and I am glad it is set to become a part of the main program. Especially the ability to find duplicates by the file itself - and not just duplicate names. That is very powerful and useful.

I do have one suggestion - although I am not sure if it is possible (it seems like it should be) - about how to make it even better.

I would really like to be able to limit my checking for duplicates, at times, to a selected set of books - rather than the entire library.

This would especially be useful in order to focus on cleaning up one area of my library. One thing I should note - is that I do keep (intentionally) multiple copies of some books. Those copies, however, different in the file (not the name/identity). So I often wind up with multiple copies of the same file by accident - and like to routinely clean that up. (This is especially the case as I build the library from my files - often times having old drives contents dumped in - to find out what is not there and what is duplicated).

kiwidude · 08-24-2011, 12:26 PM

@Philosopher - you can do this already. Find Duplicates will respect any search restriction you have put in place. So do a search to bring back just the subset of books you are interested in. Then in the Restriction dropdown on the top left, select "*Current Search". Now if you use Find Duplicates (or indeed the Quality Check plugin as well) all operations are limited to just those books. When you are finished, clear the search restriction in the restriction dropdown to go back to your full library.

Philosopher · 08-24-2011, 04:01 PM

OK - didn't realize that. But that is because I tried it using the User Category to select the group - and it didn't seem to restrict it. I'll have to go back and see the difference. Thanks. (I thought that the User Category effectively does a search itself - but perhaps there is something different).

08-09-2011, 08:02 AM	#126
mbovenka Wizard Posts: 2,018 Karma: 13471689 Join Date: Oct 2007 Location: Almere, The Netherlands Device: Kobo Sage	Plugin starts eating huge amounts of memory I found out that using 'Soundex' for the Title and 'Ignore' for the Author in an Author/Title duplicate search doesn't work well. With my library (~40K books) the plugin starts eating memory like mad, in the end crashing Calibre when it runs out, which happens in half a minute or less (this on a 2.4GHz Corei5 with 2GB RAM + the same VM) It doesn't do this when using Soundex for both, or indeed any other combo I have tried (mostly Fuzzy/Fuzzy or Fuzzy/Ignore) or when using ISBN matching. Those all work just fine, and have weeded out literally thousands of dups (probably close to 4000) from the mess that was my ebook collection

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Quality Check	kiwidude	Plugins	1184	04-17-2024 06:17 PM
[GUI Plugin] View Manager	kiwidude	Plugins	414	04-13-2024 01:41 PM
[GUI Plugin] Open With	kiwidude	Plugins	403	04-01-2024 08:39 AM
[GUI Plugin] Generate Cover	kiwidude	Plugins	811	03-16-2024 11:31 PM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

08-06-2011, 01:13 PM	#122
Noughty Addict Posts: 352 Karma: 103850 Join Date: Apr 2011 Device: Kindle NT	But it only counts epub and mobi books. And I mostly have pdf as before calibre integrated reader I found it best for me. So now my library has lots of old pdfs. I need to find dupes among pdfs (like is it a dupe or a converted version of an original)

08-07-2011, 07:50 AM	#124
Noughty Addict Posts: 352 Karma: 103850 Join Date: Apr 2011 Device: Kindle NT	As a solution I made a new library for Dupes. I leave one book in my main library and transfer another to Dupes library. This way there is no danger of deleting an original or a different copy and it doesn't mess up my library. Books don't take up much place so deleting dupes ain't so important as them not messing with your library. And now I do hate pdf too. Impossible to count pages, hard to convert (author,title remains in every page for epubs etc.) Thanks for the help

08-16-2011, 01:19 PM	#127
nynaevelan eBook Junkie Posts: 1,526 Karma: 1464018 Join Date: May 2010 Location: USA Device: Kindle Fire 2020, Kindle PW2	Hi Kiwidude: I ran into some files that seem to have the author name reversed in my db, such as Brockmann Suzanne and Suzanne Brockmann, is there any way to use the plugin to find these files?? Nyn

08-16-2011, 03:23 PM	#128
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@Nyn - if you do an "Ignore Title, Similar Author" search that should help you find those.

08-17-2011, 08:51 PM	#131
nynaevelan eBook Junkie Posts: 1,526 Karma: 1464018 Join Date: May 2010 Location: USA Device: Kindle Fire 2020, Kindle PW2	Hi Kiwidude: Me again, I am not sure if this should be put here or in the Quality Check plugin. But, I was wondering if it would not be too difficult to add a check that looks for series with similar titles, to ensure that the series are named correctly. Nyn

08-24-2011, 12:21 PM	#133
Philosopher Connoisseur Posts: 77 Karma: 12 Join Date: Jun 2010 Device: Kindle	This is certainly one of the most useful and significant plugins - and I am glad it is set to become a part of the main program. Especially the ability to find duplicates by the file itself - and not just duplicate names. That is very powerful and useful. I do have one suggestion - although I am not sure if it is possible (it seems like it should be) - about how to make it even better. I would really like to be able to limit my checking for duplicates, at times, to a selected set of books - rather than the entire library. This would especially be useful in order to focus on cleaning up one area of my library. One thing I should note - is that I do keep (intentionally) multiple copies of some books. Those copies, however, different in the file (not the name/identity). So I often wind up with multiple copies of the same file by accident - and like to routinely clean that up. (This is especially the case as I build the library from my files - often times having old drives contents dumped in - to find out what is not there and what is duplicated).

08-24-2011, 12:26 PM	#134
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@Philosopher - you can do this already. Find Duplicates will respect any search restriction you have put in place. So do a search to bring back just the subset of books you are interested in. Then in the Restriction dropdown on the top left, select "*Current Search". Now if you use Find Duplicates (or indeed the Quality Check plugin as well) all operations are limited to just those books. When you are finished, clear the search restriction in the restriction dropdown to go back to your full library.

08-24-2011, 04:01 PM	#135
Philosopher Connoisseur Posts: 77 Karma: 12 Join Date: Jun 2010 Device: Kindle	OK - didn't realize that. But that is because I tried it using the User Category to select the group - and it didn't seem to restrict it. I'll have to go back and see the difference. Thanks. (I thought that the User Category effectively does a search itself - but perhaps there is something different).