[GUI Plugin] Find Duplicates

[theducks]

Originally Posted by AnselmD

Nice Idea, thies could help me partly.

i tried, it, but it does not work.

After defining the VL, starting Duplicate finder:
Analyzing 0 books for duplicates...

After that the VL is empty, i think Duplicate finder deletes the marks.

Correct. Duplicate finder Uses Marks, so it deletes any that WERE there at the beginning of the run.
That does not stop you from modifying (adding/removing) those present DURING the run.

As long as you do not fire off something else, marks remain at the end of the run

[BetterRed]

Originally Posted by theducks

Correct. Duplicate finder Uses Marks, so it deletes any that WERE there at the beginning of the run.
That does not stop you from modifying (adding/removing) those present DURING the run.

As long as you do not fire off something else, marks remain at the end of the run

If you use Find Duplicates->Clear duplicate results (ctrl+shift+c) when you've finished deduping then all Marks are removed. With the exception of any duplicate groups that are flagged as 'exempt', they will retain their Marks and those Marks will persist because they get reinstated when calibre starts.

This has been a problem ever since the Mark books function was made available as a user initiated feature. Prior to that Mark books was an internal function, which was only used by this PI and maybe one or two others*.

Added - * like Extract ISBN, see latest post in its thread, same problem.

BR

[Kokoro Fasching]

I would like to confirm something please. When I do a Find Duplicates in Libraries, the list that is returned is only the duplicate books in the current active library? It does not show the books that was in the library that was compared against, correct?

i.e. I am in Library Import - and I do a Compare Libraries, and select library Master, the list of possible duplicates are only the books in library Import?

Thank you in advance.

[BetterRed]

Originally Posted by Kokoro Fasching

I would like to confirm something please. When I do a Find Duplicates in Libraries, the list that is returned is only the duplicate books in the current active library? It does not show the books that was in the library that was compared against, correct?

i.e. I am in Library Import - and I do a Compare Libraries, and select library Master, the list of possible duplicates are only the books in library Import?

Thank you in advance.

That's how I use it - i.e. run it from Secondary and compare to Primary.

To look into the Primary at the same time, I access it via the calibre-server command.

BR

[stanmarsh]

I like to share a faster hash checker i found called xxHash, https://cyan4973.github.io/xxHash/

This could be used to make the plugin binary compare faster.

[BetterRed]

Originally Posted by stanmarsh

I like to share a faster hash checker i found called xxHash, https://cyan4973.github.io/xxHash/

This could be used to make the plugin binary compare faster.

@stanmarsh - the developer of the plugin is currently inactive

Unless one of the other PI developers is especially interested in doing the work, the best way to see if there's any interest is for you to incorporate your find into the plugin and then to offer it as an 'experimental' version to those interested in testing it (not me - I never do binary compares).

BR

[davidfor]

Originally Posted by stanmarsh

I like to share a faster hash checker i found called xxHash, https://cyan4973.github.io/xxHash/

This could be used to make the plugin binary compare faster.

As BR said, the author isn't active, but if problems are found in his plugins, someone usually fixes them.

But, this wouldn't be an attractive change. It's a compiled library. I don't think including it in the plugin is practical as there would need to be different versions for each OS. As calibre comes with its own Python installation, I think it would need to be included there to be useful.

[BetterRed]

@davidfor - if the xxHash library could be encapsulated into an executable, then maybe it would be possible to loosely couple' it into a PI, much as the wordconv.exe is in the Doc Input PI or the Prince program is in Jelby's Prince PDF PI.

BR

[theducks]

Wouldn't the hash be affected by fairly 'minor' Calibre activities, like Polish, Which updates the cover and jacket?

The book is the same otherwise

[davidfor]

Originally Posted by BetterRed

@davidfor - if the xxHash library could be encapsulated into an executable, then maybe it would be possible to loosely couple' it into a PI, much as the wordconv.exe is in the Doc Input PI or the Prince program is in Jelby's Prince PDF PI.

Yes, but it means someone has to create it. Both of the other examples are using pre-existing applications. Plus, they are run once per book. In this case, I suspect the cost of starting the executable to calculate the hash would outweigh the benefit. I might be wrong, but I'm not interested enough to do the work needed to do the tests.

Originally Posted by theducks

Wouldn't the hash be affected by fairly 'minor' Calibre activities, like Polish, Which updates the cover and jacket?

The book is the same otherwise

Yes, but stanmarsh is suggesting it for the binary compare. And that is going to be affected by these things. But, if there is a match, then the files are definitely identical.

[kovidgoyal]

I haven't looked at how this plugin does binary compare, but surely it only computes hashes if file sizes are first found to be identical? In which case, is hash implementation speed really a significant bottleneck?

[BetterRed]

Originally Posted by davidfor

Yes, but it means someone has to create it. Both of the other examples are using pre-existing applications. Plus, they are run once per book. In this case, I suspect the cost of starting the executable to calculate the hash would outweigh the benefit. I might be wrong, but I'm not interested enough to do the work needed to do the tests.

@davidfor - I did write '...Unless one of the other PI developers is especially interested in doing the work...' - I didn't think any of them would be. But we ought not discourage stanmarsh from experimenting - new PI developers may not be hens teeth but nor do they grow on trees

If you drill into the link provided you get to code, at a glance it appears to written in C++, can a PI call a dll, can a...

<joke>Maybe one could run an always enabled service that encaps the algo</joke>

Originally Posted by kovidgoyal

In which case, is hash implementation speed really a significant bottleneck?

Only on very, very, very large libraries

BR

[whismerhill]

Hi, this plugin is great
however I wanted to report that the "find duplicates by title" option didn't always work out for me
seems like for example it would never find out that :
"[series name - series number]bookname"
is the same as :
"bookname"
(all in the title of course)

I tried : Fuzzy, Similar, Soundex(99)

Thanks though, it's still a superb plugin.

[JohnnyBook]

I see the Author is not active.

Question, is there a way to search for Binary dups in a portion of my library? verses doing a dup search of my whole library? For example I want to do a search for duplicates among only the books by "Steven King"?

I can have Calibre display only the books by a specific author, but how do I tell the plug in I only want to check for Duplicates in that set of books?

Thanks

[theducks]

Originally Posted by JohnnyBook

I see the Author is not active.

Question, is there a way to search for Binary dups in a portion of my library? verses doing a dup search of my whole library? For example I want to do a search for duplicates among only the books by "Steven King"?

I can have Calibre display only the books by a specific author, but how do I tell the plug in I only want to check for Duplicates in that set of books?

Thanks

Most plugins respect Virtual Libraries as limits <<<
Some Plugins have a Selected books option