Big messy library, low skill - What's a good workflow?

Findingaway · 07-18-2026, 03:06 PM

My ebook library folder has around 57 GB. Per AllDup, a freeware deduplicator, when searched by file name including extension, there are ~ 6 GB of duplicates.

I've never successfully applied a sufficient organization scheme to the contents. They are split into multiple Calibre libraries. I also have Genre and other columns (like DCC / LCC from when the plugin worked) and a mygenre column.

I attempted a hierarchical genre scheme without success. By "without success," I mean there must be error in how I tried it, because the results do not match Calibre documentation. Deciding on a scheme and partially implementing it has been useful though.

I can use my library as it is and can dedicate some but not unlimited time to fixing it. But I would like to improve its functionality.

The question is, with low coding and low Calibre expertise, what's a good way to tackle this?

What I'm thinking of doing is - after getting the hang of Alldup - deduplicating the whole 6 GB and then rebuilding my Calibre libaries (for now plural), i.e. deleting and then replacing the library to get a correct database).

I think some of my past attempts have failed due to biting off more than I could chew. I'll think "I just have to install x," and then x has 5 substeps, and y has 7 more substeps, it's all new to me and it doesn't get done.

Thanks everyone for listening to my TED plea.

JSWolf · 07-18-2026, 03:18 PM

Use the plugin Find Duplicates. Use it to find binary duplicates and then do a similar duplicate find. That's the best way to start cleaning your library. Don't use Alldup on your library. Don't use any external program on your library,

Sirtel · 07-18-2026, 03:24 PM

With a really messy library, there's unfortunately no other way to do this than manually. That is, going through it book by book, correcting metadata, deleting duplicates. With correct metadata you can use the Find Duplicates plugin to find and remove duplicates, but the plugin doesn't work when metadata is different between copies. So you'll have to fix the metadata first.

I know it's a lot of work, as I have a huge calibre library myself. But there's just no automated way to do it and get correct results.

Personally I don't bother with detailed or hierarchical genres - I just use a handful of broad genre categories and for more detail there's the synopsis in comments. Anything else would just be too much work when you have thousands of books. But YMMV, of course.

Start going through your books a few a day. Take your time. Don't try to do it all in a couple of weeks or even months. There's no hurry.

Sirtel · 07-18-2026, 03:30 PM

Quote:

Originally Posted by JSWolf

Use the plugin Find Duplicates. Use it to find binary duplicates and then do a similar duplicate find. That's the best way to start cleaning your library. Don't use Alldup on your library. Don't use any external program on your library,

While I agree with you in principle, this only works with identical copies. I'm pretty sure most duplicates in a messy library are not identical, unless they've just been copied between different libraries. But then the metadata would presumably also be identical.

It's more likely that the duplicates are different versions of the same book, downloaded at different times, maybe from different vendors and websites. There's no automated way to find duplicates with such books.

JSWolf · 07-18-2026, 03:32 PM

Quote:

Originally Posted by Sirtel

While I agree with you in principle, this only works with identical copies. I'm pretty sure most duplicates in a messy library are not identical, unless they've just been copied between different libraries. But then the metadata would presumably also be identical.

It's more likely that the duplicates are different versions of the same book, downloaded at different times, maybe from different vendors and websites. There's no automated way to find duplicates with such books.

The binary duplicate would find any binary duplicates if there are any. That only needs to be run once. The similar duplicate could find some to delete.

After that, clean up the library manually and then run the similar duplicate finder.

maddz · 07-18-2026, 04:32 PM

Quote:

Originally Posted by Sirtel

It's more likely that the duplicates are different versions of the same book, downloaded at different times, maybe from different vendors and websites. There's no automated way to find duplicates with such books.

That's quite common in my library - I often buy bundles, and quite often have a different version of the same title in my library - for example, earlier this year I purchased the Discworld bundle - all the Discworld books bar one in US editions. I had some already in UK editions (I'd been slowly buying the Night Watch titles)...

Or I'll have an eARC and also have a purchased copy.

Sirtel · 07-18-2026, 04:43 PM

I have some books where I've bought a book, say, a decade ago and later bought a more recent version which was on sale or even temporarily free. Usually I remove the earlier version from my library and store it elsewhere, but I haven't gotten to all such books yet.

Karellen · 07-18-2026, 05:15 PM

Quote:

Originally Posted by Findingaway

My ebook library folder has around 57 GB.
...
there are ~ 6 GB of duplicates.

This is meant to sound like a lot, but how many actual books are there?

theducks · 07-18-2026, 05:55 PM

Quote:

Originally Posted by maddz

That's quite common in my library - I often buy bundles, and quite often have a different version of the same title in my library - for example, earlier this year I purchased the Discworld bundle - all the Discworld books bar one in US editions. I had some already in UK editions (I'd been slowly buying the Night Watch titles)...

Or I'll have an eARC and also have a purchased copy.

I modify the title by adding the alternate locale to the title (I leave the series alone for this, so a series search allows me to pick the edition, if there)
Since UK is your home, I would *assume* UK was the default and add (US) to the title
stuff in ( ) does not mess the title_sort

theducks · 07-18-2026, 05:57 PM

BTW I also add (eARC) to a title if that is what it was, so I can also have the released version without triggering 'duplicate'

Sirtel · 07-18-2026, 07:11 PM

Personally I don't keep multiple different editions in my calibre library (US vs UK, for example); I pick one of them and store others elsewhere. There are a few exceptions, but only a handful (mostly self-published books that the author has modified).

As to eARCs, I don't do them; I want a proper retail edition.

DNSB · 07-18-2026, 07:39 PM

Am I the only one wondering where a 57GB library with multiple GB of duplicated book titles/authors was sourced from? It's good enough for training AI, it's good enough for the end users?

Sirtel · 07-18-2026, 08:03 PM

Quote:

Originally Posted by DNSB

Am I the only one wondering where a 57GB library with multiple GB of duplicated book titles/authors was sourced from?

No, you're not the only one. But I don't consider it my mission in life to judge other people, unless they do something really abusive. Copyright infringement or petty stealing (even if proven, which in this case they're not) aren't in the category "really abusive", IMNSHO.

Of course YMMV.

McStubb · Yesterday, 02:52 PM

When dealing with libraries from different sources/era's metadata management is key.

1) Find Duplicates: Binary file compares
2) Standardize Author/title metadata
** Make sure J R R and similar names get formatted the same. IE J.R.R.
** Make sure authors are in a standard First Last or Last, First format
3) Make sure the metadata is current (Find at least 1 ID and make sure all your books have that ID)
4) Find Duplicates: Search for ID duplicates
5) Find Duplicates: Search for author/title matches
6) Find Duplicates: Search for similar author/titles

07-18-2026, 03:06 PM	#1
Findingaway Enthusiast Posts: 35 Karma: 10 Join Date: Jun 2023 Device: Kobo Elipsa 2e, Galaxy Tab A	Big messy library, low skill - What's a good workflow? My ebook library folder has around 57 GB. Per AllDup, a freeware deduplicator, when searched by file name including extension, there are ~ 6 GB of duplicates. I've never successfully applied a sufficient organization scheme to the contents. They are split into multiple Calibre libraries. I also have Genre and other columns (like DCC / LCC from when the plugin worked) and a mygenre column. I attempted a hierarchical genre scheme without success. By "without success," I mean there must be error in how I tried it, because the results do not match Calibre documentation. Deciding on a scheme and partially implementing it has been useful though. I can use my library as it is and can dedicate some but not unlimited time to fixing it. But I would like to improve its functionality. The question is, with low coding and low Calibre expertise, what's a good way to tackle this? What I'm thinking of doing is - after getting the hang of Alldup - deduplicating the whole 6 GB and then rebuilding my Calibre libaries (for now plural), i.e. deleting and then replacing the library to get a correct database). I think some of my past attempts have failed due to biting off more than I could chew. I'll think "I just have to install x," and then x has 5 substeps, and y has 7 more substeps, it's all new to me and it doesn't get done. Thanks everyone for listening to my TED plea.

07-18-2026, 03:24 PM	#3
Sirtel Grand Sorcerer Posts: 15,461 Karma: 248000019 Join Date: Jan 2014 Location: Estonia Device: Kobo Sage & Libra 2	With a really messy library, there's unfortunately no other way to do this than manually. That is, going through it book by book, correcting metadata, deleting duplicates. With correct metadata you can use the Find Duplicates plugin to find and remove duplicates, but the plugin doesn't work when metadata is different between copies. So you'll have to fix the metadata first. I know it's a lot of work, as I have a huge calibre library myself. But there's just no automated way to do it and get correct results. Personally I don't bother with detailed or hierarchical genres - I just use a handful of broad genre categories and for more detail there's the synopsis in comments. Anything else would just be too much work when you have thousands of books. But YMMV, of course. Start going through your books a few a day. Take your time. Don't try to do it all in a couple of weeks or even months. There's no hurry. Last edited by Sirtel; 07-18-2026 at 03:45 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Proposal for an OpenClaw Skill: Automated Library Procurement	lpassey	Library Management	0	02-23-2026 06:20 PM
eBook/Comic/Manga Library Management Workflow: How do you manage your library?	Inukami	Library Management	16	08-02-2017 12:24 PM
What is a good workflow/process to read later web pages offline /w custom highlights?	avid01	Android Devices	2	11-09-2014 08:08 AM
What is a good workflow/process to read later web pages offline /w custom highlights?	avid01	Apple Devices	2	11-09-2014 05:02 AM
Second library very messy	Berni	Calibre	4	06-15-2012 08:20 AM

07-18-2026, 03:18 PM	#2
JSWolf Resident Curmudgeon Posts: 84,218 Karma: 153715495 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	Use the plugin Find Duplicates. Use it to find binary duplicates and then do a similar duplicate find. That's the best way to start cleaning your library. Don't use Alldup on your library. Don't use any external program on your library,

07-18-2026, 04:43 PM	#7
Sirtel Grand Sorcerer Posts: 15,461 Karma: 248000019 Join Date: Jan 2014 Location: Estonia Device: Kobo Sage & Libra 2	I have some books where I've bought a book, say, a decade ago and later bought a more recent version which was on sale or even temporarily free. Usually I remove the earlier version from my library and store it elsewhere, but I haven't gotten to all such books yet.

07-18-2026, 05:57 PM	#10
theducks Well trained by Cats Posts: 31,889 Karma: 64184592 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	BTW I also add (eARC) to a title if that is what it was, so I can also have the released version without triggering 'duplicate'

07-18-2026, 07:11 PM	#11
Sirtel Grand Sorcerer Posts: 15,461 Karma: 248000019 Join Date: Jan 2014 Location: Estonia Device: Kobo Sage & Libra 2	Personally I don't keep multiple different editions in my calibre library (US vs UK, for example); I pick one of them and store others elsewhere. There are a few exceptions, but only a handful (mostly self-published books that the author has modified). As to eARCs, I don't do them; I want a proper retail edition.

07-18-2026, 07:39 PM	#12
DNSB Bibliophagist Posts: 53,208 Karma: 181177243 Join Date: Jul 2010 Location: Vancouver Device: Kobo Sage, Libra Colour, Lenovo M8 FHD, Paperwhite 4, Tolino epos	Am I the only one wondering where a 57GB library with multiple GB of duplicated book titles/authors was sourced from? It's good enough for training AI, it's good enough for the end users?

Yesterday, 02:52 PM	#14
McStubb Connoisseur Posts: 58 Karma: 10 Join Date: May 2014 Device: None	When dealing with libraries from different sources/era's metadata management is key. 1) Find Duplicates: Binary file compares 2) Standardize Author/title metadata ** Make sure J R R and similar names get formatted the same. IE J.R.R. ** Make sure authors are in a standard First Last or Last, First format 3) Make sure the metadata is current (Find at least 1 ID and make sure all your books have that ID) 4) Find Duplicates: Search for ID duplicates 5) Find Duplicates: Search for author/title matches 6) Find Duplicates: Search for similar author/titles

Advert

Advert