Looking for suggestions on library management

[dedeaux]

Hey everyone!

I've been using and E-reader since a gifted Kindle back when it still had a keyboard. In the beginning I *only* bought and loaded books via Amazon and WiFi. As my children grew up and started reading E-Books on their devices and my wife got a Kindle as well, we found our E-book library growing.

I moved to Calibre and pulled all my purchases out of the Amazon cloud and into Calibre.

Later, I moved to a Kobo Aura H2O and absolutely love it over my Kindle. I still use this. My kids all have tablets and my wife moved over to iBooks and Kindle.app on ios after receiving an iPad.

That said... I now have a large collection of books and it is, in my opinion, a mess. I have a mix of purchased books, free books, and books that friends and family have shared with me. Regardless of origin, my collection is now what I feel to be too large. I have mixed fiction and non-fiction and many genres.

Calibre is reporting over 15K of books, which is likely due in part to my pulling in a large collection older theology texts.

I want to:
-- consolidate to ePub format only -- get rid of the rest (personal preference, open for arguments against converting certain formats...)
-- clean, fix, fill metadata for everything I can as some books are missing this
-- move non-fiction and fiction to separate libraries -- my family typically only uses the fiction section and I want them to access this themselves w/o having to rely on my laptop and Calibre installation
-- clean up duplicates -- I am finding dupes especially in my theology collection

So... Anyone have a suggested plan to accomplish this?

[MSWallack]

I'm short on time, so I can't answer all of your questions, but let me take address a few items, quickly.

1) I would probably convert everything to ePub, but there is probably no reason to delete other versions (unless you're short on space).

2) As to cleaning metadata, I'd suggest adding a custom column (I call mine "MetaFix"). The column should be a three-way Yes/No column. When you fix metadata for a book, change that column to Yes. This way, you can do searches for books that have metadata that needs to be cleaned (or have Calibre temporarily hide books that have already been cleaned). I also tend to sort by author and then title (or date) when cleaning metadata so that I can work with similar books in sequence.

Hope those ideas help a bit. Good luck.

[Joanna]

Use a custom column or a tag to clearly distinguish between the books that have already been 'cleaned' and the rest.

Proceed step by step, looking for books that have e.g. no date of publication, no author, multiple formats etc. You can search for them, use the tag browser or use the table view sorted according to various criteria (you can install the View Manager plugin for your regular and your 'edit' view).

Be patient... really patient .

Personally, I wouldn't remove all the formats but epub, as it doesn't save that much space - but that's just me.

[theducks]

Another 'Keep the Original' fan here.
You might just need it later.

Conversions do not always go as smooth as one wishes
Conversion of a conversion is usually a worse idea. (IMHO Edit the converted book to put the fine finish on it. Not convert it again)

to 3 state status columns (preferences: Behavior: Yes No columns ...

Yes means looked and it is good to go
No means you looked and it failed (the first go)
blank means you have not looked

[ekbell]

You will also want the Find Duplicates plugin. When doing the same sort of work I ran it at the beginning to find obvious duplicates and after adding metadata to find the rest of my duplicates.

Once everything is properly labelled it is actually quite simple to copy desired books into another library (select book(s) -right click -select copy to library).

[BookJunkieLI]

There is no reason to get rid of the originals, especially if you haven't had a chance to check the converted copy. As others have pointed out things don't always go smoothly, plus they don't really take up a lot of space.

A few plugins I highly recommend installing if you haven't already:
Find Duplicates
Manage Series
Count Pages
Goodreads Metadata
EpubSplit - if you have a lot of box sets/omnibuses and prefer individual books

Virtual Libraries are your best friend.

Everyone has their preferred process for going through and updating/fixing their library - this is the one I've been using to fix my 10k library. YMMV

Spoiler:

First thing - Visually scan through the books to make sure Titles are in the title field, Authors are in the author field, and that the Title is an actual title and not a random alphanumeric code.

Run Find Duplicates.
-I check the results first for duplicates records that are just different formats ie EPUB vs MOBI vs PDF etc. Those I merge into one record.
-The remaining duplicates I like to run Count Pages for both word and page counts. This helps to easily weed out book samples vs complete books.
-After that I start opening and comparing each set of duplicates checking for which book has the better formatting/table of contents/cover, etc.

Next is the long, boring part
I have a Yes/No column called Done, a Yes/No column called Fix, and a tag-like column called Original Tags. My "editing" tags are Round 1/2/3, Boxed Set - Split Up, Check Quality, Poor Quality, Poor Formatting, Needs TOC,Check Series, Update Cover, Update Tags, Complete.
I have a Virtual Library called Not Done where the setting is #done:false so if a book hasn't been marked Yes/No in the Done column it will appear in my library.

My personal problem children as far as Poor Quality/Formatting goes are usually books where the only format is one of the following: PDF, DOC, RTF, TXT. These are all automatically marked Fix:Yes and Done:No. A lot of these were acquired before the Major Publishers had started converting over their backlists to ebooks so by the time I get around to being ready to 'fix' them I may have purchased a proper copy and can just delete them.

Next I begin the process of downloading metadata. I get the best results with Goodreads, you might prefer a different source.
I try to download only 10 at a time. If I'm feeling particularly impatient I may go up to 25 at a time but I really try to limit myself as it's only polite to not inundate the online databases with requests.
Plus as many books as you download metadata for at a time you then need to verify that data before accepting. You should NEVER EVER blindly accept that the metadata downloaded is correct without verifying it first. Out of every 100 books close to a quarter of them are matched to the wrong book entirely, the wrong language edition, or the wrong cover. ALWAYS check your data before updating.
Once I've verified the data all I do the following:
Good Metadata Downloads:
-move tags to Original Tags
-clear tags and add Round 1/2/3, Check Quality and Boxed Set-Split Up(if applicable)
-mark Done column as No
Bad Metadata Downloads:
depending on my mood I either begin the process of figuring out why it didn't download/grabbed the wrong metadata or I mark the Fix column as Yes and ignore it for the time being.

I'll then run Find Duplicates again just to check.

Once Round 1 is complete everything with the tag Round 1 will have the Done column switched back to Undecided.
I then begin checking the book quality looking at formatting, text quality, if it has a table of contents, if it has a decent cover.
If it matches all my criteria I'll either clear Tags, put in my preferred book tags including Complete, and mark it Done:Yes or I'll change the Tags to Round 2, update the tags to what needs to be fixed and mark it Done:No.
If it doesn't match my preferred quality criteria I'll update the tags to indicate what needs to be fixed, add Round 2, then mark it Done:No.

Round 3 I'll do the easy ones like Update Cover or Add TOC first, then decide if I care enough about the book to fix formatting issues or see if I can find a better quality copy for a decent price.

Of course if there's a book I want to read immediately I'll bypass the entire process to fix that one right away and upload it to my reader.

As I said others have different processes but this is what currently works for me. I hope this at least gives you an idea of how you want to proceed.

You mentioned wanting to separate your Fiction and Non-Fiction libraries. I would probably wait until you have a batch of fixed/updated Fiction books completed before moving them into a separate library and then onto a computer other than your laptop. This way your family is only accessing the 'complete' books and not possibly interfering with your updating/processing method.

[theducks]

Add from List is another for those who document in the library, Series Gaps

[yonkyunior]

Maintaining large library requiring a lot of time..
i found the convention by the bootstraps would be great use for long time..
here
How I manage eBooks with Calibre by the bootstraps
https://www.mobileread.com/forums/sho...d.php?t=185452

read it slowly


i not yet using Quarantine&Scrub or CALM plugin, could be useful if you can understand how to use it,

[dedeaux]

Awesome! Thank you so much for the suggestions. I know I have a lot of work ahead, and your suggestions will help me start moving in the right direction.

[Paulie_D]

Originally Posted by theducks

Add from List is another for those who document in the library, Series Gaps

Could you point me at how that works...sounds useful.

[theducks]

Originally Posted by Paulie_D

Could you point me at how that works...sounds useful.

Is useful for things like a Wish list

Add From List PI (by Kiwidude )
Can 'match an empty book'.

(I use Fantastic Fiction for my match source)

I search FF on Author, and use that URL

It matches 3 books in a Series I read (shows on right)
I see 2 more books in that series on the Left that I do not have (an entry for in my Library). Right click: Match Empty Book <continue matching as needed>.: Next (finish up)


This will make a entry that MAY include the Author and Series along with the Title. Be sure you set any Unknown Authors (using bulk mode) before continuing. This does not populate covers or othe metadata. I suggest also doing this now

[Paulie_D]

Thanks so much...it's just that I can't find the PI.

Am I missing something blindingly obvious?

[theducks]

Originally Posted by Paulie_D

Thanks so much...it's just that I can't find the PI.

Am I missing something blindingly obvious?

Import List is the official name
(I happen to use it to Add)

[BetterRed]

Originally Posted by yonkyunior

Maintaining large library requiring a lot of time..
i found the convention by the bootstraps would be great use for long time..
here
How I manage eBooks with Calibre by the bootstraps

https://www.mobileread.com/forums/sho...d.php?t=185452

@yonkyunior - your link gives 404 error

==============================


@dedeaux - when I went though a similar exercise I used a process of moving a batch of 50 books from my 'messy' library into a 'workbench' library where I would fix the metadata, covers, do conversions etc. Once satisfied I moved the books from the 'workbench' library to a 'clean' library, on 2nd and subsequent batches before doing the move I resolved what ever the Find Duplicates (PI)->Find library duplicates function found.

Doing the above in situ gave me no sense of progress and it didn't encourage me to work methodically, whereas in doing the above I got to see the 'messy' library dwindling and the 'clean' library growing - i.e. more 'job satisfaction'.

Today I import books into the 'workbench' library where I do what ever needs doing with metadata, covers, conversion, editing etc before moving them to the 'clean' library. Most (> 90%) of my books are non commercial public domain texts from government agencies, quangos, ngos, etc, so I'm importing 10-20 new 'books' a day.

BR

[Paulie_D]

Originally Posted by theducks

Import List is the official name
(I happen to use it to Add)

Aha..