Taming My Wildly Disorganized Library Questions

[adolwyn]

Hello everyone,

I'm on day two of hyperfocusing on consuming all the content that's ever been produced on youtube, this forum, and the Calibre subreddit to try to help my brain figure out how to organize my wildly disorganized library. I've found lots of great ideas which have resulted in my brain flipping fully into paralysis mode and now I'm stuck.

I'm sure the answer really is: "just start somewhere," but if any of you have some time or ideas for how to start and a way to make this efficient (probably impossible) and at least orderly, I would appreciate your thoughts. I've read through this whole library management forum back at least 20 pages and have searched and haven't come up with anything that seems to help, but maybe I'm trying the wrong words.

The setup:

• I have a Calibre library with just over 2500 books in it. They're in various stages of disarray, from all over the internet, including my kindle purchases that I deDRMed, kobo, free books, PDFs, almost 20 years work of accumulation.
  
• I also have an "Ebooks" folder with many (but not all) of my accumulated ebooks in all different formats. Many are in my Calibre library, but plenty are not. There are also books in my Calibre library that are not in this ebooks folder. This folder has both straight book files and also books in folders.

Where I'm stuck:

1. Is there a good way to bulk add everything from my ebooks folder and Calibre can tell me if the book is already in it or not?
2. Can it add a folder with subfolders? I'd like to make sure I have added all my purchased books/free books/found books/etc into my library as it will drive my brain to distraction knowing there might be amazing books in that folder that I've just forgotten about.

Once all my books are in, the task of cleaning them up and organizing them must begin.

1. Should I begin by running quality check on everything first and fixing any formatting errors? I have the Quality Check plugin downloaded already.
2. Is there anything else I should do to make sure the books are all formatted properly?
3. Should I convert them all to epub first?
4. I have books in there that are PDFs, DOCs, TXTs, etc (as I said, I have some old ones). Should I just remove those and keep an eye open for updated ones if they go on sale somewhere?
5. Is there a 'best practices' for where to start fixing the metadata? Covers first? Download metadata? Page count plugin?
6. I have the following metadata sources installed: Amazon.com Multiple Countries, Barnes & Noble, Fantastic Fiction, Goodreads, Kindle hi-res covers, Kobo Metadata.

I will be loading the organized books to my Kobo Libra Colour.

1. Is there any prep that I should do to make the process as easy as possible?
2. I understand the KindleTouchExtended plugin is no longer needed in Calibre 8.5 (the version I'm on), but KoboUtilities I'm assuming is needed?
3. I tried to install some of the other recommended Kepub plugins and they aren't showing up, so I'm assuming those are no longer needed as well?

Is there anything I should know that might not be obvious? Are there any search terms that I might not be using to do more reading of the great info on these forums?
What is your favourite tip or trick or neat hack or custom column or anything else that gives your brain dopamine?

Thank you all for reading and I appreciate any time/help/answers/suggestions you may have the time to pass along to me. In the meantime, I'll keep working to educate myself as I dig into all the posts in here.

[adolwyn]

Ok, follow up post here for anyone inclined to read a few words. I've patched together a bit of a workflow from all the resources in here and was hoping someone more experienced in Calibre could look at it and let me know if I have any obvious/helpful things that I've missed and should be doing, or if I'm duplicating or doing anything that isn't needed or could cause problems down the road?

If it's good/reasonable, then maybe it can serve as help to someone down the road who shows up feeling as overwhelmed as I was. haha.

Part Zero: Importing

• Select ten books (just a number that worked for me) from my old, messy Calibre library and copy it to my Calibre wookbench library (deleting from the old library once copied over).
  
• For new books not already in my Calibre library, then I would add them to my workbench (or just my 'clean' Library down the road), and if they aren't already ePub, I would convert them to ePub.

Part One: Checking & Quality Assurance

1. Open each book individually in book editor.

• Check that each book is what I think it should be based on the filename, etc.
• Run the "Check Book" job in the book editor to see if there are errors and fix any that book editor can fix.
• Save each book and close back to the regular Calibre library.

2. Run Quality Check on all ten books at once.
- Quality check items I am using are:

• Missing Container
• Invalid namespaces
• Non-dc: metadata
• Manifest files missing
• Unmanifested files
• Unused CSS files
• Unused image files
• Broken image links
• TOC with < 3 entries
• TOC with broken links
• (not checking hierarchical TOC/flattening as I read somewhere that it was no longer needed?)

3. Run Modify ePub for the books with problems that can be fixed by Modify ePub, making sure the following are checked off:

• Remove unused image files
• Smarten punctuation
• Remove broken image pages
• Remove Adobe resource DRM meta tags
• Update metadata
• Remove non dc: metadata elements
• Remove all metadata jackets (so I can add a fresh one with Polish later on)

Part Two: Individual Books, Metadata, & Covers

1. Bulk remove all tags, ratings, and identifiers from all ten 'workbench' books.
2. Manually add the correct edition isbn or asin based on what's in the book or what's available on Goodreads to each book's identifier field.
3. (sidenote: I tried to use the extract ISBN plugin and it wasn't able to find any ISBN numbers in the 20 books I've done this with so far so I've gone back to manual adding the right ISBN (preferred) or ASIN (if no ISBN is available).
4. In the "Edit Metadata Individually" section, click "download metadata" and get the metadata correct, select a good, higher resolution cover, check that series data is correct.
5. Update the tags with the silly ones I don't want (I wish I could tell Goodreads I never want the audiobook tag ever in a hundred million years), update my Main Genre custom column tag, and my Shelves custom column (used for syncing to Kobo's collections).
6. Ensure each book has a goodreads identifier as part of the metadata download process. If it doesn't, then manually add it for the correct edition.
7. If non-fiction, remove the long subtitle from the title field and put it in the custom subtitle column.
8. Make sure the comments box has a usable summary without any promo/advertising.

Part Three: Finishing
1. Select all ten workbench books, select the "Count Pages" (estimate page/word counts option) - using the ADE estimation as I'm working with ePub and eventually kePub.

2. Move on to Polish books.

• Make sure there's no "original epub" in the book files by going to Remove Books > Remove files of a specific format from selected books.

- In polish, select:

• Smarten punctuation
• Update metadata in the book files
• Update the cover in the book files
• Add/replace metadata as a book jacked page
• Remove unused CSS rules from the book
• Losslessly compress images
• Download external impages
• Upgrade book internals

- Polish the ten books.

3. Make sure all books open and look ok after polishing. If so, remove the original epub that is created.

4. If everything is good, select all ten workbench books and move them to the "clean" Library (deleting from the workbench after copy).

----

Other setup items:
1. I have KoboUtilities set up with the following save template:

{author_sort:.1}/{author}/{series:||/}{title} - {series}{series_index:0>2s| [|] - }{author}

I have the driver set up as the instructions show to have the subtitle show main genre and # of pages and it seems to be working to upload to Kobo and to download the Kobo information into those custom columns.

2. I'll eventually also finish setting up Goodreads sync, but I have at least connected my goodreads account with the plugin.

----

So, is this a reasonable workflow for someone whose goal is to have accurate metadata, a well-formatted eBook to send to my Kobo, and a library with good organization, tagging, and series info?

Have I missed anything? Am I duplicating my efforts anywhere? Should I be doing anything else?

Thank you to anyone who read all that and is willing to provide any feedback, ideas, suggestions, or improvements!

[Sirtel]

The best way to get a well-formatted ebook is to edit it manually with Calibre editor or Sigil. All the bulk operations with Quality Check, Modify Epub, Polish and so on only go so far; they don't really fix broken formatting.

That said, few folks can be bothered to edit all their books manually. Most just manage with what they have.

As to whether your workflow is a good one, it's good if it does what you want. Everyone has different goals and does things in a different way. Personally I don't do all this bulk checking and fixing - I don't need most of that, I fix the formatting manually and fill the metadata in manually too. It's just faster for me this way. I do use Modify Epub for a couple of bulk changes, but not for fixing bad formatting.

[theducks]

Originally Posted by Sirtel

The best way to get a well-formatted ebook is to edit it manually with Calibre editor or Sigil. All the bulk operations with Quality Check, Modify Epub, Polish and so on only go so far; they don't really fix broken formatting.

That said, few folks can be bothered to edit all their books manually. Most just manage with what they have.

As to whether your workflow is a good one, it's good if it does what you want. Everyone has different goals and does things in a different way. Personally I don't do all this bulk checking and fixing - I don't need most of that, I fix the formatting manually and fill the metadata in manually too. It's just faster for me this way. I do use Modify Epub for a couple of bulk changes, but not for fixing bad formatting.

↑ ↑ ↑ ✔ ↑ ↑ ↑ ✔ ↑ ↑ ↑ ✔
Never trust Otto
IMHO Repeated metadata downloads rarely improves existing data.
When I have skimpy metadata, that inclueds stuff I added, I use the Ctrl-D get metadata and make liberal use of the arrows to copy existing to the left.
Note: you can hand edit the left side

[adolwyn]

Thanks to both of you! It sounds like I'm mostly doing what you two also recommend - from opening the book in the Calibre book editor to fix what I can in there, to not re-downloading metadata a bunch of times.

Appreciate the thoughts!

[Sirtel]

By fixing the formatting in the editor I meant editing the css/html. Checking for errors does help, but like all other automatic solutions it doesn't fix bad formatting. If your aim is to have all your books well formatted, that can only be done by editing the code manually.

[fogice]

One thing I do in the Editor is to selectively delete front and back matter. I've seen books that are 10 times the size they should be because the back matter included full-size cover art for all the other books by that author. I also remove things like "sign up for the newsletter".

[Sirtel]

Originally Posted by fogice

One thing I do in the Editor is to selectively delete front and back matter. I've seen books that are 10 times the size they should be because the back matter included full-size cover art for all the other books by that author. I also remove things like "sign up for the newsletter".

Yes, I do this too. And all the excerpts from other books, because they screw up the overall length.

[theducks]

I get rid of the body of 'sample chapters' (I leave the 'coming soon') sort of to match what I stopped doing with the Paper editions as I would later discover that I had only Previewed, and not read a book I was following.
I also shrink covers. I have no need of a HiRes cover on my e-ink device. 450 is my maximum width (I leave Maps and charts full size, but may rotate the for best aspect fit).

[Sirtel]

Originally Posted by theducks

I also shrink covers. I have no need of a HiRes cover on my e-ink device. 450 is my maximum width (I leave Maps and charts full size, but may rotate the for best aspect fit).

I don't shrink covers. Both my Kobos have 32GB and I don't keep all my ebooks on my ereaders, so I don't have even a third of the space on them filled.

I hate seeing low-res covers on my computer with its 4K screen. Things were different 15 years ago, but these days I want my covers to be high res.

[adolwyn]

Originally Posted by Sirtel

By fixing the formatting in the editor I meant editing the css/html. Checking for errors does help, but like all other automatic solutions it doesn't fix bad formatting. If your aim is to have all your books well formatted, that can only be done by editing the code manually.

Ahh, ok! Thanks for clarifying! That might have to be stage two once I have my library at least well tagged/organized/and with glaring errors done. I haven't found a good tutorial that explains what a well-formatted ebook looks like/how to achieve it yet, so good to know there's more to learn (there's always more to learn). Appreciate the info!

[Sirtel]

There's no One Rule to Rule Them All as to what a well formatted book looks like. It depends on individual preferences.

My personal adjustments are as follows:

- remove any embedded font for ordinary paragraphs (I leave embedded fonts for specific sections, like chapter headings, letters, text messages and so on)

- delete any size for body font other than 1em

- remove any specified line height for ordinary paragraphs

- delete spaces between ordinary paragraphs if there are any (not meaning section breaks and such. OTOH, there are folks who want spaces between paragraphs and insert them if missing)

- change text indents to 1em (I can't stand larger indents)

- delete margins for body text if there are any

- fix margins for blockquotes and special sections if needed

- reduce huge white spaces above and below chapter headings (those seem to be all the rage nowadays, but as they're a total waste of screen real estate, I get rid of them)

- justify text (I dislike left-aligned text on my ereaders)

- other mostly cosmetical adjustments if needed

- fix TOC if needed

[theducks]

↑ ↑ ↑ ✔ (mostly I have MY fav styling. Some were because my first reader had Adobe block numbers on the right so I made margin-right: 16px)

I start with the Debug in the Calibre editor. There are some 'errors' I ignore (eg Webkit, more than 1 CSS occurrence (I do check, but that is really legal)

Syntax errors are always to be fixed as some devices ignore a CSS with those. Practice
Books by Vellum all PASS, but have those huge chapter headings (i have saved searches to find and reduce (or comment out the 12%Minimum)
I comment out, rather than totally hide my CSS adjustments YMMV

It will take some time to get fast at fixing or just plain touchup. Saved searches are handy. Some fixes take me 30 seconds to mash it my way. Some take because the selfpub had no clue what a clean coded book can look like (eg seperate (and different) stylesheet per chapter)

[adolwyn]

Thanks to both of you! These are great tips/suggestions. Seems like I'm going to have to brush up on my CSS knowledge from the Geocities era to dive into ebook editing.

[theducks]

Originally Posted by adolwyn

Thanks to both of you! These are great tips/suggestions. Seems like I'm going to have to brush up on my CSS knowledge from the Geocities era to dive into ebook editing.

EPUB2 CSS is a (fairly small) subset of what is used on WEB pages.
Usually you will be dealing with stuff within the Body tags The editor has a powerful S&R, The book browser has built in tools to link stylesheets and there are many 3rd party plugins for specific tasks.