Setting-up a big library - where to start?

[greenrun]

Hello all ... I'm relatively new to the forum and to Calibre, yet I've inherited the task of setting up a Calibre library for a fairly large collection of eBooks. We have a collection of several thousand eBooks in all different formats on a wide range of topics within the company. Can someone suggest three or four key posts that I should read to help build a good foundation for our library files?

A couple of key questions:

1) Do I organize the files first, then add to Calibre or just add them and then clean-up organization and metadata?

2) How crucial is the file structure of the eBooks prior to adding them to Calibre? Is it difficult to change the location and hierarchical position of a book once in Calibre?

Thanks in advance for any insights you can share.

Cheers, George

[eschwartz]

If the books in question have valid metadata, simply import them all and then do cleanup in calibre.

If the books do not have good metadata (PDF is notorious in this regard) you will probably want to use a Regular Expression to control adding the books, this will extract metadata from the filename.
Preferences ==> Import/Export ==> Adding books
Uncheck "Read metadata from file contents rather than file name"

[greenrun]

Originally Posted by eschwartz

If the books in question have valid metadata, simply import them all and then do cleanup in calibre.

If the books do not have good metadata (PDF is notorious in this regard) you will probably want to use a Regular Expression to control adding the books, this will extract metadata from the filename.
Preferences ==> Import/Export ==> Adding books
Uncheck "Read metadata from file contents rather than file name"

So, should the first priority be to separate all those with "valid" metadata from those without? Then batch process those with metadata and add those without, individually?

[theducks]

I prefer to use the term 'Process by Lot' if you are using the GUI

Calibre has a CLI and a number of folk do OS level 'Batch Processing'


For the stuff with Good metadata. Process by big lots is probably OK

'Good' is a vague term
Series info is NOT a DC standard and is missing from a lot of books .
You might want to do those each 'by lot'


BTW

Manage Series PI can be a great help tool in assigning order (best with a contiguous series group)

Search the Internet PI to help resove all sorts of metadata issues that the Basic 'Get Metadata' can't resolve.

'Patience Grasshopper'
Nibble away at your backlog (those not in the Library). Clean a bit as you go rather than face 5K of OMG, I still need to fix this

[eschwartz]

Originally Posted by greenrun

So, should the first priority be to separate all those with "valid" metadata from those without? Then batch process those with metadata and add those without, individually?

Correct.

[eschwartz]

Part two of my first post (I got distracted while editing. ):

The structure of the ebooks prior to adding doesn't really matter. The metadata should be getting pulled from the embedded metadata fields, but if you have to rely on the filename, the full path won't get used anyway. There are utilities for bulk renaming files, which could consolidate hierarchical folder metadata into the filename, I believe. (If you need it.)

Once in calibre, you have absolutely no control over the folder structure (see Worldwalker's informative post on the matter. )
You can, however add tags as well as arbitrary custom metadata types (for example, "genre" is popular).
Under Preferences ==> Interface ==> Look & Feel ==> Tag Browser (tab) you can set "Categories with hierarchical items" which will show as trees based on the period separator.

You can export books from calibre using metadata templates (see Preferences ==> Import/Export ==> Saving books to disk) including various transforming functions, allowing you to save books outside of calibre's black box library. You have tremendous flexibility, and all you need to do is update the metadata associated with various book records.

[BetterRed]

@greenrun - above replies - and

Identify a representative sample of the books, with & without metadata, the different formats you have etc, and get them into a library and have a play around with it, be prepared to throw it away and start over;

First things I would get right for each book is the Author, Title and ISBN. Then decide (and play with) things like Author handling, particularly sorting, Initials, Tag browser layout etc and set the appropriate options. You can change options later, but if you take a bit of time exploring what's available now, then you will have an appreciation of what's possible when you want to change something.

Also consider having a separate library into which you do the initial importing, there you can get the basic metadata correctly set up, and then move (right click Copy to Library) to the good books library

There are lot of features in the right click menu's, you wont remember what they all are - but you'll often find that what you want is under your right mouse button

Another issue to consider is how to detect and handle duplicate books - importing into a separate library first will give you of the possibility if using the Find Duplicates plug in feature of comparing books in two different libraries.

As well as the base product there are a lot of very useful Plug-Ins - when looking for a feature first look in the calibre User Manual next look in Index of plugins

I would import the books in batches and get each batch into shape before starting on another batch - the size of the batch will depend on what sort of 'mess' I was starting from and the time I could allot to the task.

One the first things to do is to establish a backup regime for the library(ies) you create and make sure you know how to recover a library from the backup. I suggest an incremental mirror backup - Free File Synch is popular with many calibre users. I backup at least once a day. I've never had to restore more than a single book - and the need to do so was invariably and entirely my own fault

BR

[toomuchreading]

Welcome to the world of Calibre. All of the above, are very valid, especially the backup, and loading in batches and cleaning as you go.
I'm doing pretty much the same thing you are, and I am loading my books in batches. Once a batch is loaded, I then individually go through each book to check and verify that the metadata is OK. Some books are better than others. You will get a feel for those you can skim over, and those you have to go so far as to open up the book to make sure you even have a right title.
As "theducks" said, 'patience grasshopper'. It's far better to nibble away at a smaller amount of books at a time than the '5K OMG' situation. Ask me how I know...
Take your time, in manageable batches, and make sure you have a backup of your library, and the books you have not loaded yet.
Once you've loaded a book into Calibre, and have the metadata correct, you can use the GUI to do extensive searches, so you don't have to worry about the Calibre library, other than to back it up. As you get familiar with Calibre, you will learn that the GUI is everything, and treat the data library as a 'black box'.
As you run into problems, and I'm sure you will, search the forums. There is a lot of information here, and if you can't find what you need, drop us a line, and someone will be sure to lend a hand.

[kacir]

Originally Posted by greenrun

Hello all ... I'm relatively new to the forum and to Calibre, yet I've inherited the task of setting up a Calibre library for a fairly large collection of eBooks. We have a collection of several thousand eBooks in all different formats on a wide range of topics within the company. Can someone suggest three or four key posts that I should read to help build a good foundation for our library files?

A couple of key questions:

1) Do I organize the files first, then add to Calibre or just add them and then clean-up organization and metadata?

2) How crucial is the file structure of the eBooks prior to adding them to Calibre? Is it difficult to change the location and hierarchical position of a book once in Calibre?

It depends.
Just start adding the books to the library. You can add columns and configure things later.
Much depends on how good the metadata associated with the book files are. If you are lucky and you have most of books with a good metadata, just drag & drop them into Calibre.
You are extremely lucky, because adding of large numbers of books to a large library just (five days ago as of writing this post) got significantly faster in the latest version of Calibre - 2.10.
There is new version of Calibre available every Friday. It often brings significant improvements - like last one did.

If your metadata is not so good, it depends on how the book files themselves are named. If you have them named following way: AuthorName AuthorSurname - series # - title you just install plugin "Quick preferences" and use built in "patterns" - regular expressions - to recognize book name and authors. Do explore pattern, or template or regular expression for adding books in preferences. "Quick preferences" is just a quick way to switch between several options when adding books so you do not have to dive deep into preferences dialog panel. At this moment it is broken a little bit - the first book you add after changing settings is added with the old settings.

If your books are not so conveniently named, you might have them in directories that contain author name and other valuable info. If that is the case, get a listing of file names including complete path and ask some Unix administrator or DOS commandline guru to help you to write a [shell] script to copy books to a new filenames that contain all the info you can process them using template for adding books described above.

Learn what Regular expressions are, because they are an extremely powerful tool for importing books, searching library, changing 'surname, name' author name to 'name surname' and lots of other things. You do not want to rename 500 authors manually, do you?
The best book to learn about Regular expressions is "Mastering regular expressions". Remember, you do not have to be a master to do some powerful things. Calibre is written in Python, so you are interested in learning Python type of RE.

Bear in mind that you can search for books using regular expressions. So it you write in the search bar Authors:"=Stephen King" you get books with EXACT match. When you type Authors:"King" you get also manes like Hawking - so you are looking for a substring. You can also search like this: Authors:"~King" where word King in this example represents Regular expressions. Look up this in Calibre Documentation on the net - you have to escape backslash [you might have no idea what I am talking about - read about Regular expressions first". There is wonderful thread about Rebular expressions in Calibre here, I will try to find it later.]

Get plugin for downloading metadata. You import a book, with just author and name and then hit <Ctrl>D and Calibre will find and download other metadata for the book - this works extremely well for fiction and is well worth the try for your books.

Get plugin for finding duplicates - this will help you locate many, many duplicates in your library much more quickly than manual search.

Read the Calibre manual thoroughly.

Read monster thread about Calibre written by Unbogling. (I will try to find that one later as well and post link).

Get an SSD for your library and operating system, if possible. You are going to do this at work and saved minutes when doing large changes to your libraries will quickly add up to pay 400 Eur you pay for a really fancy 1TB Crucial M550 many times over. A modest 120GB SSD to start with is just about 60Eur and there is absolutely no reason not to use at least that in professional environment.

BACKUP your library often - this is where your SSD will really shine once you have tens of thousands of books. Copying many thousands of files scattered all over disk anywhere - even to a network storage - is an extremely I/O intensive operation and this is where SSD is the king. Backup your library especially before bulk renaming authors using regular expressions and other large changes. Do not overwrite previous backups, it might turn out that some bulk change you did a week ago had some nasty side-effect.

Calibre will not touch your original files, it will create its own file structure. Treat is as a black box and access your books using only Calibre. The file structure comes into play only when a disaster strikes and you need to recover your books manually or do something desperate. There are *many* threads discussing this.

Learn how to start Calibre server from within Calibre - so your coworkers can access the library using just a web browser (it has to be able to see the IP address of your Calibre server - on the intranet or fixed IP on the net.)

Have a look at CalibreCompanion app for Android - this is great way of how to access library form a tablet or a phone.

Consider persuading your company to donate some money for the development of Calibre. The author of Calibre Kovid Goyal works on Calibre full time and improves [and supports] the program at an incredible rate. I have seen many great and/or expensive pieces of software, and Calibre is perhaps the best one regarding rate of improvement, user support, friendliness, promptness of fixing bugs or adding requested features. Plus, its powerful features will save you many, MANY man/hours.

[kacir]

Here are some links I promised in previous post:
https://www.mobileread.com/forums/sho...d.php?t=185452

https://www.mobileread.com/forums/sho...d.php?t=119175

https://www.mobileread.com/forums/sho...d.php?t=145808

https://www.mobileread.com/forums/sho...d.php?t=118569

[kite]

All of the above is good.
If your books have ISBN numbers then that alone is (IF CORRECT) all you (USUALLY) need as the other metadata can be downloaded automatically from the ISBN.
There is a Calibre plugin that automatically searches text ( but not image) based PDFs (?and other text formats) for an ISBN. It does a good job however it can be misled by text referring to another book's isbn.
You may need to consider how critical correct metadata is for the library. For example I wouldn't have faith in my library a metadata in a life or death situation.

If after using all the automated processes you may find that renaming individual files to reflect their author - title by keyboard is quicker before adding them to Calibre. Bulk Rename Utility and others will let you rename using a list of "old file name|new file name" pairs.

I'd interested to hear how you go and what work flow you find for your task.

[LadyKate]

My personal method of adding large numbers of books is to clean up the author so they are all in the same form (for instance all are as firstname lastname not lastname, firstname). I then check the language and make sure that the title is actually the title and not the author name.

Once I get to this point I start checking the books and doing conversions if necessary. I use a custom column to note the quality of the book and if additional work is needed on them. (technical papers in pdf format tend to need work to display well).

I would also recommend that you decide on what metadata tags you want to use. Most books have tags like you would see on a website... in otherwords, everthing but the kitchen sink. If you don't start at the beginning you could have Non Fiction, Non-fiction, Nonfiction, Nonfiction History, Nonfiction Civil War etc. This can become very cumbersome and not very helpful when looking for a particular genre. You can always add more tags as you refine your library but putting thought into it at the beginning will be less work in the long run.

For the next batch of publications to add, I would recommend a temporary library where you would do the initial cleanup of name and title prior to running a duplicate check using the duplicates plugin (library option). After removing duplicate or marking them by using a custom column that exists in both the main library and the temporary addition library.

I would also recommend frequently running the duplicate check plugin. If there are books with duplicate titles and authors that are different editions you can mark them as "to ignore" in the duplicate check plugin if you want to keep both versions. Don't hesitate to use the "Comments" or add a "Notes" column for information you need on the books.

Good luck and don't get discouraged if it seems like a lot of work. The most important data to have correct are the Name, Title and ISBN as others have said. Get the names formatted right and the title and if the tags imported are garbage, clear those. Also double check the language. I have found books with two words of french in them labeled as French by mistake lol.