Understanding Calibre Features

[Rellwood]

Let me preface this with the statement that I am on "the Spectrum" so it explains why this post is written the way it was, which is much shorter than I really wanted.

I know there is a better way to do things in Calibre than what I am doing, but no matter how much I try to read the Calibre help pages, there are still a lot of features that are lost on me.

I have to know, is there a book, guide, or manual that can be suggested that will help me to get a better understanding of how to make Calibre work better for me? I know there are a lot of features that I just don't know how they work, or how I can make them work better.

The goal I have in mind is to create a single perfect library, and save it so that later when I get that urge to "make things better" I don't spend a month trying to rebuild it, using books rescued from the trash or DropBox.

There are many features in Calibre that I still haven't figured out, and so haven't used.

Save to Disk- what exactly is this, and why does it seem that everyone is creating plugboards to use it? Is this just a way of getting the books into a reading device? Or is it a way to save the library and metadata so that later when I get it in my head to "make it better" I am not left weeding through thousands of files in my recycle bin.

Automerge- I really, really need to understand this better. I read that the choices of overwriting, ignoring, and creating a new record indicate the "format" of the book, but what about the metadata? What do I need to do to blend books together to get the metadata to merge? Has anyone come up with a plug-in that allows the users to choose per column what happens? If not, then this would be an excellent feature. If some formats of columns won't blend, then which order do I copy them to get the metadata of a specific one to populate, or leave alone. This will help me to weed down the library with 23,000 books that are basically various copies of 1800 books.

Finally, am I mistaken or did the newest version have a feature that allows copying or importing libraries to be done with a hard link, thereby making it quicker? If this is true, then were do I find out how to use it. I have spent hours waiting on my libraries to copy or import.

Ok, I know that was long, but really, I needed it to be out there.

Also, please do not just point me in the direction of the online users manual. I have read it many times over, some of it was helpful, and a lot of it was still lost on me. I don't want someone to babysit me, but if there are some facepalm things I need to know it would be appreciated.

Thanks in advance...

[theducks]

I have no idea what 'the spectrum' is.

Calibre IS the Library. Self contained. Metadata and books.

There is no need for Save to Disk. That is just a choice when you go to EXPORT a book to a unsupported device.

Any linking is to enable coupling additional unsupported items into the Library.

There is no official Manual other than the users guide. MR members have come up with creative solutions and creative adaptations Those are documented in these forum

[dickloraine]

Do you delete your whole library and restore it then from the bin to make changes? I don't understand what you mean by needing to rebuild the library. Why would you do that?

There is a plugin for finding duplicates. And I think more of aproblem than merging the metadata would be to find the best version of the book.

[Rellwood]

@theducks...Thank you for clarifying that the "save to disk" is another way of saying "send to device" and it is for devices not already calibrated or recognized by the library. Saving to disk doesn't mean saving to another area on my computer as a back up. I already knew that Calibre is a filing system and is self contained. I just thought that saving to disk was a way of backing it up. Thank you.

I understand about there not being a manual, I have been on the forums looking for ways to better understand. I have make my own workarounds, but I am certain that I am just missing things that would make my life easier. I was just wondering if one of the users has made a book or guide.

@dickloraine....I usually have anywhere between 5 and 10 working libraries. I have very few responsibilities beyond working on my MAS in Counseling so Calibre is my hobby, reading books is my hobby. I often spend more time messing with Calibre than reading.

When I get my working libraries to where I like them, I inevitably find a reason to do something drastic thinking that I am making things better. I might decide to delete all my custom columns thinking that I don't really need all that information, and it would be nice to not have to fill it in for all my books. I may decide to blend all my libraries into a single one and when I go to "copy to library" I realize I have forgotten that the settings for auto merge have basically ruined the library I am sending to. Whatever reasons I manage to come up with, I have had to "rebuild" my library from both restoring the database, and hunting and gathering from various sources books and metadata.

Regarding "the spectrum" it is a relatively common term used to indicate that a person falls on the Autistic Spectrum. There are a lot of different autistic problems, and mine are more communication and comprehension. I communicate differently (or rather oddly) and I comprehend things differently than non-autistics.

Anyway, thanks for the comments and i will keep hunting for more and better ways to screw up my libraries

[BetterRed]

@rellwood - Firstly calibre has two main export mechanisms - Send to Device and Save to Disk.

As I understand them Plugboards are a mechanism for mapping the metadata columns (fields) in the calibre database to the metadata fields within an ebook format. So lets say I have a custom column called Status. I know for certain that none of the formats or devices out there will know about my Status column. So via a plugboard I could plug the value of my Status column into the Title in the ebook metadata as a prefix.

I can't address the Automerge issue, I don't use it.

I gave up on trying to de-dup my initial library in-situ. I did this: I processed books in smallish batches (about 50 I think). I added 50 books to a "Workbench" library and focuss[sic]ed on getting the author, title and ISBN correct and consistent. I deduped that set of books with the Find Duplicates plugin, then moved what was left to a "Good" library. For the second and subsequent batches after deduping the batch, I used the Find Duplicates->Library Duplicates feature (checks for duplicates across two libraries) to weed out inter library duplicates. I dumped duplicates into a "Bad" library... rinse and repeat....

My plan was to use the "Bad" library as a source of 'missing' metadata, I soon discovered that it was far easier to download metadata from Amazon, Goodreads etc and/or roll my own - especially with tags.

Calibre does use hardlinks for reasons of performance when it makes sense to do so - in move and rename operations. But they are transient, ie when calibre has done what it needs to do there is only one link. So users are not aware of them.

The word 'link' is also used within the calibre forum in the context of linking calibre authors and books to supplementary data about the author or book via a URI.

For authors there is a specific feature, the Author Link property which is accessible via Manage Authors. For books you can insert <a href= style links into long text columns (including Comments), you could put your URI into the Identifiers column, I found this awkward so I prefer the long text column method.

And Author Link or a link in a the long text column can contain any valid URI - I know these work, Web urls (http(s)://...), FTP urls (ftp(s)://...), FILE uri's (file:///...), Evernote uris (evernote:///...) and OneNote uris (onenote:///...)

BR

[Rellwood]

@BetterRed...Thank you. I will now check out the Author Links. I'm still a bit unclear about the linking between libraries. I guess my main question would be, is there a way to copy to library, or an import go faster using a feature that has until now not been available?

As far as de-duping. I wish that was my only problem. Somehow -I don't know if it happened when I linked my two laptops onto a family network, and attempted to bring old Calibre info from one into the other -but I ended up with a library that "lost" every download from March to January. EVERY ONE. This was AFTER I screwed up my libraries. The reason why I was attempting to gather the books from the old computer. It has the original dates for the books.

I was in a situation where I was attempting to restore a few libraries (or at least have a single one with all the books and meta) and went hunting for untouched books in an old computer, and dragged God knows how many different versions of the same book and threw them into one folder. I went through the trash, through the Drop Box, through The Amazon Cloud, all my downloads, etc. I amassed 23,000 books. This is my official "bad" library. I know I don't have 23,000 books, so this is where the auto-merge would come in handy. I find that having a mess of "backups" just makes things more difficult. The KISS method really should be applied here.

Thanks for putting it in terms I understand.

[BetterRed]

Originally Posted by Rellwood

@BetterRed...Thank you. I will now check out the Author Links. I'm still a bit unclear about the linking between libraries.

There is no out-of-the-box way to link books to one another within or between libraries. They can be of course, be associated within a library, by virtue of having common properties (column values) - authors, tags, series, publisher etc.

The Reading List plugin can be used to create arbitrary lists of books, and a book can be in multiple reading lists - but I'm not sure I would want to use it to create permanent arbitrary links between books. I might use it to create of list of cookbooks having recipes for Rabbit Pie that I could peruse at leisure. But once the rabbit was caught, the pie baked and eaten then I'd probably remove the Rabbit Pie Reading List

But let's say I have Madame Bovary by Gustave Flaubert in both French and English.

A book has a Language metadata column, so in this instance they will be different, so I need two book records.

If I order the book list by author and title (or title and author) then the two editions will be adjacent (or close to) thus it should be apparent to me that I have the two editions. But if I order the book list by publisher and acquisition date then that wont be true.

This is where file links can be used. I have a long text column called Notes. So into Notes for the French edition I would insert a link to the folder for the English edition labelled "Language Edition - English", and into Notes for the English edition would I do the reverse. Additionally I have a column icon rule that displays a chain icon in the book list if a Note exists containing "Language Edition". The icon is merely an indicator, I look to the Book Details to see why its there. If I were to click on the "Language Edition - English" link in book details, that folder will open up.

Or, da-da-de-dah... I could tab to the Author cell and tap 'Q' and a Quick View window would 'popup' showing all the books I have that were written by Flaubert in all languages. See Quickview

Originally Posted by Rellwood

I guess my main question would be, is there a way to copy to library, or an import go faster using a feature that has until now not been available?

When you copy or move books or libraries it will be relatively slow no matter how you do it, because every bit and byte has to be read from one location and written to another location; and with a move the book or library must deleted from the source location. How slow will depend what sort of devices, and if there are two devices (source and destination) how they are connected etc.

The exception is with a move, if the source and destination are on the same device, and the file system on the device supports hard-links (and maybe some other factors) then calibre will make use of hard-links to effect the move, which should be considerably faster because the contents are not read, written, and deleted. Only the file system indexes are changed, and they are optimised for speed.

A recent enhancement to calibre is a feature targeted at making it easier to Export one's calibre data (libraries and configuration) from an 'old' computer and then Import that data to a fresh calibre install on a 'new' computer.

One suggestion - if you're using Windows, install a copy of calibre portable as a testbed, learning vehicle. Use it to explore calibre features and experiment with ideas before applying the lessons learnt to your real library. And be prepared to throw it away if its gets broken.

One other thing. IMO it's best not to view the Trash Can as a part of your backup scheme, treat it as an 'oops' bucket - ie 'oops' I didn't mean to do that, lets get it back. This is especially true for calibre - if you remove a book from a library you can't restore it directly from the Trash Can, you have to restore it to 'working storage' and add it back into the library

I'm going to leave it at that for now.

BR

Working Storage ??? Where did that pop up from, I did write a few COBOL programs in the 60's for Caterpillar - possibly the worst job I ever had.

[Sabardeyn]

Originally Posted by Rellwood

The goal I have in mind is to create a single perfect library, and save it

We all want the perfect library. But I'm not sure that anyone has actually managed it. The "perfect library" is a transitory goal based on your current usage of calibre and your preferred reading material, so what you need/want/expect from that "perfect library" changes as your taste changes.

The closest you might ever come to a universally approved "perfect library" is if each and every user were to submit a sample database, with a detailed workflow write-up along with edge-case explanations and various reports, displays and other expected output. Merge the whole shebang into a unified whole - and hope that calibre doesn't get a major new feature that week!

[theducks]

Just to clear things up.

Save to disk CAN save to your computer disk. But the (any Calibre) LIBRARY should never be the target.

Copy TO Library (on context menu) is an approved way of moving (or copy) a selection to another Calibre Library on (connected drive) the same system

I have an 'Intake Library' where I import books... Beat the metadata into submission ... Only then, do I Copy to Library 'Main"

I don't trust full AUTO anything. Auto Merge only allows Merging if the format is not duplicate. (My wish list is an Empty Book in 'Main' ). All other merges are either Manual or a cut and paste (typically comments. The other field values are usually available from the lookup pulldown)

Combining books is NOT automatic. For that, there is EPUBMerge (and lots of user carefulness)

[eschwartz]

Originally Posted by Rellwood

Save to Disk- what exactly is this, and why does it seem that everyone is creating plugboards to use it? Is this just a way of getting the books into a reading device? Or is it a way to save the library and metadata so that later when I get it in my head to "make it better" I am not left weeding through thousands of files in my recycle bin.

As others have said, you probably don't need this at all, unless your device is not supported.
Although some people use save-to-disk because they dislike using calibre to organize their books, and only use calibre to convert.

Automerge- I really, really need to understand this better. I read that the choices of overwriting, ignoring, and creating a new record indicate the "format" of the book, but what about the metadata? What do I need to do to blend books together to get the metadata to merge? Has anyone come up with a plug-in that allows the users to choose per column what happens? If not, then this would be an excellent feature. If some formats of columns won't blend, then which order do I copy them to get the metadata of a specific one to populate, or leave alone. This will help me to weed down the library with 23,000 books that are basically various copies of 1800 books.

I don't believe there is any way to choose per-column, but the first-selected book will be the "better" metadata AND book formats -- see the options under Edit Metadata ==> Merge selected book records.

The Find Duplicates plugin can help you sort through your current library and deal with duplicates, it is somewhat more powerful than Automerge.

Finally, am I mistaken or did the newest version have a feature that allows copying or importing libraries to be done with a hard link, thereby making it quicker? If this is true, then were do I find out how to use it. I have spent hours waiting on my libraries to copy or import.

Nope -- if you are using the latest calibre, you already have those speed improvements.
They only affect moving libraries on the same HDD.

As BetterRed said, moving/copying files generally requires copying all that data, byte by byte.
It is possible to "cheat", if you only want one copy, and you don't want to cross physical HDDs when moving. calibre did recently gain some optimizations for that, but...

If you aren't seeing any speed improvements, that is because it doesn't apply to what you are doing.

NB: When adding books, calibre creates a second copy, then later deletes the original -- this is for security purposes.