I have Binary distribution of Calibre installed on my Mint Linux machine.
I have just unpacked a source tree somewhere.
I have set one system variable, telling Callibre where to run from
I have made a small change - added a print "hello world" as suggested in Tutorial to see if I am running from source.
I have started Calibre from commandline and saw the "hello world" and I knew that I was ready to poke inside. In a small fraction of time I thought it would take.
I have Binary distribution of Calibre installed on my Mint Linux machine.
I have just unpacked a source tree somewhere.
I have set one system variable, telling Callibre where to run from
I have made a small change - added a print "hello world" as suggested in Tutorial to see if I am running from source.
I have started Calibre from commandline and saw the "hello world" and I knew that I was ready to poke inside. In a small fraction of time I thought it would take.
Your thread in development, interesting. It's something I'll want to look at doing too probably some months in future when I have more time and a burning need to change something too. Very interesting that you thought it was so easy, since I'd been assuming it'd be difficult.
Device: PocketBook 360, before it was Sony Reader, cassiopeia A-20
Quote:
Originally Posted by unboggling
Your thread in development, interesting. ... Very interesting that you thought it was so easy, since I'd been assuming it'd be difficult.
The thread is meant as an encouragement for people like you.
Calibre development isn't easy, if you want to have good understanding of the entire source and if you want to make "big" changes that have impact on many parts of program. But making one tiny change, using 95% of the code that is already working might be easier than you think.
Python programming language used by Calibre is known for it's good readability and for encouraging programmer to write easily maintainable code (as opposed to, for example, Pearl)
KISS for New calibre Users, Version 0.70, 2011-09-09
Keep it simple. Keep your use of calibre as simple as possible to meet your specific needs. When I first started with eBooks and calibre, I wished I could find examples of how other people used calibre to manage eBooks. That's all this is - examples put together in a map of how I use calibre. Not a user guide. Not advice, except to keep it simple. When I keep things simple, I can venture into learning any new area using small steps from that simple baseline.
Version Changes
Spoiler:
This Version.
All sections, nested spoilers deleted, revised and refined.
Starting Steps, recast to my-experience-idealized, instead of advice.
General Work Habits, deleted, content moved to relevant sections.
Metadata, expanded, split into 3 sections Metadata Workflow Preliminaries, Metadata Workflow, Metadata and Tags. Expanded workflows.
Conversion, split into 2 sections Conversion, Format Clean-Up.
Project, expanded preferences current configuration (post-baseline).
Resources (Progress renamed), added links, deleted personal content.
Starting out would have been smoother if I'd done these steps, in this order and timing:
Read the calibre Quick-Start Guide found in the library after installation.
Downloaded and added some eBooks to calibre, continued doing so very slowly and gradually.
Read the calibre User Manual during the first couple weeks, and studied the section called Frequently Asked Questions (FAQ).
Learned more about eBooks, calibre, my reading device by reading various entries in the MobileRead (MR) Wiki and reading recent threads in MobileRead Forums.
Learned how to deal with Digital Rights Management (DRM) as it relates to converting eBooks to format of choice for reading device of choice. Searched for "Apprentice Alf" on the internet and read that blog. Realized DRM plugins are not supported or endorsed by MobileRead Forums or the calibre developers. Installed and configured a few plugins.
Chose a Preferred Output Format in calibre/Preferences/Behavior with forethought, decided on EPUB as "Preferred" rather than MOBI, even though I had a MOBI-reading Kindle and had no EPUB-reading device.
Learned how to get desired metadata onto my reading device.
Added and revised custom columns gradually, considering what information to keep about books.
Learned how to color text in book records and decided what to base that coloring scheme on.
Installed additional plugins available in Calibre Forum Plugins Sub-Forum, after 2 or 3 months getting comfortable in calibre first without them.
Calibre Overall
Spoiler:
Help.
Mouse Tips and Stickies. I pay close attention to Mouse Tips and Stickies. The little boxes that come up when hovering the cursor over something contain important help messages about how calibre works. They are more up to date than the manual and tutorials due to the calibre software improving so rapidly through revisions, additions, and updates. That's also true for most Stickies at the top of the calibre forum, each calibre sub-forum, and other forums at MobileRead.
Help Documents. I use the Calibre Quick Start Guide, User Manual, and Tutorials as references now, but I've read them through twice and keep finding out I've missed things.
Asking Questions. After the first month, I thought I had a handle on everything and didn't ask questions right away in the appropriate forums when I couldn't find an answer in the Help documents. That was a mistake. For example, after 8 months of using calibre, I asked a question about how to right align data in a column and found out I'd never thought to doubleclick a column heading and use any of the commands to be discovered there - mildly embarassing in that case, but worth it for productivity.
More Than One Way. There is usually more than one way to accomplish something in calibre. Remembering that helps me change work habits when circumstances indicate a change would be good. And helps me keep an open mind when people on MR Forums suggest solutions or other ways.
Computer and Calibre.
Backups. Backup software automatically backs up my internal disk to an external drive on an hourly basis. The calibre application and all associated files are on my internal disk. I have file hosting/syncing services such as DropBox but haven't used them with calibre because I didn't want to add another layer of complexity. In the future if I do use one of them with calibre, I'll continue doing my own automated backup rather than depending on a server owned by someone else. I've had to restore from backup three different times after making various blunders in calibre.
Security. My antivirus software auto-scans all volumes, but is set to exclude calibre libraries from scans. The books that I add to calibre were previously scanned at download, scanned again if they were accessed by other applications like compression expander or reader, scanned again when calibre copied during Add Books. The exclusion prevents antivirus software causing slow-downs in calibre performance.
Automation. Trying to combine different types of automation (complex scripts, macros, computed columns, regular expressions, templates) when I didn't know what I was doing led to me feeling frustrated and overwhelmed the first couple of months. Now I like to do a process manually for awhile until I'm familiar with it before trying to automate it. I still don't use scripts or macros, use only two simple computed columns, use only one template. I want to keep things simple to avoid confusion, frustration, and extra work from unnecessary complexity.
Libraries.
Number of Libraries. At present I use two: Core and Test. Core is the primary library for all activities except testing possible library structures or big changes in dealing with books. Test is temporary, frequently deleted then recreated empty to match Core's current structure so I can test from that baseline on books then copied from Core.
Library for Evaluation and Cleaning. When I first started with calibre I used an additional library called Add for evaluating books, working on the metadata, and cleaning books of headers and footers. Now I do all that in Core instead, adding books a few at a time by author slowly, and working on them then and there just after I add them.
Reasons for Just One Primary Library:
Copy and Paste. I can't copy and paste metadata across Libraries without using Library/Quick Switch and search to find what I want to copy, copy it, Quick Switch again, and then search to find where I wanted to paste. So it means more time spent or more typing of metadata such as series information when adding new books unless I just copy and paste from an internet site, which might not be using the same standards for series name or whatever that I use.
Library Structure Changes. After I learn something new or realize something I'm doing isn't working as smoothly as it could, I avoid changing Library structure - adding columns then moving metadata around - until I have a lot of time. With more than one Library, it's more work to implement Library structure changes across Libraries to make them consistent.
Content Server. Accessing books in calibre through the Content Server allows access only to whatever is in the Library that the Content Server is looking at. There's presently no way I know of to focus that Content Server on more than one library at a time.
Catalog. A Catalog may include all the books in the Library where it is generated, but can't include other books from other Libraries. An instance of calibre on one computer presently can't look at more than one Library simultaneously, or different parts of the same Library simultaneously. So when I want to compare different Libraries or parts of one Library visually, one method is to use a Catalog to see what's in one Library (or part) and use the calibre Library View booklist for the other, but the catalog on device is often awkward for seeing things at a glance.
Other Features.
Plugins. I use these Plugins frequently: Find Duplicates, Open With, Search Internet, Extract ISBN, Count Pages. Less frequently: View Manager, Quick Preferences, Manage Series, Quality Check. Occasionally I try other plugins to see if they'll fit my needs.
Regular Expressions (regex). Calibre uses regex in several areas to manipulate strings of text. The only regex I used for 6 months was menu-supplied for detecting metadata while adding books by filename. At about seven months into using calibre I began to incorporate simple regex into my workflow using Edit Metadata in Bulk's Search and Replace in regex mode. I'm not yet familiar enough with regex to use the Conversion Search and Replace regex on a book format.
Unused Features. I've had no need yet to use these features of calibre, other than trying them: Get Books, Content Server, or Command Line Interface. I want to learn and use these soon: Cascading Style Sheets (CSS) and Search and Replace Regular Expressions (regex) as set in Preferences/Conversion Options or the convert selection dialog box; Fetch News.
Adding Books
Spoiler:
Strategy. I'm "going slow" downloading books and adding them into calibre. There are learning curves for calibre's more advanced features and for eBook formats and conversions. The more books in the library, the more work to do in applying newly learned knowledge and skills across an entire library. Upgrading 100 books takes significantly less time than 10,000 books. I add books a few by one author at a time, then process those in calibre before adding more.
Methods for Adding Books, Choices in Preferences/Adding Books.
Auto-Merge. I leave Auto-merge turned off, and later when the little dialog box comes up asking "Add Duplicates?" I always choose to add them because I generally want to see and evaluate any new duplicates of book records or even formats before any deletion, using the Find Duplicates plugin.
Tags to Add. I set Tags to Add: "_New" (without the quotes).
Reading metadata from file contents. If the eBook source provides good metadata in the formats, it's easiest for me to add them to calibre by reading that metadata from internal file rather than filename. Works with most relatively recent retail EPUBs and MOBIs.
Reading Metadata from Filename. The best method choice here is dependent on the origin of the books and their filename structure. The choices involve fixing author, series, and title in the filename out in the Operating System (OS) or fixing it inside calibre later - either manually or using regex. Regex can be used in File Renamers out in the OS before Adding to match one of the Add Books regexs, or in the Add Books regex box on the fly during Adding each batch of books with similar file structure, or after Adding books as an author/title/series mess within Bulk Edit Metadata Search and Replace.
Methods for Adding Empty Books.
I don't use the Empty Book command to create a book record without a format, except when I want to drop a format into it.
Save to Disk doesn't copy empty books (without formats) to disk. So I created a folder containing empty text files titled Empty01 through Empty10 by author "Empty AAA". (First as text files then later converted to EPUB format and saved out to disk.) When necessary for wishlist items, I add a group of 10 of those "empty" book formats and change the metadata of one or several appropriately, assigning a tag "_q0" indicating it's a wishlist item.
Copy to Library (on my context menu) will copy empty books (without formats) to a different library. When I routinely worked with two libraries in the past, I preferred using Save To Disk followed by Add Books to a different library instead of Copy To Library for moving books between libraries because Save to Disk ensured the metadata was saved to my EPUBs' internal metadata fields during the Save, while Copy to Library didn't. The only commands that do that are Convert format, Save to Disk, and Send to Device.
Metadata Workflow Preliminaries
Spoiler:
Custom Columns. I try to minimize my use of custom columns. These are what I use now:
Act, for temporary working tags to batch process groups of books (comma separated text like tags).
Note, for variant titles, pseudonyms, misc. - used rarely for those (text, show in tag browser).
Source, for origin of book (text, show in tag browser).
Pages, (integers, format for numbers {0:,}, for use by Count Pages plugin).
Formats, to see at a glance (computed from other columns).
ISBN, to see at a glance (computed from other columns).
Act and Note also are useful occasionally as temporary storage places for column information when restructuring library or moving metadata around in bulk.
Order of Columns. The order of all my columns doesn't match my workflow sequence exactly. I prefer to leave columns in my preferred order for browsing books in library view. The View Manager plugin allows switching to different column orders and sorts at the click of a button. I've tried it but don't use it often because I like leaving my columns in one place after I get comfortable with my preferred order for doing everything. The trade-off between being out of order of an exact workflow sequence and never getting startled or confused where columns are located balances positively for me on the side of being comfortable and unconfused. As I learn more, the workflow sequences change more often than the preferred column order or the library structure.
Raw Books. I keep all downloaded files that calibre copies when I Add them. I put the original downloaded files in a Raw Books folder out in the Operating System. I've found myself searching Raw Books numerous times and re-adding book formats into calibre for one reason or another. They have bad metadata or haven't been cleaned up but at least they are the original incoming formats; keeping them available is an insurance policy against future need. The Raw Books folder functions as a second kind of backup, but raw. This is especially important for me as a new user because I keep making a lot of mistakes and trying things with different formats from that Raw Books bin. I rely on it being there.
Workflow in Operating System
Gather Odds and Ends. If books by various authors were scattered around my computer elsewhere other than calibre libraries, I'd gather them into my Raw Books/Pending folder first before proceeding. I keep my Raw Books folder on an external drive, containing subfolders Pending and Calibred.
Browse and Buy. I browse the internet, find something I want, buy if it isn't free, then download directly from that site to computer Downloads folder located on same drive as Raw Books folder. I rarely download direct to device because that requires extra steps later. I've never used Get Books feature for obtaining books because I'm not yet adept using search terms and I like browsing other authors and titles out there all at once to help me decide what I want. I download only a few in one session from that site, usually everything by just one author that I want at that point.
While still on that site, I enter the Source (usually site name) in a Download Group folder name, put those downloaded books by one author into Download Group folder in a sub-folder labeled with that author's name, and move that Download Group folder to Raw Books/Pending folder. The folder labeled with source, my Download Group, is usually named like this: "Amazon - Clancy - XXXX" where XXXX is some unique number or phrase to reduce folder-name conflicts later. In addition to noting source, I can add other metadata I want at the time from that site in associated folder-names, filenames, or a new text-file - but usually don't do much of that at download time except to make sure I note source.
When I'm ready to add books to calibre, I find a "New Author Group" by doing a search in the operating system for all books by one author in Raw Books/Pending. To keep things simple I want to add to calibre only one group of books by one author at a time.
After completely Adding an Author Group I append "- Added" to that Author's folder name in Downloads Group. If other books for other authors exist in that same Download Group folder, I leave the just-Added author's folder there.
Eventually I add the other authors from Download Group (if it contains any other authors) one at a time, and when they are all Added I append "- Added" to that Download Group name and move that Download Group folder from Raw Books/Pending to Raw Books/Calibred making sure not to replace any older folders with it - which is why it's helpful to use a relatively unique folder name.
I don't bother to re-order all books by one author into one relevant author folder per author throughout Raw Books folder, because Operating System search can find an author name wherever it resides across the OS as long as author name is relatively intact somewhere in the folder path or filename - so it's not worth the extra work.
Metadata Workflow
Spoiler:
0. General Progression:
Moving from left to right, my columns are in this order: Authors, Series, Title, Tags, Act, Note, Source, Pages, Size, Formats, Published, Publisher, ISBN, Date, Rating. I don't use Modified or Languages.
The general progression for entering and editing metadata is similarly left to right, with a few exceptions that are not in the exact order of columns.
1. For Entire New Author Group:
Make sure Preferences/Add Books is set to add the tag _New to Tags column.
Add Books into calibre, just books by one author out of that download group.
Enter Source. I enter the source of those books first so I don't forget. If necessary I look to see what folder I put it in at download time, where I indicated source information.
Do conversion to EPUB, my preferred format. I convert everything except PDFs, which I evaluate in next step using Acrobat - set to open it automatically from double-clicking in calibre by unchecking PDF in Preferences Behavior and setting Acrobat for PDF in the operating system. For most PDFs I convert them to EPUB immediately after that initial Acrobat evaluation. That EPUB is usually a good enough quality conversion for my fiction-reading purposes, after clean-up, excepting certain PDFs noted in the Conversion section below. I would guess this would hold true for people converting most formats to MOBIs too, though I haven't tested that much.
2. For Each Book one by one:
Do format evaluation. While evaluating quality of format I also get correct authors, series, title from the book itself, as well as other metadata like editor and edition. I do this using calibre viewer for EPUBs, Acrobat for PDFs, other tools using the Open With plugin for other formats when necessary, such as Kindle Previewer for accurate rendering on display of MOBI. If I decide to delay cleaning I tag it "needs cleaning," but if it looks like I can successfully fix it, I usually do clean-up at that point (5 minutes per maximum time I'm willing to spend). If the eBook's clean-up needs are beyond my present skill level, I tag it as a wishlist/placeholder item and go looking for a better format for that title, either soon after finishing the metadata for the others or later.
Enter format quality rating. I put it in Tags. They're defined below under Metadata/Tags/Format Quality Rating.
Correct Authors.
Correct Series Name and Series Index. At this point I use whatever series name and index number is in the book. Later I correct it to match a series convention obtained at a web site, or a convention I've previously used on other series members in the library.
Correct Title. Do necessary corrections, including edition information and variant titles in parentheses as well as titles of major elements of omnibuses (Title of E1; Title of E2; Title of E3). So a search of Title field later will find that information. Temporarily the extra metadata meant for Title column goes into the Note column leaving Title pure for the purpose of metadata download, then appended to Title after the metadata download.
3. For Entire New Author Group:
Extract ISBN. I use the plugin Extract ISBN to get an ISBN when possible, facilitating a more specific and accurate metadata download.
Do metadata download. Prior Authors and Title corrections are vital for this, and ISBN narrows it down. I do a limited Metadata Download making sure not to overwrite any of the columns I've just filled in, by checking only the desired fields for download in Preferences/Metadata Download. My download choices usually are: Published date, Publisher, Comments, Cover. Calibre grabs ISBN automatically if it's not already in the record. I always grab a cover, even when book has internal cover already. I keep only a few sources checked (figuring the more checked, the slower the grab). Amazon's seemed more consistently accurate with broader item availability than others. By default I also use ISBNdb and Open Library. Others I keep unchecked and only use on a case by case basis when needed.
Optionally, do metadata download testing. Presently I'm testing Goodreads. When I'm done with testing Goodreads, I'll test Barnes & Noble, then Fantastic Fiction. The tags and ratings I sometimes download from Goodreads are a temporary thing while testing. I delete their tags after using them to help me decide on my own tags, and I'll probably eventually hide the ratings column and stop using Goodreads tags and ratings because Ratings needs too much screen width for me and I prefer my own tag and rating schemes. My Tags column has only _New, Format Quality Rating, and possible Needs Cleaning tags at this point except for any tags from testing Goodreads.
4. For Each Book one by one:
Enter my tags. Now I know enough from the format evaluation and downloaded metadata such as comments to decide on what tags I want to use for genre, book-type (omnibus, collection, anthology, short story), and so on. I use my own tag scheme for those, explained below.
Delete any temporary tags I may have downloaded unless I'm keeping them longer for testing purposes.
Move any extra Title metadata from Note column, appending to Title. Or, now that I know enough regex, do this step in bulk to New Author Group with Edit Metadata in Bulk Search and Replace/Regex Mode.
5. For Entire New Author Group:
Search for all books in calibre by that author.
Do final additions and corrections on the New Author Group. I double-check all metadata for that author, and add more or correct it manually if necessary by comparing with existing books in calibre by that author and using good web sites that use relatively standard conventions across metadata categories. I get better covers, correct ISBN13s for editions, edition metadata, Series Name and Series Index, Published-dates, or other relevant metadata from relevant internet sites. My overall goal here is to make all the metadata standardized and consistent for that author and ultimately for all books across the entire library.
I like Internet Speculative Fiction Database (ISFDB) best for metadata standardization and consistency (except for genre tags, where they're not) and it's great for series name standardization, awards, covers, ISBN13s. Sometimes I also use the original download site (indicated in Source column), Amazon, WorldCat, or Wikipedia. Most of my books are speculative fiction so using ISFDB where possible works well for me, even for some paranormal romance. For thriller or suspense novels Wikipedia is often good if it's a popular author. I use the plugin Search the Internet sometimes when I remember to, but often it's just as easy to go to a site navigating by browser then searching for author when I'm not looking for one specific book title.
Delete the tag _New.
That New Author Group is now completed.
6. For Download Group:
Choose another author in that Download Group when I have time, including any by that author from other unprocessed Download Groups if they're there.
Repeat all steps for that New Author Group.
Repeat until all authors in that Download Group are processed.
Move Download Group folder from Raw Books/Pending to Raw Books/Calibred
7. For Other Download Groups:
By then there may be other newer Download Groups that need processing, so when I have time I repeat the process for those.
The process is fueled by downloading more books. With that fuel the process never ends.
Metadata and Tags
Spoiler:
Metadata Loss. I postpone any decisions that result in losing metadata until becoming aware of potential ramifications by asking about it on MobileRead or learning for myself through further experience. I don't want to simplify or streamline things to the extreme of losing data. Examples of what I did that needed considerable work later to fix: deleted the articles "The", "An", and "A" from Title; changed Spectra (an imprint of Bantam) to Bantam so publishers were more consistent; used only one of the 3 or 4 co-authors of an anthology when editor name wasn't available rather than taking the trouble to enter all the authors.
Less is More. I use this rule of thumb while entering metadata in tags or columns. Less is more. It's better for me to have fewer tags and columns that are relatively consistent and standard than a whole lot of tags and columns in which the information is an unstandardized and inconsistent mess. Making the metadata consistent across a library requires a lot of work, steadily increasing in time required as the library grows and I want to make changes, and the more metadata there is per record, that requires even more time. So less is more. Less time for maintaining the library. More time for reading.
Tag Abbreviations versus Human Readability.
I don't suggest that anyone use tags in abbreviated fashion the way I do. I like doing it this way. The abbreviated tags became a personal language-slang after a couple weeks using them, good for myself but not good for communicating tags to others who don't know that slang. But there is some reasoning behind why I started using them in the first place.
I use abbreviations for tags to see them all in a small space in narrow columns without needing to scroll through long wide columns or scroll back and forth through a sequence of different columns when I'm browsing the booklist and don't want to do a search to limit the display of books or highlight certain ones.
They're also helpful in that dense way in some uses for Catalog and Content Server on small devices.
Many experienced users say it's better for them to use fully expanded "human readable" tags, but I've found the abbreviations work much better for me.
I wasn't importing outside tags from anywhere when I began (and still don't except when testing a tag source.) I wanted to use my own tag scheme which is generally simpler and more consistent, easier for me to remember, harder for me to make mistakes with, and quicker to type in search boxes or in metadata entry.
For people who use tags that come in with the books or are downloaded from outside, those tags usually arrive in human-readable form as expanded rather than abbreviated words, and it takes time to change, standardize, or even just delete those tags. I got tired of doing that with tags that weren't standard or very consistently applied to similar books in one quality or another.
I didn't like using abbreviations until after the first few days when I originally started using them due to a series of blunders I made by focusing calibre instances on different computers at the same Library files successively, using calibre on either computer to make changes in the same set of files, before I realized I'd ruined the metadata in metadata.db, OPF files, and formats by doing that. By the time I tried to restore the database from OPFs they were bad too. My custom columns metadata would no longer save out consistently, while tests of default tag column did save out. So while frantic and in a hurry, I manually moved all that metadata out of custom columns into the default tags column and abbreviated things on the fly. (I didn't know enough regex yet to do it the easy way in Bulk Metadata Search and Replace.) Saved everything out. Added into a new Library with no custom columns. It worked.
Now I love my abbreviations. I tried to give them up for a week while testing Goodreads tags. Couldn't do it. I'm addicted.
How I Use Tags on a Book.
Tag Types: _FormatQuality, ((GenrePrimary, (GenreSecondaries, [Type, %status, miscellaneous
That means: Format Quality Rating = 4 and readable with no annoyance, Paranormal, Vampire, Young Adult, Omnibus, Series-Multi-Author, Series-Up-To-Date, book missing ISBN13, myRating = 3 (okay) and I've read it.
I use other tag types and abbreviations where relevant, such as for award @h for Hugo or @n for Nebula, format-fix-problem $xHFP for header/footer/page# or $xMT for empty placeholder format.
My Format Quality Rating Tags. After evaluating the format quality of recently added books I assign each book a Format Quality Rating tag. For example, in tag "_q4" the "_" sorts it to the beginning of the list of comma separated tags in the Tags column. The "q" reminds me that it's a format-Quality rating, and the "4" is the rating.
_q0, wishlist item or bad format, both useful placeholders.
_q1, indicates "needs clean-up" if I delay clean-up until later, formerly I didn't use it.
_q2, rare cases where it's more than minor annoyance, not fixable, but retained anyway.
_q3, okay, readable with only minor annoyance.
_q4, good, readable with no annoyance.
_q5, excellent. I don't bother with this, except for a few examples.
Note. I also use _q0 as the basis to color text red in Authors, Series, Title, and Tags (Preferences/Look and Feel). For bad formats, it saves the trouble of using an empty book or empty book placeholder format. When using catalogs or content server with devices, it indicates wishlist items.
Tag Abbreviations versus Human Readability.
I don't suggest that anyone use tags in abbreviated fashion the way I do. I like doing it this way. The abbreviated tags became a personal language-slang after a couple weeks using them, good for myself but not good for communicating tags to others who don't know that slang. But there is some reasoning behind why I started using them in the first place.
I use abbreviations for tags to see them all in a small space in narrow columns without needing to scroll through long wide columns or scroll back and forth through a sequence of different columns when I'm browsing the booklist and don't want to do a search to limit the display of books or highlight certain ones.
They're also helpful in that dense way in some uses for Catalog and Content Server on small devices.
Many experienced users say it's better for them to use fully expanded "human readable" tags, but I've found the abbreviations work much better for me.
I wasn't importing outside tags from anywhere when I began (and still don't except when testing a tag source.) I wanted to use my own tag scheme which is generally simpler and more consistent, easier for me to remember, harder for me to make mistakes with, and quicker to type in search boxes or in metadata entry.
For people who use tags that come in with the books or are downloaded from outside, those tags usually arrive in human-readable form as expanded rather than abbreviated words, and it takes time to change, standardize, or even just delete those tags. I got tired of doing that with tags that weren't standard or very consistently applied to similar books in one quality or another.
I didn't like using abbreviations until after the first few days when I originally started using them due to a series of blunders I made by focusing calibre instances on different computers at the same Library files successively, using calibre on either computer to make changes in the same set of files, before I realized I'd ruined the metadata in metadata.db, OPF files, and formats by doing that. By the time I tried to restore the database from OPFs they were bad too. My custom columns metadata would no longer save out consistently, while tests of default tag column did save out. So while frantic and in a hurry, I manually moved all that metadata out of custom columns into the default tags column and abbreviated things on the fly. (I didn't know enough regex yet to do it the easy way in Bulk Metadata Search and Replace.) Saved everything out. Added into a new Library with no custom columns. It worked.
Now I love my abbreviations. I tried to give them up for a week while testing Goodreads tags. Couldn't do it. I'm addicted.
Tags Discussion.
Over time using calibre I gradually used the tag browser less and the search box more.
I don't use tag browser partitioning and I'm trying to wean myself off the tag browser and use searches more often.
I don't use tags in a hierarchical structure within tag type except genre in the cases of genres that are relatively new to me that are similar to one I'm familiar with. I know some Urban Fantasy but not much about Paranormal Romance, so I've been using secondary genre tags to help get a feel for their similarities and differences. So I have Paranormal or Urban Fantasy as primary genre and one or more secondaries such as Vampire, Shifter, Demon, Fae, Magic. After I'm more familiar with the paranormal romance genre in general, I'll stop using genre secondaries on those books.
I prefer the other tags all flat structure except for the tag category prefix so I can assign more than one on the fly without thinking about it.
At about 3 or 4 months in I stopped using a lot of columns for genre, booktype, multi-level series, series-status and so on, and started using the default tags column, with tag prefixes to designate tag type, accompanied by abbreviations. I'm overall happy with the way I do tags except for the myRating tag ("r3") needs a prefix symbol or better method to enable faster searching on my ratings, and Missing ISBN13 ("mi") and Missing Cover ("mc") need a unique prefix. I'm still thinking about what to use and what else I'd have to change to use them.
Conversion
Spoiler:
Preferences/Behavior/Formats For Internal Viewer. I uncheck PDF, leave the other formats all checked. PDF extension is set in my operating system to open Acrobat automatically.
Preferences/Behavior/Preferred Output Format. I prefer EPUB format because it opens fast in the calibre viewer, works on my iPad without conversion, usually converts well to MOBI format for my Kindle, and is useful for clean-up purposes.
Preferences/Conversion Settings. I leave calibre's conversion settings at default except for testing a setting enough to know what it does before implementing it. For most conversion settings I never implement them in Preferences on a wider scale, just remember to choose those I want while converting one or more books. This applies to all Preferences Conversion Options and to Conversion dialog box that comes up before converting.
Formats To Keep.
EPUB (my Preferred Format, initial evaluation, possible clean-up, reading on iPad, conversion for Kindle). if the original incoming format converted well to EPUB, after cleaning up any format problems it has if necessary, that EPUB becomes my "master" copy to generate conversions for various reading devices when I want to read something. This works for me for most fiction books.
Original incoming format. If it wasn't an EPUB to begin with, and it converted well to EPUB, I delete the original from calibre's book record. I still have the downloaded original copy in Raw Books out in the operating system and I keep that there.
Original incoming format. If it didn't convert well to EPUB, I keep it in the book record alone and delete the EPUB that was generated for initial evaluation. This allows easily double-clicking it to open it in the appropriate application. For example, this happens often with PDFs with complex graphics, old image-based PDFs, newer technical PDFs with complex layouts, or many textbooks. In these cases I evaluate it and read it in its native format using the appropriate application.
Nearly all of my book records, after updating metadata and format cleaning, contain just one format, with most as EPUB and only a few as PDF.
Most formats that I keep in calibre are readable with only "minor annoyance" or "better" on my reading devices, once metadata has been updated and corrected and any format problems cleaned up. If a format is not readable without major annoyance, I either delete it completely or tag it as "bad format" ("_q0") and keep it as a placeholder and don't read it yet, hoping I'll find a better format for that book some day.
Format Clean-Up Initial Conversion. I want to use the cleanest and least converted format available as the conversion Input Format. Potential choices for Output Formats are:
EPUB for fix in Sigil or other EPUB editor.
RTF for fix in Open Office, Word, or other editor.
HTMLZ for fix in any HTML editor, might be useful after I learn HTML.
PDF for fix in Acrobat or other PDF editor.
MOBI for fix in MOBI editor.
TXT or TXTZ lose formatting such as Bold/Italic, sometimes useful.
Format Clean-Up Conversion Sequences. These are some of the conversion sequences I've tried. I like #1 then #2 for quality of results, but I'm not yet comfortable enough in HTML to edit HTML directly in HTML editor or in Sigil. (EPUB innards are mostly XML, XHTML, and HTML, plus images.) Now I use #3, fix in Open Office, working on an RTF format, save to ODT, convert that to EPUB in calibre - this is so far best for me in simplicity and ease, with good readability though not finely-tuned format quality. I always work on a copy first saved out of calibre.
Original Format --> HTMLZ --- HTML editor fix, save as HTMLZ (or HTML then zip) --> Preferred Format (EPUB for me).
Original Format --> EPUB --- Sigil fix, EPUB, save as EPUB --- Already was/is my Preferred Format.
Original Format --> RTF --- Open Office fix, save as ODT --> Preferred Format.
Original Format --> RTF --- Open Office fix, save as RTF --> Preferred Format.
Original Format --> HTMLZ --- Unzip, Open Office fix, save as HTML, zip --> Preferred Format.
Original Format --> RTF Word fix, save as DOCX --> Open Office ODT --> Preferred Format.
Original Format --> RTF Word fix, save as RTF --> Open Office ODT --> Preferred Format.
Original Format --> RTF Word fix, save as RTF --> Preferred Format.
Original Format --> RTF Word fix, save as HTML, zip--> Preferred Format.
Note 1: Calibre supports ODT as input format but not as output format.
Note 2: Calibre does not support Word DOC and DOCX as input or output formats.
Note 3: Writer2ePub extension to Open Office is an option for converting to simple EPUB after fix in Open Office.
Format Clean-Up
Spoiler:
Extremes. Regarding fixing problem formats, there are two extremes for eBook people. One is to just want to read eBooks and not care much about the formatting and any format problems. The other is to want to make each eBook format as perfect as possible at that person's current skill level (which tends to keep increasing) sometimes generating a need to spend a lot of time going back to re-fix older books to bring them up to par across the library. But this is also the main reason I decided to go slow adding eBooks. I'm trying to sit the fence between those extremes. I don't worry about all format problems, just the ones that annoy me the most that I currently know how to fix.
Technology Changes. That's another point to consider for how I fix books and how much time I spend doing it. Format and device technology change over time, so in three years all that work on older formats may become relatively obsolete after buying the new superduper device that supports the new improved version of an older format, or that even supports a new super-duper-best-format-ever-invented. In ten years (if not three) those old formats in the library might look pretty poor in comparison and in that case, changing 20,000 books over in an effort to upgrade them wouldn't be pleasant and probably still wouldn't compare in quality. It's like old 78 rpm audio to 45 to 33 to tape cassette to CD to mp3 (or a better lossless audio format). Listening to an audio piece that went through all those upgrades - which took a lot of work to do at each stage - is no longer satisfactory in comparison to going out and buying an audio piece online for couple dollars per track or less and listening to that.
It's Not My Job. I'm not a publisher, distributor, or editor. I'm an eBook reader. Cleaning up eBook formats takes valuable time away from reading. My goal as a reader isn't a perfect eBook, but to spend the least amount of time to make it "readable by me with as little annoyance as possible." I examine all new incoming formats for format quality. If it looks like I won't be able to clean it up in five minutes or less, I scrap it as not worth it or tag it "_q0" and add a tag for the type of format problems it has. I always work on a copy saved out of calibre rather than on the format in calibre, and that will have to be more flexible when I learn enough regex, CSS (Cascading Style Sheets), and mechanical details of various formats to fix any problems directly during conversion, but calibre will still leave an original format unchanged automatically, renaming it to ORIGINAL_XXXX where XXXX is the format type.
Won't Fix Everything. I haven't worried about or cleaned up most Table of Contents (TOCs) yet because for most books except big omnibuses I don't use or care about TOCs. I do strip out headers, footers, and page numbers when I can without causing a lot of split paragraphs, because those annoy me by interrupting text flow on my reading devices. Except for problem PDFs (graphics, technical), if I can't fix that I scrap it or code it "_q0" meaning "bad format, replace with better someday." If in the meantime I learn how to fix that kind of problem, I may end up fixing it rather than replacing it.
Comfortable Ways Work. I'm comfortable in Word so I had been using the conversion sequence discussed above that includes Word DOCX. The conversions from RTF to DOCX to ODT each reduced size considerably but I'm not experienced enough with evaluating formats to know much about their resulting quality except that it "looks okay for me to read now." During this next iteration of calibre use, I want to reduce the number of conversions and simplify that process so I'm now doing that simpler sequence of Open Office for clean-ups of RTF saved as ODT then dropped into calibre where I then delete the uncleaned original format, the RTF used for the clean-up, and any other uncleaned format for that book.
Learning Better Ways Works Better. After I'm more sophisticated using regex, I'll switch to using calibre's conversion search and replace to remove headers, footers, and page numbers. Once I learn enough HTML to be comfortable, I will switch to doing clean-ups of most problems using the simplest path available, either Sigil or HTML editor or calibre conversion's Search/Replace. I'm making it a high priority to learn regex, Sigil, and HTML.
Reduce Conversions. I want to reduce the number of conversions in sequence for several reasons. Save time. Simplify workflow. And most important, achieve higher quality of format. Like photocopying copies of copies of copies, or successively converting audio files through "lossy" compressions or types of recording media, each step loses more formatting and content information while introducing more errors. Experts in conversion highly recommend minimizing the number of conversions and starting with as original and "unlossy" a format as possible.
Specifics for PDFs.
With non-fiction I often don't care about headers, footers, and page numbers interrupting the flow on reading device, and often prefer them to be there for reference. With fiction and some non-technical non-fiction, I do care and want to remove them. With all books I also want to fix any other format problems I know how to quickly fix.
After initial evaluation in Acrobat Pro, and still within Acrobat Pro, it's possible to edit out headers, footers, and page numbers from "unlocked" PDFs, but not possible to unlock "locked" PDFs at that point to remove headers, footers, and page numbers. From Acrobat the choices are to save it out directly in a different format, which means Acrobat does the conversion, or closing the book in Acrobat and converting the PDF directly in calibre to some other format. I haven't yet tested the resulting quality and suitability of any of Acrobat's possible save/exports, which include HTML, XML, DOCX, and RTF, and testing those won't be a priority until I get some problem PDFs to test on. Presumably that Acrobat-generated HTML or XML can be fixed in an HTML editor, and Acrobat-generated DOCX or RTF in a word processor - depending on how the Adobe PDF "lock" affects the Acrobat saves.
For PDF formats imported to calibre or converted there, the discussion still applies, in Conversion section and above this sub-section.
Some tools are listed in the Resources section that can help with cleaning-up PDFs.
Devices
Spoiler:
Device Choice for Type of Reading. Different reading devices have different characteristics and capabilities (such as weight, color display or not) that make them suitable for different types of reading. Generally I use Kindle for fiction and iPad for technical and graphics reading. Due to my iPhone's display being small, I'm not comfortable using it for long-term reading.
Conversion Before Device Loading. I delay conversion until shortly before reading, convert on-the-fly to whatever format is necessary for the desired device. Preferred conversion input format for me is the EPUB I converted to originally after Adding, or a cleaned up version of EPUB. In the case of PDFs or any other problem or complex formats, I'll try reading it without conversion first on Kindle for fiction or iPad for graphical or technical, in PDF or other format the device can read, and where that doesn't work well try determining how to convert it or fix it in a way that does work well for one of my devices.
Device Loading. I load my devices only a few books at a time then when finished reading, delete off the device using calibre directly (preferred if possible) or directly from device. If I travelled more frequently, I'd load more books. After reading I assign a content rating in calibre metadata and adjust any other tags if necessary.
Collections. I don't use collections on devices or in columns. To me they're just extra work. Usefulness of collections on devices probably depends on how well each type of device handles them.
Catalogs. When I need to put a catalog on one of my devices, I check only "Books by Authors" in the Catalog dialog box and indicate a wishlist item or bad format with "_q0" in Tags. (That means Format Quality is zero.) Using a Catalog on a device is an easy way to compare two different Libraries simultaneously (one in calibre, the other on device in Catalog), or to look at two different parts of one Library simultaneously.
Templates. Calibre uses a template language to assign a column name to substitute metadata from that column. I currently use only one template. It's in Metadata Plugboards and adds series and series index to title for my Kindle.
Code:
{series}{series_index:0>2s| - | - }{title}
KISS Project
Spoiler:
Goals. KISS my use of eBooks, calibre, and other software tools. Determine strategies and methods for gathering, managing, and cleaning up eBooks. Gradually learn relevant "best practices." Learn to use calibre and supportive software tools better. Manage eBooks better. All to facilitate the ultimate purpose, reading eBooks.
Definition of KISS. I use the "verb" form of the principle "Keep It Simple Stupid" as meaning "to simplify a complex project or series of tasks in order to improve results." The word "Stupid" in the principle is not used or intended in a pejorative manner. When I say "to KISS" I mean "to simplify and improve." Brief Wikipedia explanation of KISS Principle.
Reasons to KISS. When I started out new to eBooks and calibre in January 2011, I frequently felt overwhelmed. Paper books are fundamentally different than eBooks, generating a need to determine different strategies and methods for managing and using eBooks, which I hadn't done yet. The calibre eBook library management application allows new users to use it in simple ways while also accommodating more advanced users with many features and complexities. I was overwhelmed at first because I didn't know much about eBooks in general and I let myself get tangled in complexities. That abetted sidetracking into tangents before settling in. Later I noticed that the more I learned about eBooks in general and the more I consciously simplified my use of calibre, the more I was successful in managing eBooks.
History. At seven months into using calibre, I wanted some discussion on my strategies, methods, and work habits, so I laid out what I was doing as the original post of the thread "KISS for New calibre Users". After subsequent discussion I recast the KISS version posts from "giving advice to new users" to "documenting what I'm doing, as one slightly experienced user." I also re-started at baseline zero each for eBooks and calibre. I intend to use better strategies, methods, and practices during this next iteration cycle of my calibre use.
Plan.
Add books very slowly.
Learn, and integrate into my own use: CSS, HTML, Sigil and other relevant editors or tools, other calibre features. Also: more on eBook reading devices and reader software, good metadata sources, and eBook formats, conversion, clean-up.
Continue KISS revisions corresponding to my own experience, trying to integrate suggestions made by others into my use of eBooks, calibre, and relevant tools and then into the next revised KISS post. Add or revise strategies, methods, workflows, practices, and resources after testing or researching them. Continue updating KISS posts periodically at slower pace as it stabilizes.
Baseline Configuration of calibre.
2011-08-14. Deleted calibre application, library files, configuration directory, all associated system files manually and with aid of de-installer on all computers (2 Macs).
2011-08-14. Deleted nearly all eBooks off all storage devices including backups.
2011-08-14. Installed latest binary version of calibre with its one library containing one eBook as a clean new calibre installation on primary computer.
2011-08-14. Configured Preferences with basic user information.
At that point I had only a few miscellaneous eBooks scattered around: calibre Quick-Start Guide (in calibre library), Kindle User Guide, various user guide PDF formats, and a few other technical and reference PDFs.
That was my starting situation and baseline for gathering and managing eBooks as well as helping me move closer to a situation somewhat similar to that of a starting-out new user of eBooks and calibre, with the addition of some experience-based perspective.
Current Configuration of calibre.
Look and Feel. Various changes to suit.
Look and Feel/Column Coloring. _q0 value in Tags for red in Author, Title, Series, and Tags.
Behavior. Preferred Output Format: EPUB. Unchecked PDF format, checked all others.
Behavior. Peferred Input Format order: Moved EPUB to top, MOBI 2nd, others left in default order.
Add Columns: described elsewhere.
Toolbar. Set main toolbar and library context menu. I don't use the others.
Searching. All unchecked and unused.
Conversion/Input Options/Comic Input. Checked Disable conversion of images to black and white.
Conversion/Common Options/Page Setup/Output Profile. Set to Kindle.
Conversion. All other conversion options at default.
Adding. Checked: Read metadata from file contents, Copy To Library preserve date. Unchecked: Auto-merge. Tags to Add: "_New" (no quotes). Regular Expression the longest most complex choice in default menu for adding by filename.
Saving. Checked: Save cover, Update metadata in saved copies, Save in OPF, Convert Non-English. Everything else default.
Metadata Plugboards. Template for Kindle, described in Devices section.
Sharing by email. Set up, tested, but seldom used.
Sharing over net. Set up, tested. Used for loading iPad through iTunes. Content Server seldom used.
Metadata Download. Described in Metadata Workflow section.
Plugins. Described in Starting Steps, Calibre Overall, and Metadata Workflow sections.
Tweaks. Format PublishedDate year only as yyyy, title and series sorting strictly alphabetic.
Miscellaneous, Keyboard, Template Functions. All at defaults.
Feedback and Input. Request feedback and input on KISS post revisions after each is posted. All strategies, methods, practices, workflows are offered not as advice but as examples of what one new user is doing, struggling with, or trying to do. I hope this may be useful to new users, increasing in usefulness over time as it is refined in successive iterations. Feedback, input, and discussion will be helpful in correcting or improving any assumptions, strategies, methods, practices, and workflows. Please post in this thread.
Thank You. Thanks to everyone who posted on MobileRead, where I learned most of the content contained in this post. Particular thanks to those of you who posted in this thread and helped me try to internalize, integrate, and improve all this content or format it better.
Resources
Spoiler:
For tools to use in association with calibre I try to find those recommended by experienced people at MobileRead. Good tools have their own learning curves. For any I haven't tried enough myself, my comment includes "noted".
@all: Again, request feedback and input on this revision of KISS for New calibre Users. All strategies, methods, practices, workflows are offered not as advice but as examples of what one new user is doing, struggling with, or trying to do. I hope this may be useful to new users, increasing in usefulness over time as it is refined in successive iterations. Feedback, input, and discussion will be helpful in correcting or improving any assumptions, strategies, methods, practices, and workflows.
@mods: Things seem to be stabilizing. Other than the number of hits which doesn't tell me a whole lot, I've no idea how useful it may be as a map for new calibre users.
@new calibre users: So, is it useful?
Last edited by unboggling; 09-09-2011 at 03:51 PM.
unboggling, yes! This is useful for this new user. I'm so new, I can't comment intelligently yet. I just wanted to say thank you for your time and effort.
Oh, and I first heard KISS as "Keep It Simple Sweetheart", said with a Humphrey Bogart lateral lisp, if you can imagine such a thing.
unboggling, yes! This is useful for this new user. I'm so new, I can't comment intelligently yet. I just wanted to say thank you for your time and effort.
Oh, and I first heard KISS as "Keep It Simple Sweetheart", said with a Humphrey Bogart lateral lisp, if you can imagine such a thing.
You're welcome. Bogart - oh, the nostalgia…. Big smile.
Sigh. My beloved tag scheme, abbreviations and all, is trash as soon as I figure out what to replace it with. (takes long breath, with a hitch in it…)
Why? Because I played around with a lot of different kinds of searches today, and some of the symbols I was using as tag prefixes to distinguish and sort kinds of tags didn't work in informal searches, and some of my abbreviations without those prefixes weren't unique enough for easy searching.
So, my two most used and necessary kinds of tags are Format Quality Rating and Primary Genre. I can leave those two in Tags and use a better prefix on the genre tag, so it's easy to get them to be useful in Catalog and Content Server. Everything else can go into [sigh again] custom columns, such as book type (anthology, omnibus, etc), awards, format problems…. Those new columns will make searching more precise anyway. [sigh…]
So, if anyone was thinking of switching to a tag scheme like mine as previously posted, I'd recommend doing something else that works better with the search box. My scheme worked when I used the tag browser and only formal searches from the binoculars search dialog, but now I want to rely on the search entry area next to the binoculars on the fly, much more than I did before.
Last edited by unboggling; 09-14-2011 at 12:18 AM.
Unboggling you are a great helper! Your information here is now on my desktop to aid me. I have found Calibre to be filled with complexities-ideas that connect and connect to another and another. Simplicity is not inherent. I was searching when i got my Kindle 3Gen for a way to add books that were not in the required format when CALIBRE's name frequently popped up on tech-pc-mac sites for conversion tool being the best. Especially its ability to convert from and with numerous different formats. I think of it as my First and Best Translator-Converter. Yet, I too have had various things to explore and research within Calibre so that I could complete a task. This is a great "tips" guide to add to The Quick Start Guide by John Schember.
Unboggling you are a great helper! Your information here is now on my desktop to aid me. I have found Calibre to be filled with complexities-ideas that connect and connect to another and another. Simplicity is not inherent. I was searching when i got my Kindle 3Gen for a way to add books that were not in the required format when CALIBRE's name frequently popped up on tech-pc-mac sites for conversion tool being the best. Especially its ability to convert from and with numerous different formats. I think of it as my First and Best Translator-Converter. Yet, I too have had various things to explore and research within Calibre so that I could complete a task. This is a great "tips" guide to add to The Quick Start Guide by John Schember.
Thanks, TD. Your feedback helps.
But this isn't ready yet for the big time, it's still rough and still being put through major revisions about once a week. So I wouldn't rely on the copy on your desktop too long without checking back for new revisions. For example, per my last post here, some new columns and new way of using tags (new to me, anyway) will be in next revision, along with a major overhaul of a lot of the rest of it. Just after I post each revision I think it's stabilizing, then a day or two later I learn something important, then try it out, and then start revising.
Unboggling your suggestion to check frequently what you do with this KISS Tips is a good one. It did occur to me later that as it is an ongoing project of yours that I should do that (check frequently).
I am very basic at the moment.
If you are interested I do use Collections on my Kindle. I have a collection/folder for purchased ebooks, for wish list, and then for my fav topics. This way I do not have to go thru the list as Kindle has arranged. Kindle as far as I can tell allows me to organize by most recent ebook viewed, by author, title or collections.
For me I often forget title or author name so to place it in a folder such as "Classic Mystery" is helpful.
I am not familiar with the tags you speak of. I will explore it further. Ciao.
If you are interested I do use Collections on my Kindle. I have a collection/folder for purchased ebooks, for wish list, and then for my fav topics. This way I do not have to go thru the list as Kindle has arranged. Kindle as far as I can tell allows me to organize by most recent ebook viewed, by author, title or collections.
I haven't had a need to use collections much myself lately. When I tried managing collections directly on the Kindle, I found them awkward to use, but to avoid that awkwardness, in case you didn't know, there is a good calibre plugin for managing Kindle collections from calibre, here: Kindle Collections.
I haven't had a need to use collections much myself lately. When I tried managing collections directly on the Kindle, I found them awkward to use, but to avoid that awkwardness, in case you didn't know, there is a good calibre plugin for managing Kindle collections from calibre, here: Kindle Collections.
09-06-2011, 02:38 AM
Similar Threads