[unboggling]

---

Link to latest version: Version 0.80, 2011-09-16, Post #243.

---



KISS (v 0.40)


KISS Version Changes

Version 0.40, 2011-08-23, KISS Post #145. Restructured work habits to correspond to calibre's interface. Improved human readability. Added version changes. Added workflow sequence in Metadata section. Clarified and expanded Conversion section and format topics. Many minor changes, a few minor additions.

Spoiler:

Version 0.31, 2011-08-21, KISS Post #119. Added formatting, spoilers, minor corrections and additions.

Version 0.30, 2011-08-20, KISS Post #91. Restructured as project at zero baseline for eBooks and calibre.

Version 0.21, 2011-08-12, KISS Post #52. Restructured as advice accompanying work habits.

Version 0.10, 2011-08-08, KISS Post #1. Original Post casually proposed KISS, with tips and work habits.

Project Information

Spoiler:

Project. KISS my use of eBooks. KISS my use of calibre.

Purpose of Project. Learn about eBooks. Determine strategies and methods for gathering, managing, cleaning, and reading eBooks, and gradually learn relevant "best practices." Learn to use calibre and associated software better. Manage eBooks better.

Baseline. My baseline is similar to what a brand new user to eBooks and calibre might have after installing calibre then accomplishing these important tasks:

• Soon, read Quick-Start Guide thoroughly.

• Soon, set system security auto-scan antivirus to exclude calibre libraries.

• Soon, choose a "Preferred" Format in calibre/Preferences/Behavior with considerable forethought.

• Soon, learn and set how to get the specific metadata desired onto current primary reading device.

• Soon, learn and set how to deal with Digital Rights Management (DRM) as it relates to converting eBooks to format of choice for reading device of choice.

• At leisure, add custom columns and install plugins.

• At leisure, learn how to color text in book records.

Request Feedback. All strategies and work habits are proposed and are not offered as advice to anyone, but as examples of what one new user is doing, struggling with, or trying to do. I hope this may be useful to other new users, increasing in usefulness over time as it is refined in successive iterations. I request feedback and discussion so I can correct or improve any assumptions, strategies, methods, workflow, or practices. Please post feedback and discussion in this thread "KISS for New calibre Users" or send me a private message.

Explanation of KISS. I am using the "verb" form of the principle "Keep It Simple Stupid" as meaning: to simplify a complex project or task. The word "Stupid" in the principle is not used or intended in a pejorative manner against myself or anyone else, but instead has a connotation of praise. A brief Wikipedia explanation is here.

Reasons to KISS. When I started out new to eBooks and calibre in January 2011, I frequently felt overwhelmed. Paper books (pBooks) are fundamentally different than eBooks, generating a need to determine different strategies and methods for using eBooks, which I hadn't done yet. The calibre application for managing eBooks is simple to use and accomodates more advanced users with many features and complexities, but I was overwhelmed at first because I didn't know much about eBooks in general and I let myself get tangled in calibre complexities. So I sidetracked into tangents for awhile before settling in. Later I noticed that the more I learned about eBooks in general and the more I consciously simplified my use of calibre, the more I was successful in managing eBooks.

Project Summary To Date. At 7 months into my calibre use, I realized I needed feedback on my then-current strategies, methods, and work habits, so I laid out what I was doing hoping to get discussion. That's all available in the forum thread "KISS for New calibre Users". As discussed there, I recast the KISS posts from "giving advice to new users" to "documenting what I'm doing, as one new user." I also re-started at baseline zero each for eBooks and calibre (reasons in Post #65). I intend to use better strategies, methods, and practices during this next iteration cycle of my calibre use.

Status at Present. I use a MacBookPro17 running latest released version of OS X Snow Leopard with latest released binary versions of calibre for OS X, and an older MacBook set up the same way, both on an Airport wireless network. Both computers use auto-updated ESET Cybersecurity on auto-scan with calibre libraries excluded. My devices Kindle3, iPad1, and iPhone3 are also up to date with OS's and firmware. I have one Library named Main containing one eBook in a clean new calibre installation. With plugins added and preferences configured, user-specific info added (such as user name or ISBNdb key), 6 custom columns added, 1 template in metadata plugboards added, no new regex menu items added, no tweaks. Conversion settings are at default except for Input Options/ComicBook yes to "Disable conversion of images to black/white". I set a tag to color a book's metadata red. In Behavior I checked yes to all listed formats figuring I can always uncheck any later. My Raw Books folder outside of calibre is empty and I now have only a few miscellaneous eBooks scattered around: calibre Quick-Start Guide (in Main), Kindle User Guide, various user guide pdf formats, and a few technical, reference, and other old pdfs.

Strategy for eBooks

Spoiler:

Overall. Obtain, clean-up, and read "on demand." I plan to make exceptions where warranted for good reasons such as "found a great edition that's hard to find" or "it's temporarily on sale" and "I want it enough to break the rule." But my primary purpose is to read eBooks, not to gather, clean, and hold them. And I want to "go slow" this iteration of my calibre use for several reasons. I'm still in steep learning curves for converting and cleaning-up eBooks. If my library grows fast, most of those books will have been previously format-cleaned at lower quality levels as I rise up the learning curves. The greater the quantity of older books there are at any point, the more time it will take to re-fix them to make them more consistent in quality across the library. Re-fixing previously fixed books adds more conversion errors and problems into the mix. So it makes sense to go slowly and build carefully at least until those steep learning curves level out more toward a slow and gradual learning process.

Gathering. Gather for read-on-demand strategy as possible. Primarily by browsing the internet. Places like Amazon, Barnes & Noble, Baen, and so on.

Cleaning. Lean and mean according to my skill level. Minimize multiple conversions in sequence. Retain original incoming format. Always do major clean-up work on copies first saved outside of calibre just as a matter of habitual hygiene.

Managing. Learn and use the calibre eBook Management application.

Learning. Learn CSS style sheets and Sigil ePub editor. Start gradually learning HTML. Learn better strategies and methods on Mobile Read. Start improving eBook-related skills in an ongoing manner.

Changing Habits. All work habits described below are "proposed" habits that I've been doing, am in the process of changing, or want to do from now on. When I determine a new strategy or learn a new method or skill, my work habits change to accomodate it.

General Work Habits

Spoiler:

Cybersecurity. My anti-virus software is set to auto-scan all volumes, with specific settings to exclude calibre libraries from scans. That prevents the security software from causing performance slow-downs when calibre accesses the book files for one reason or another, which it does frequently. The books that were added to calibre had been scanned at download, then scanned again if they were accessed by any other application such as a compression expander or a reader application, then scanned again when calibre accessed them to copy during Add Books.

Backups. I use backup software (Time Machine) to automatically backup my internal disk on an hourly basis to an external drive. The calibre application and all associated files are on my internal disk. I have file hosting/syncing services (Mobile Me, DropBox, SugarSync) but haven't used them with calibre because I didn't want another layer of complexity yet. In the future if I do use a file hosting/syncing service, I will continue to do an automated backup of my own rather than depending on a server owned by someone else.

Raw Books. I keep all downloaded files that got copied by calibre when I Added them. After Adding to calibre, I put the original downloaded raw eBooks in folders in this structure: Raw Books/Source/DL Group/Original Filename. DL Group refers to identifying a group of files downloaded at once, or to a broader category such as Baen Free CDs. I've found myself searching Raw Books numerous times and re-adding books into calibre for one reason or another. They have bad metadata or haven't been cleaned up but at least they are the original incoming formats; keeping them available is an insurance policy against future need. It functions as a second kind of backup, but raw.

Tangents. I want to avoid running off on complex time-consuming tangents that take me away from my main purpose, reading eBooks. I wasted time doing that merely to avoid feeling overwhelmed. For example, writing scripts to print booklist, worrying about other database interfaces, and doing other things that had little direct bearing on reading eBooks or using calibre to manage eBooks and associated reading devices.

Automation. I like to do a process manually for awhile until I'm familiar with it before trying to automate it with scripts, regular expressions, templates, or computed columns. Attempted automation without prior innate understanding of the manual mechanics usually wastes lots more of my time than it saves. Not knowing what I was doing but trying to combine different types of automation all at once anyway led to me feeling frustrated and overwhelmed.

Calibre Overall Work Habits

Spoiler:

Mouse Tips. I pay close attention to them. These little boxes that come up when hovering the cursor over something contain important help messages about how calibre works. They are more up to date than the manual and tutorials due to the calibre software improving so rapidly through revisions, additions, and updates.

Stickies. I pay close attention to the Stickies at the top of the calibre forum and each calibre subforum. They contain important information.

Workflow Timing. Using the calibre viewer I evaluate an incoming eBook's format shortly after it's added then assign a format quality rating tag. If it looks like I can successfully fix it, I delay any clean-up or conversions until shortly before reading on the reading device. If the eBook's clean-up needs are beyond my present skill level, I tag it as a wishlist/placeholder item and go looking for a better format for that title, either right away or later. After that initial format quality evaluation I work on filling in the remaining metadata.

Regular Expressions. Calibre uses Regular Expressions (regex) in several areas to manipulate strings of text. The only regex I used for 6 months was menu-supplied for detecting metadata while adding books by filename. I'm not yet familiar enough with regex to use the Conversion Search and Replace regex on a book format. At about seven months into using calibre I began to incorporate simple regex into my workflow using Edit Metadata in Bulk/Search & Replace in regex mode.

Templates. Calibre uses a template language to assign a field name (the column lookup name) to substitute metadata from that field (column). I currently use only one template. It's in Metadata Plugboards and adds series and series index to title for Kindle.

{series}{series_index:0>2s| - | - }{title}

Libraries. Presently I use only one library named "Main." When I first started with calibre I used an additional library named "Add" for evaluating and cleaning books and working on the metadata. Now I'm going to try doing it all in "Main" for awhile.

Catalogs. I rarely use this feature. When I do need to put a catalog on one of my devices, I check only "Books by Authors" and indicate wishlist item with "_q0" (defined in Conversions section below).

Plugins. I use these Plugins frequently: Find Dupes, Open With, Search Internet, Quality Check. Recently added Count Pages.

Other. I've not used these parts of calibre due to no need or desire to use them yet: Fetch News, Get Books, calibre Server. I've not used these parts of calibre due to personal preference not to use: collections in calibre or on devices. I've not yet used these parts of calibre but want to learn and use: CSS (style sheets), Search and Replace regex on book during conversion process.

Adding Books Work Habits

Spoiler:

Methods. Methods for adding books depend on a group of books' source (origin) and their filename structure. If that source provides good metadata in the formats, it's easiest for me to add them to calibre by reading that metadata from internal file rather than filename. If not, there are several methods using filename. Manually fix all the author/series/title info in the filename out in operating system (OS), or do it there using renaming tools (which use regex also) by batching together files with similar structures to fix, then Add Books to calibre once filenames are fixed to match one of the Add Books by filename regexs in the menu. If I knew enough regex I could write and use successively a different regex to match each varying file structure of those books and import the appropriate metadata directly into the appropriate column during Add Books - but I don't know regex enough yet. There are also scripts and tools available on this forum to help solve this problem, standardizing names before import. Or I could add that group of books as a mess without fixing names in the OS, then fix everything selectively using regex in the Edit Metadata in Bulk Search/Replace window. I've tried it all those ways except with renaming tools or scripts, and they all require work.

Empty books. I don't use Empty Book command or Empty Book records. I created a folder containing empty files (originally text, later converted to epub) titled Empty01 through Empty10 by author AAA, TBD. When I need to, I add a group of 10 "empty" book formats and change the metadata of one or several appropriately, keeping it format quality tag _q0. The reason I do this is because empty books don't get included when I SaveToDisk a selection of books out. When I want them included in saves that eventually get added to a different library, book records indicating wishlist items need to hold a format. If I want to avoid all that I can just use CopyToLibrary instead which does copy empty books without formats.

Metadata Work Habits

Spoiler:

Metadata Loss. I don't make any decisions that result in losing metadata, until I'm more experienced and become aware of potential ramifications. Examples of what I did that needed considerable work later to fix: Deleted The, An, A from title. Changed Spectra to Bantam so publishers were more consistent. Used only one of the 3 or 4 co-authors of an anthology where the actual editor's name wasn't available, rather than taking the trouble to use all the names. I don't want to KISS things to the extreme of losing data.

Custom Columns. I try to minimize my use of custom columns. These are what I use now (with recent addition of "Source" and "Pages"):

• ISBN, to see at a glance (computed from other columns).

• Formats, to see at a glance (computed from other columns).

• Act, for temporary working tags to batch process groups of books (comma separated text like tags).

• Notes, for variant titles, pseudonyms, miscellaneous (text, show in tag browser). I use it rarely.

• Source, for origin of book (comma separated text like tags).

• Pages, (integers, format for numbers "{0:,}" for use by Count Pages plugin.)

Workflow Sequence. For a newly added book, the workflow order for editing columns can be important. Moving from left to right, my columns are in this order: Authors, Title, Series, Tags, Act, Notes, Source, Pages, Size, Formats, Publisher, Published, ISBN, Date. I don't use Modified, Rating, or Languages. The sequence for editing metadata is similarly left to right. I correct Authors and Title first because they are vital later for Metadata Download. I get series information by using browser at relevant sites. I then evaluate the format and learn enough about the book to assign tags into Tags, using my own tag scheme, and not using downloaded tags. If the format needs clean-up I also assign a "needs clean-up" tag. I enter the origin of the book in Source. Then I do a limited Metadata Download making sure not to overwrite any of the fields I've just filled in, by checking only the desired fields for download in Preferences/Metadata Download. After the Metadata Download I doublecheck the new downloaded metadata and correct it manually if necessary by finding better covers, ISBN13s, or other metadata from relevant internet sites such as Amazon, Internet Speculative Fiction Database (ISFDB), or WorldCat.

Metadata Downloads. I grab only published-date, publisher, comments, cover, and isbn (isbn is calibre-automatic). I always grab cover, even when it has an internal cover already. I keep only a few sources checked (figuring the more checked, the slower the grab.) Amazon's seemed more consistently accurate with broader item availability than others. By default I also use ISBNdb and Open Library. Others I keep unchecked and only use on a case by case basis when necessary.

Series. I tried multiple series columns but eventually went back to using just the one default series column. When it's important to get several subseries in the right order, I handle multi-level series like this: SeriesUniverseAbbreviation; Series Name (a); SubseriesName. For example for Star Wars, as follows. SW; Clone Wars (b); SubseriesName. It's easy to search on "SW;" while ANDing any other desired keywords. If I use (a)'s, (b)'s, etc correctly, sorts by series come out in chronological order per original pubdate or recommended reading order, whichever I initially preferred it to be. But for most books in most series I don't worry or care about all that, and just use the smallest/lowest level subseries name. Sometimes I use the broadest series name only, put in reading order, such as "Valdemar" but I prefer to do that only when I'm certain of reading order and a series is complete. When a series is up to date, in Tags I add "%su" which means none missing and up to date through the most recent series member in the library - a particularly useful tag in the case of a multi-author series. For multi-author series I use author name rather than seriesname in author column; if I want to see all series members in a list I do a search or just sort by series.

Tags. Since beginning with calibre I gradually used the tag browser less and the search box more. At about 3 or 4 months in I stopped using a lot of columns for genre, booktype, series-status and so on, and started using the default tags column, with tag prefixes to designate tag type, accompanied by abbreviations. Example of how I use tags now on a book:

• _FormatQuality, ((GenrePrimary, (GenreSecondaries, [Type, %status, miscellaneous

• _q4, ((fn, (mgc, (ya, [om, %sma, %su, r3

• That translates to: Format good, fantasy with magic, young adult, omnibus, in multi-author series, series up to date, myRating=3stars (and I've read it, otherwise it wouldn't be rated).

Tag Abbreviations versus Human Readability. The reason I use abbreviations for tags is to see them all in a small space in one relatively narrow column without needing to scroll through a long wide column or scroll back and forth through a sequence of different columns. Some experienced users say it's better for them to use fully expanded "human readable" tags, but I've found the abbreviations work much better for me at least until I become more adept at complex searches on the fly. Another factor in my decision to use abbreviations is that I don't import outside tags from anywhere; I'd rather use my own tag scheme which is generally simpler and more consistent. For people who use tags that come in with the books or are downloaded from outside, those tags usually arrive in human-readable form as expanded rather than abbreviated words, so it makes sense for them not to use abbreviations.

Conversion Work Habits

Spoiler:

Preferred Format. I prefer ePub format (Preferences/Behavior) because it opens fast in the calibre viewer, works on my iPad without conversion, usually easily converts to Mobi format for my Kindle, and is useful for clean-up purposes.

Formats To Keep. Mobi (for my Kindle), ePub (for my iPad, initial evaluation, and possible clean-up), original incoming format (for clean-up-related or other conversion).

Formats To Read. When the device supports it, I want to read the book in it's native incoming format if it doesn't need clean-up, otherwise I'll read a conversion after any necessary clean-up.

Format Quality Rating Tag Abbreviations. I use the calibre viewer to evaluate the format quality of recently added books, and then I assign each book a Format Quality Rating tag. For example, in tag "_q4" the "_" sorts it to the beginning of the list of comma separated tags in the Tag column. The "q" reminds me that it's a format-Quality rating, and the "4" is the rating.

Format Quality. I want to achieve the highest level of format quality my skills allow, but I also want to minimize the time I spend fixing things. My clean-up skills gradually improve over time so older "cleaned" books in my library tend to have lower format quality levels than newer books. That's why I want to delay clean-ups until just before reading.

For each new added book I examine it once then assign a format quality rating tag, as well as a separate tag identifying it as needing cleaning if it needs it. In the case of a book with multiple formats each with possibly differing quality levels, that one format quality rating applies (using deduction): to the "Preferred" format I read the most if it's there, if not then the next format I read the most if it's there, if not then apply judgement on case by case basis. When I find and compare title duplicates, the main criterion I use is the format quality rating and retain the record holding the higher quality format. I may have a "_q3" sitting in the library for 6 months and then a "_q4" for that title shows up, so eventually I replace the "_q3" format with the "_q4". I rarely use anything except "_q0", "_q3", or "_q4". Doing it this way, worse formats have a chance of getting replaced by better later. That applies to anything that has something wrong with it: Advance Reader Copies, incomings in any format that have no bold/italics (usually caused by it previously having been a text format), problem formats of any kind, all of which are convenient to keep as placeholders.

My Format Quality Rating Tags.

• _q0, wishlist item, placeholder, or bad format.

• _q1, not used.

• _q2, very rare cases where it's more than minor annoyance, not fixable, but retained anyway.

• _q3, okay, readable with only minor annoyance.

• _q4, good, readable with no annoyance.

• _q5, excellent. I don't bother with this, except for a few examples.

• _q0, also I use it to color text red in Authors, Title, Series, and Tags. For bad formats, it saves me the trouble of creating an empty book or empty book placeholder format. For catalogs it indicates wishlist items.

Format Clean-Up Initial Conversion. When I want to clean-up a format, it's best to use the cleanest and least converted format as the Input Format. These are potential choices for Output Formats listed by order of my estimation of usefulness to me at this time:

• 1. RTF for fix in Open Office, Word, or other editor.

• 2. EPUB for fix in Sigil or other ePub editor.

• 3. HTMLZ for fix in any HTML editor, might be useful after I learn HTML.

• 4. PDF for fix in Acrobat or other editor, which I try to avoid.

• 5. MOBI for fix in Mobipocket Creator (for Microsoft OS users only) or other editor.

• 6. TXT or TXTZ which would lose formatting such as Bold/Italic I don't want to lose.

Format Clean-Up Conversion Sequences. Here are a few I''ve tried:

• Calibre OriginalFormat --> Calibre ePub --> Sigil fix ePub --> calibre PreferredFormat.

• Calibre OriginalFormat --> calibre RTF --> Open Office fix RTF --> calibre PreferredFormat.

• Calibre OriginalFormat --> calibre RTF --> Open Office fix ODT --> calibre PreferredFormat.

• Calibre OriginalFormat --> calibre RTF --> Word fix DOCX --> Open Office ODT --> calibre Preferred Format

• Calibre OriginalFormat --> calibre RTF --> Word fix RTF --> Open Office ODT --> calibre Preferred Format

• (Note 1: Calibre supports ODT as input format but does not support ODT as output format.)

• (Note 2: Calibre does not support Word DOC and DOCX as input or output formats.)

• (Note 3: Writer2ePub extension to Open Office is another option to generate EPUB after fix in Open Office.

Format Clean-Up Discussion.

I'm not a publisher, distributor, or editor. Any clean-up I do takes valuable time. My goal isn't to make it perfect, but to spend the least amount of time possible to make it "readable by me with as little annoyance as possible." I examine all new incoming formats for format quality, and then tag each with format quality tag. If it looks like I won't be able to clean it up in five minutes or less, I scrap it as not worth it or tag it "_q0" and add a tag for the type of format problems it has. I always work on a copy saved out of calibre rather than on the format in calibre.

I haven't worried about or cleaned up most Table of Contents (TOCs) because for most books except big omnibuses I don't use or care about TOCs. I do strip out headers, footers, and page numbers when I can without causing a lot of split paragraphs, because those annoy me by interrupting text flow on my reading devices. If I can't fix that I scrap it or code it "_q0". I'm comfortable in Word so I had been using the conversion sequence discussed above that includes Word DOCX. The conversions from RTF to DOCX to ODT each reduced size considerably. During this next iteration of calibre use, I want to reduce the number of conversions and simplify that process so I'm now looking to learn better and simpler sequences. I will switch to using the simpler fix in Open Office sequence starting now. After I'm more sophisticated using regex, I'll switch to using calibre's conversion search and replace to remove headers, footers, and page numbers. Once I learn enough HTML to be comfortable, I will switch to doing clean-ups using the simplest path available, either Sigil or HTML editor. I'm making it a high priority to learn Sigil and HTML.

I want to reduce the number of conversions for several reasons. Save time. Simplify workflow. And most important, achieve higher quality of format. Like photocopying copies of copies of copies, or successively converting audio files through "lossy" compressions and file types, each step loses more formatting or content information while introducing more errors. Experts in conversion highly recommend minimizing the number of conversions and starting with as original and "unlossy" a format as possible.

I leave calibre conversion settings at default until I've tested a setting enough to know what it does.

Devices Work Habits

Spoiler:

Device Choice for Type of Reading. I've noticed different reading devices have different characteristics and capabilites (such as weight, color display or not) that make them suitable for different types of reading. Generally I use Kindle for fiction and iPad for technical and graphics reading. Due to my iPhone's display being small, I'm not comfortable using it for long-term reading.

Conversion Before Device Loading. I want to delay conversion until just before reading, convert on-the-fly to whatever format is necessary for the desired device. Preferred conversion source format is the original incoming format that had been retained after adding to calibre, if it didn't need fixing, otherwise a cleaned-up format.

Device Loading. I load my devices only 2 or 3 books at a time then when finished reading, delete off the device using calibre directly (preferred if possible), directly from iTunes (haven't tried that), or directly from device. After reading I assign a content rating in calibre metadata, then decide if there's even a slight chance I'll read it again. If not, I'll delete it from my library unless it can be useful as a placeholder.