KISS for New calibre Users

[unboggling]

Originally Posted by Manichean

Not that I know of.

Edit2: Never mind, I think I can do this mostly in 1 posted document, just marking where things are more advanced topics. If this thread gets very long, okay with me.

Eventually there may be one or more stickies, in various places. I'm working on the one for just-starting-out new calibre users, that eventually would be in the main calibre forum stickie section. I can see there maybe eventually being one on metadata workflow and topics in Library Management subforum. And maybe eventually one on conversion workflow and topics in the Conversion subforum.

During the drafting, comments integration, and revisions of those, none of them should be stickies until refined and stable enough. But maybe they should be their own threads?

Edit1: Or is it better to just do it all in one thread, here? This thread is going to get huge if I (and we) do it all here.

[unboggling]

This is the main message (and tests the sig too): Please avoid the newest KISS version posted 2011-08-26 in Calibre Forum until a major revision is posted next week. Or read it skeptically followed by the subsequent discussion after that post particularly on strategies and workflows. KISS for New calibre Users Post #154

[cjallan]

Hi unboggling,

I read through your earlier posts without finding the answer to my questions, so now will read this one #154...

But in case my question still isn't there, can you direct me where to look for these?

I am using Calibre to convert to MOBI format... to be read by the Kindle.

I use Word, converted to HTML, filtered, and I am quite pleased with my results, with just two exceptions:

My input file has first line indents for all paragraphs except the first in every Chapter.

According to the typographical convention I am following, the first line of every paragraph should be flush left, not indented.

But, Calibre ignores my flush left first line, and adds an indent for every paragraph.

My other question is... how do I tell Calibre where "Start" is?

Currently, my Calibre MOBI file (when read with Kindle) opens to the second page, even though I have placed a Start tag on the Forward page in my HTML code.

I'm not having any luck trying to search the "help" files...

Can you direct me to the right page to find this information?

Thanks!

[unboggling]

Originally Posted by cjallan

I am using Calibre to convert to MOBI format... to be read by the Kindle….

My input file has first line indents for all paragraphs except the first in every Chapter.

According to the typographical convention I am following, the first line of every paragraph should be flush left, not indented.

But, Calibre ignores my flush left first line, and adds an indent for every paragraph.

Hi cjallan. Try using calibre to convert it HTML to MOBI or MOBI to MOBI with preference setting Look&Feel tab checked for "Insert blank line between paragraphs." Edit1: If that doesn't work or you already have a blank line between paragraphs, try it again with that unchecked or checked (I'm not sure both checked does anything) and with checked "Remove spacing between paragraphs." "Insert blank line between paragraphs" usually sets it to flush. If that doesn't work try changing Look&Feel tab paragraph indent size to zero. If none of that works, uncheck the paragraph line settings and try: Heuristic tab check yes Heuristic processing which may cause other problems. If that doesn't work I'm stumped, report back and someone will be along eventually who knows the right answer. Or ask in Kindle forum linked below.

Originally Posted by cjallan

My other question is... how do I tell Calibre where "Start" is?

Currently, my Calibre MOBI file (when read with Kindle) opens to the second page, even though I have placed a Start tag on the Forward page in my HTML code

I'm not having any luck trying to search the "help" files...

Can you direct me to the right page to find this information?

Thanks!

I don't know enough HTML yet to even try to answer that one, but you might get a quicker response asking again in the eBook Readers forum (section above the calibre forum on the main Forums page, in the Kindle sub-forum). Edit2: Oops, link didn't work, this does: https://www.mobileread.com/forums/forumdisplay.php?f=140.

Or someone may be along here in awhile who knows the answer.
Please report back and let me know how it goes.

Edit: Oh, BTW, if the Kindle sub-forum doesn't help, the calibre Conversion sub-forum might know those answers too. And "Remove spacing between paragraphs" adds whatever indent is set in the paragraph indent box in Look&Feel tab, while "Add spacing between paragraphs" removes that I think. Both globally on all paragraphs. So for how to have both flush and indented in same book in same conversion process, I'm stumped. Edit: Sorry I didn't think earlier to put this paragraph at the beginning.

[cjallan]

Thanks very much, unbloggling,

I'll check out those sources.

btw... I've been writing user manuals for many years, and still have the same problem you were expressing a few messages back... how to explain enough so the newbie understands without driving the experienced user nuts with too much detail.

[unboggling]

Originally Posted by cjallan

Thanks very much, unbloggling,

I'll check out those sources.

btw... I've been writing user manuals for many years, and still have the same problem you were expressing a few messages back... how to explain enough so the newbie understands without driving the experienced user nuts with too much detail.

Thanks, that helps. And btw, I added some edits to my first response to your questions, in case you missed those.

[bookfanmd]

Originally Posted by jackie_w

Well done for producing an articulate and nicely put together thread. What a pity that someone tried to spoil it.

I'd like to offer the following thoughts for those new to ebook reading.

• Don't be in too much of hurry to convert all your books. When I started 2 years ago with my Sony 505 I spent a lot of time creating LRFs...
  
  Then I realised that this was going to be a dead end so I spent a lot of time re-creating them as epubs...
  
  Time passed and I realised I had got better at conversions and I was irritated that my early epubs weren't as good as my later epubs...
  
  Then I learned about epub font customisation so I changed my epub conversion settings again... then I installed PRS+ for the Sony and realised that I could achieve font customisation without having to do anything special with Calibre settings...
  
  Then Kovid introduced the 'smarten punctuation' conversion setting and I was irritated that my early conversions didn't have curly quotes ... oh and the covers on recent conversions now looked better than they used to... and on and on it goes.
  
  This probably says a lot more about my obsessiveness than anything else but the point I'm trying to make is that I wasted a lot of time converting that would have been better spent learning (or even just reading). I now only do a conversion just before I'm going to actually read the book. Then I know it's going to look the best I can make it.
  
  And even if you feel you absolutely must convert every book in your library, whatever you do don't delete the original source. Just because commercial PDFs are a PITA now doesn't mean that in a few years technology won't have moved on and they'll look great on your future reader. Also, conversions of conversions of conversions... may not be as good as conversions from an original source (a bit like photcopying).
  
  
• Secondly, there appear to be two ends of the spectrum for ereader users,
  - those who, on the whole, just want to read and only really care that the words are in the right order with no pages missing,
  - those who insist on very precise requirements from their ebooks, nested TOCs, curly quotes, mdashes, correct indents, line-spacing, fonts, margins, covers, small-caps, dropcaps etc.
  
  My husband is very much in camp 1. I honestly thought that I was too. However I very soon realised I was deluded. Much against my better judgement, I'm in camp 2 and at the time I didn't have the basic technical skills to handle it.
  
  There isn't a problem with being in either camp. But, if you're in camp 2, you have to accept that the current state and diversity of ebooks and readers is such that you need to be conversant with HTML and CSS -- whether you like it or not. There's no point coming on forums like this complaining that Calibre should be easier, more intuitive, have more/less features... Manually tweaking ebooks to your very exacting requirements requires some knowledge. Either roll your sleeves up and acquire it or learn to be less picky. Ranting on this forum that you don't have the time/inclination to educate yourself and that Calibre should do it all for you is a waste of energy, and rather insulting to the developers.

Anyway, that's more than enough from me

Jackie_W,

I'm also in camp 2 and can't really enjoy the book if there are too many formatting issues. Could you please elaborate on the "epub font customisation" and how to use it? I've looked in the documentation but haven't found anything or if I did it was over my head.

I have a lot of old lit files I'd like to convert but many of them have ugly fonts. I've had better results converting to mobi first and then epub but it would be great to be able to set the font the first time it's converted. I've recently started adding the metadata to the beginning of the book when converting and my epub metadata is always in san-serif which is not pleasing to my eye.

I love calibre but I'm very far from understanding all the options.

Thanks.

[unboggling]

Hi, bookfanmd. Not sure when jackie_w will be back through here so I'll take a shot at answering your question. Sounds to me like you want to learn about CSS - Cascading Style Sheets to be applied during conversion. So do I, so I can't answer in detail except to refer you places.

You might ask your question again in calibre's Conversion SubForum and ask for some help with CSS in relation to your conversion from lit format and font problem.

Also there's a link to w3schools, a place with CSS tutorials here.

Another thing would be to do a search of the entire Mobile Read forum, using the search menu in the blue bar near top of page, on CSS, which should result in a list of CSS posts and maybe even a wiki entry or two.

[bookfanmd]

I'll check those out. Thanks.

[jackie_w]

Originally Posted by bookfanmd

Jackie_W,

I'm also in camp 2 and can't really enjoy the book if there are too many formatting issues. Could you please elaborate on the "epub font customisation" and how to use it? I've looked in the documentation but haven't found anything or if I did it was over my head.

I don't want to hijack unboggling's thread so I'll restrict my reply to general info which may be of use to other new users. Much of it is not Calibre-related so apologies if it's not appropriate here. Nevertheless, Calibre can play a part in customising epub fonts.

There are many threads in the MR forum discussing this subject.
This is the one I used to get started - unfortunately it's quite large, start at post #9.
This post may be easier to digest. It contains one user's step-by-step guide in a PDF attachment.

Reading epubs in the font of your choice is not actually as difficult as many people imagine. Any reader which uses an Adobe epub renderer requires only 2 things:

• the actual font files (.ttf or .otf) for the font you want to use, e.g CharisSIL.
  There are usually 4 of them, one each for rendering Regular, Italic, Bold and BoldItalic text.
• a bit of extra css, which specifies which font files you want to use and tells the reader where to find them.
  The extra css is a series of @font-face{} statements, one for each of the font files, i.e. usually 4.

There are 3 ways to achieve epubs-with-customized-fonts. Each has its pros and cons:

1. Embedding. Font files are copied into the epub, @font-face css is added to the epub's existing css file.
   
   This method is often used by publishers for DRM'd retail epubs. But anyone can do it for an epub not locked by DRM. There is no Calibre feature to do this (probably Sigil can, but I don't use it, so not sure)
   
   Pros
   - epub should work on any reader which uses the Adobe renderer.
   Cons
   - non-DRM epubs only
   - The font files would be contained in every single epub. Some (e.g. CharisSIL) are quite large. This can add up to a lot of disk space if you keep thousands of books on the reader.
   - You have to manually edit each epub to embed font files, extra @font-face css and make some changes to the epub's .opf file.
   
   
2. Font files are copied to the reader, @font-face css is placed in a specially-named .css file on the reader.
   
   Unfortunately, to date, this requires the firmware on your reader to be slightly modified to recognise the .css filename containing the @font-faces. This is because Adobe has seen fit to hide its default css file deep inside the firmware where users can't get at it. For Sony readers, the PRS+ customised firmware project provides the modification.
   
   Pros
   - Can be used for all epubs whether they have DRM or not.
   - You only need to copy the font files and @font-face css to your reader once.
   - You don't need to do anything special in Calibre, nor do you need to edit each epub.
   Cons
   - You have to 'hack' your reader firmware. In my own experience, this was not actually as scary as it sounds.
   
   
3. A kind of hybrid. Font files are copied to the reader, @font-face css added to each epub's css file.
   
   You can add the @font-face css into each epub's css file manually using Calibre's Tweak-epub feature (and probably Sigil). However, you can also do it with a Calibre conversion to epub, by copying the @font-face css into Calibre's Convert - Look&Feel - ExtraCSS box.
   
   Pros
   - no hacking required
   - You only need to copy the font files to your reader once.
   - If you add the @font-face css to Calibre Preferences, you only need to do it once.
   Cons
   - non-DRM epubs only
   - Because the @font-face css is slightly different for each brand of reader, you cannot use the same epub on different readers, e.g. an epub prepared for a Sony will not display correctly on a PocketBook, and vice versa. Let me re-phrase that, the same epub will be perfectly readable on both devices, but you'll only get the customised fonts on one of them. This will only matter if you own multiple readers of different brands.

Caveat: Whichever method you use, you cannot 100% guarantee that it will work first time for every epub because it depends on the rest of the contents of the epub's css file, which in turn will depend on the quality of the source file used to create the epub. If it doesn't work it is usually because the original css contains one or more font-family statements which are overriding yours. However, I've never yet found an epub which could not be Calibre-tweaked in a few minutes so that it does work. The better you understand the 'cascading' nature of CSS the easier it is to epub-tweak-to-taste

[unboggling]

Originally Posted by jackie_w

….Whichever method you use, you cannot 100% guarantee that it will work first time for every epub because it depends on the rest of the contents of the epub's css file, which in turn will depend on the quality of the source file used to create the epub. If it doesn't work it is usually because the original css contains one or more font-family statements which are overriding yours. However, I've never yet found an epub which could not be Calibre-tweaked in a few minutes so that it does work. The better you understand the 'cascading' nature of CSS the easier it is to epub-tweak-to-taste

Wow, jackie_w.

Thank you for another wonderful post. You've started me on my CSS education with a great post on CSS with calibre that covers info and methods it would probably have taken me a year to find out on my own if not longer in a manner I could grasp in one reading. I'm impressed.

And I'm glad to find out I wasn't wrong about pegging bookfanmd's concern as a CSS area. I hope bookfanmd comes back through here soon to see your response.

Thanks again.

[bookfanmd]

Thanks Jackie_W! I printed off the pdf and will try to get method 3 working this weekend.

Bookfanmd

[cjallan]

Originally Posted by unboggling

Hi cjallan. Try using calibre to convert it HTML to MOBI or MOBI to MOBI with preference setting Look&Feel tab checked for "Insert blank line between paragraphs." Edit1: If that doesn't work or you already have a blank line between paragraphs, try it again with that unchecked or checked (I'm not sure both checked does anything) and with checked "Remove spacing between paragraphs." "Insert blank line between paragraphs" usually sets it to flush. If that doesn't work try changing Look&Feel tab paragraph indent size to zero. If none of that works, uncheck the paragraph line settings and try: Heuristic tab check yes Heuristic processing which may cause other problems. If that doesn't work I'm stumped, report back and someone will be along eventually who knows the right answer. Or ask in Kindle forum linked below.
I don't know enough HTML yet to even try to answer that one, but you might get a quicker response asking again in the eBook Readers forum (section above the calibre forum on the main Forums page, in the Kindle sub-forum). Edit2: Oops, link didn't work, this does: https://www.mobileread.com/forums/forumdisplay.php?f=140.

Or someone may be along here in awhile who knows the answer.
Please report back and let me know how it goes.

Edit: Oh, BTW, if the Kindle sub-forum doesn't help, the calibre Conversion sub-forum might know those answers too. And "Remove spacing between paragraphs" adds whatever indent is set in the paragraph indent box in Look&Feel tab, while "Add spacing between paragraphs" removes that I think. Both globally on all paragraphs. So for how to have both flush and indented in same book in same conversion process, I'm stumped. Edit: Sorry I didn't think earlier to put this paragraph at the beginning.

Well... this is probably the right place, but I didn't quite figure out what was supposed to happen.

There is a field that displays what indent it will apply to paragraphs, but it is dimmed out unless I check BOTH add space between paragraphs and remove space between paragraphs.

I worked with it for some time without figuring out what it was supposed to do.

All I wanted to do was make Calibre quit over-riding my first line indents.

I finally checked both "add space" between paragraphs and "remove space" between paragraphs, and changed the indent value to 0...

That resulted in block paragraphs with space between them and no first line indents.

That wasn't what I wanted, but it did the job. I just changed my format to "block paragraphs with space between" instead of "first line indented paragraphs with no space between.

Thanks,

CJ

[unboggling]

@cjallan,
When I answered your first post here I was tired and shouldn't have responded because what I said wasn't relevant to your concern re first paragraphs following chapter breaks flush, and the others indented getting messed up by conversion. Sorry. (I should've gone to sleep, without answering, till the next day.)

Now that I think about it more clearly, it sounds like a structure detection problem related to headers (h1, h2, h3 etc) at chapter breaks. There may be a direct way using calibre to handle it, in the structure detection tab of conversion dialog box, to include treatment of the following para's indent as a special case as part of the chapter detection during Conversion. I've never messed with that personally and aren't a conversion expert, and I don't work with MOBIs much either. So I don't know.

If you're okay with the way it looks now, there may be no need, but if you want to pursue it, you could ask more about the issue in the calibre conversion subforum.

[unboggling]

---

Link to latest version: Workflow with Examples for New calibre Users, Version 0.90, 2011-09-24, ThreadPost #288.

---


KISS for New calibre Users, Version 0.60, 2011-09-03

Keep it simple. Keep your calibre use as simple as possible to meet your specific needs for managing eBooks, eBook Readers, and news-feeds. I started with both eBooks and calibre early this year, and I wish I had tried harder since then to keep my use of calibre simple and focused on my specific eBook needs as I learned what those were. Finding practical examples all gathered together in one place of how other people were using calibre would have helped me, particularly if those examples were in a larger "bigger-picture" map for dealing with eBooks and calibre overall. So this is a map with some basic examples of strategy, methods, and work habits based on my own experience plus a lot of suggestions from much more experienced people across MobileRead Forums and particularly from this forum thread starting at Post #1.

Revision History

Spoiler:

Version 0.60, 2011-09-03, Post #195.


Major revisions and additions.

• Added new opening paragraph.
• Added a section New User Starting Steps, containing steps moved from Project section with added reference to Apprentice Alf and DRM plugins, and additional discussion of using the work habits overall as a map containing examples and not advice.
• Added to Calibre Overall Work Habits some detail on calibre documentation and asking questions.
• Revised and expanded Metadata section.
• Revised and expanded Conversion section.
• Revised and expanded Project section, moved KISS definition there.
• Added tools and other resources in Progress section.
• Throughout, other minor additions or revisions of content.
• Throughout, added more formatting using spoilers or list bullets.

Older Versions.

• Version 0.50, 2011-08-26, Post #154. Added Progress section.
• Version 0.40, 2011-08-23, Post #145. Restructured, added revision history and metadata workflow.
• Version 0.31, 2011-08-21, Post #119. Added formatting.
• Version 0.30, 2011-08-20, Post #91. Restructured as project at zero baselines.
• Version 0.21, 2011-08-12, Post #52. Restructured as advice accompanying work habits.
• Version 0.10, 2011-08-08, Post #1. Casually proposed tips and work habits.

New User Starting Steps

Spoiler:

These are important steps for a calibre user who's just starting and wondering what to do next after installation.


Soon:

• Read calibre's Quick-Start Guide thoroughly. It's in the library created during installation.
• Turn on (if it's off) system security software's antivirus auto-scan, and set it to exclude folders containing calibre libraries. If that causes performance problems with operating system or other applications, then get better security software. I for example like ESET Cybersecurity much better than I liked Norton security and have never had problems with ESET, which I can't say about Norton.
• Choose a "Preferred" Format in calibre/Preferences/Behavior with considerable forethought. I like EPUB for various reasons described below.
• Learn and set how to get the specific metadata desired onto current primary reading device. I give an example of how I did that for my Kindle below.
• Learn and set how to deal with Digital Rights Management (DRM) as it relates to converting eBooks to format of choice for reading device of choice. One method is to pay attention to DRM discussions on MobileRead Forums to learn more about DRM. Another method is to search for "Apprentice Alf" on the internet and read that blog, keeping in mind that DRM plugins are not supported or endorsed by MobileRead Forums or the calibre developers.

At Leisure:

• Install plugins. Calibre Forum's Plugins Sub-Forum is a good place to learn what plugins are available on MobileRead Forums. There's a sticky thread "Index of Plugins" at the top of that sub-forum describing plugins available there. These are installed through Preferences/Plugins button Get New Plugins. Alternatively, any plugins can be downloaded manually from a site and then installed using the button Load Plugin From File.
• Add custom columns after thinking carefully about what information to keep about books. The more books in the library, the more work it takes to maintain consistent information across the library.
• Learn how to color text in book records and decide what to base that coloring scheme on. This isn't quite as important as the others, but it's nice to use, and helpful in identifying at a glance whatever I want to color with variously colored text per various qualities I want to stand out.

Sometimes a Map Helps.

• Map. The Work Habits sections below contain strategy, methods, or work habits that are all offered together as an overall example of one user's map of using calibre.
• Not Advice. I'm not giving advice except the starting-steps above and to keep things as simple as possible and focused on your specific eBook needs, whatever they are. I'm not an expert in any of this. If something below doesn't meet your needs, then ignore it, don't do it, or do it a different way. Everybody likes to do things different ways, each person likes to do things a particular way once becoming familiar with something, and calibre in many areas allows accomplishing the same thing in various ways.
• Examples. The examples of how I work are provided merely as examples. Taken all together they may be useful as a map of how someone else is using calibre. That's all it's meant to be. It's not an instruction manual or user guide. Use calibre how you want to use calibre. Whatever works for you is good.

General Work Habits

Spoiler:

Security. I set my antivirus software to auto-scan all volumes, with specific settings to exclude calibre libraries from scans. That prevents the security software from causing performance slow-downs when calibre frequently accesses the book files for one reason or another. The books that I add to calibre had been scanned at download, then scanned again if they were accessed by any other application such as a compression expander or a reader application, then scanned again when calibre accessed them to copy during Add Books, so they don't need frequent scanning inside calibre's book folders.

Backups. I use backup software to automatically backup to an external drive on at at least an hourly basis. The calibre application and all associated files are on my internal disk. I have file hosting/syncing services such as DropBox but haven't used them with calibre because I didn't want another layer of complexity yet. In the future if I do use one of them with calibre, I'll continue to do an automated backup of my own rather than depending on a server owned by someone else. I've had to restore from backup disk three different times, after I made various blunders during the first six months of calibre use.

Raw Books. I keep all downloaded files that got copied by calibre when I Added them. After Adding to calibre, I put the original downloaded files in a Raw Books folder. I've found myself searching Raw Books numerous times and re-adding books into calibre for one reason or another. They have bad metadata or haven't been cleaned up but at least they are the original incoming formats; keeping them available is an insurance policy against future need. It functions as a second kind of backup, but raw.

Tangents. I want to avoid running off on complex time-consuming tangents that take me away from my main purpose, reading eBooks. I wasted time doing that merely to avoid feeling overwhelmed. For example, writing scripts to print booklist (which is later impractical for thousands of books), worrying about other database interfaces, and doing other things that had little direct bearing on reading eBooks or using calibre to manage eBooks and associated reading devices.

Automation. I like to do a process manually for awhile until I'm familiar with it before trying to automate it with scripts, regular expressions, templates, or computed columns. Attempted automation without prior innate understanding of the manual mechanics usually wastes lots more of my time than it saves. Not knowing what I was doing but trying to combine different types of automation all at once anyway led to me feeling frustrated and overwhelmed.

Calibre Overall Work Habits

Spoiler:

Mouse Tips. I pay close attention to Mouse Tips. These little boxes that come up when hovering the cursor over something contain important help messages about how calibre works. They are more up to date than the manual and tutorials due to the calibre software improving so rapidly through revisions, additions, and updates.

Help Documents. I use the Calibre Quick Start Guide, User Manual, and Tutorials as references now, but I've read them through twice (first after about one month, and then at about six months), and find I still missed things. They contain important information.

Stickies. I pay close attention to the Stickies at the top of the calibre forum, each calibre sub-forum, and other forums at MobileRead. They contain important information and many are revised frequently.

Asking Questions.

• One mistake I made early on was not asking questions right away on MobileRead in the appropriate forums, after the first few weeks when I thought I had a good handle on everything. I was mistaken.
• I try to remember to first double-check the calibre manual, the Frequently Asked Questions (FAQ) part of the manual, tutorial Help, and appropriate stickies before asking, or I risk embarrassment. Checking those before asking the question often solves the need to ask the question before asking it. Even though I've read Help documents at least twice I often realize I missed something important.
• It's worth any potential embarrassment even now for me to get a good answer and implement it right away, and I also sometimes see that other people respond saying they had the same question. I recently learned that after eight months, I hadn't ever right clicked a column heading or checked all the commands available there, and felt mildly embarrassed but very happy to learn, by asking how to right-align a column.

No One Right Way. I try to remember that there is usually more than one way to accomplish something in calibre, and that it's often true outside of calibre too. Remembering that helps prevent me from becoming overly invested in "my" way of doing things and makes it easier for me to change work habits when circumstances indicate a change would be good.

Libraries. Presently I use only one library. When I first started with calibre I used an additional library for evaluating books, working on the metadata, and cleaning books of headers and footers. Now I only add books to the one library a few at a time by author and slowly, and work on them then and there just after I add them. I try to avoid changing my library structure (adding columns then moving metadata around) but sometimes it's necessary after I learn something new or realize something I'm doing isn't working as smoothly as it could. Even then, I limit changing the structure to when I have a lot of time, and a lot of minor things or a major change on my library-structure to-do list.

Some Reasons For Just One Library.

• I'm retired and don't travel much anymore, but if I did travel a lot or wanted to use my mobile devices a lot to access books in calibre through the Content Server, then I'd only be able to access whatever is in the Library that the Content Server is looking at. There's presently no way I know of to focus that Content Server on more than one library at a time.
• Similarly, for the purpose of generating a Catalog, the Catalog can include a selection of books (or all of them) in that Library where the Catalog is created, but other books in other Libraries won't be in that Catalog.
• An instance of calibre on one computer presently can't look at more than one library simultaneously, so when I want to compare libraries visually one method is to use a Catalog to see what's in one Library and use the calibre Library View booklist for the other Library - but the catalog on device is sometimes awkward for seeing things at a glance.
• With more than one Library, it also takes more work to implement library structure changes across libraries to make them consistent.
• Finally, and maybe most important of all these points, I can't copy and paste metadata across Libraries without using Libraries/Quick Switch and search to find what I want to copy, copy it, Quick Switch again, and then search to find where I wanted to paste. So it means more time spent or more typing of metadata such as series information when adding new books unless I just copy and paste from an internet site, which might not be using the same standards for series name or whatever that I use.

Regular Expressions. Calibre uses Regular Expressions (regex) in several areas to manipulate strings of text. The only regex I used for 6 months was menu-supplied for detecting metadata while adding books by filename. I'm not yet familiar enough with regex to use the Conversion Search and Replace regex on a book format. At about seven months into using calibre I began to incorporate simple regex into my workflow using Edit Metadata in Bulk's Search and Replace in regex mode.

Templates. Calibre uses a template language to assign a column name to substitute metadata from that column. I currently use only one template. It's in Metadata Plugboards and adds series and series index to title for my Kindle.

{series}{series_index:0>2s| - | - }{title}

Catalogs. When I need to put a catalog on one of my devices, I check only "Books by Authors" in the Catalog dialog box and indicate a wishlist item or bad format with "_q0" as wishlist tag. (That means Format Quality equals zero.) Using a Catalog on a device is an easy way to compare two different Libraries simultaneously (one in calibre, the other on device in Catalog), or to look at two different parts of one Library simultaneously.

Plugins. I use these Plugins frequently: Find Duplicates, Open With, Search Internet, Extract ISBN, Count Pages. Quality Check, less frequently.

Calibre Other.

• I've not used these parts of calibre due to no need to use them yet: Fetch News, Get Books, Content Server, or Command Line Interface.
• I've not used collections in columns or on devices.
• I've not yet used these parts of calibre but want to learn and use during conversions (set in Preferences/Conversion or the convert selection dialog box): Cascading Style Sheets (CSS), Search and Replace Regular Expressions (regex).

Adding Books Work Habits

Spoiler:

Strategy.

• I'm much more discriminating now than when I started about the content and format quality of eBooks I download because it takes so much time to process them before reading.
• I'm "going slow" in adding eBooks into calibre. There are learning curves for calibre's more advanced features and for eBook formats and conversions. The more books in the library, the more work to do in applying new knowledge and skills across the entire library or libraries. Doing that for 100 books takes significantly less time than for 10,000 books.
• Because I didn't originally "go slow," and downloaded a lot more than I could read in 5 years, after 7 months of calibre use I intentionally trashed thousands of books and started from scratch, at a higher plateau on those learning curves. I am going slow now because I'm still learning and I'm still revising some of my work habits and even sometimes the library structure as I learn more.

Methods for Adding Books.

In Preferences/Add Books:

• I leave Auto-merge turned off, and later when the little dialog box comes up asking "Add Duplicates?" I always choose to add them because I generally want to see and evaluate any new duplicates of book records or even formats before any deletion, using the Find Duplicates plugin.
• I set Tags to Add: "_New" (without the quotes).

Methods for adding books depend on the source of a group of books and their filename structure.

• Reading metadata from file contents. If the eBook source provides good metadata in the formats, it's easiest for me to add them to calibre by reading that metadata from internal file rather than filename.
• Reading metadata from filename. Manually fix all the author/series/title info in the filename out in the operating system, or do it there using Regular Expressions (regex) in renaming tools or scripts by batching together files with similar structures to fix, then Add Books to calibre once filenames are fixed to match one of the Add Books by filename regexs in the menu. There is a good script for Windows users by MobileRead member penguinaka that renames a lot of different file structures. His thread is in the Library Management Sub-Forum. References and links to his regex list and his script are provided below in the Progress section's Other Resources.
• Reading metadata from filename. If I knew enough regex I could write and use successively a different regex to match each varying file structure of those books and import the appropriate metadata directly into the appropriate column during Add Books - but I don't know regex well enough yet.
• Reading metadata from filename. I could add that entire group of books as they are, as a mess, without fixing names in the OS, then fix everything selectively using regex in the Edit Metadata in Bulk Search/Replace window.

Methods for Adding Empty Books.

• I don't use the Empty Book command or Empty Book records.
• Copy to Library (on my context menu for calibre library) will copy empty books (contain no formats) to a different library.
• When I used to use two libraries, I preferred using Save To Disk followed by Add Books to a different library instead of Copy To Library for moving books between libraries because that ensured the metadata was saved to my EPUB's internal metadata fields in addition to the OPF file during the Save, while Copy to Library doesn't do that (as far as I know). Save also doesn't copy empty books (those without formats) to disk. So I created a folder containing empty text files titled Empty01 through Empty10 by author "Empty AAA". (First as text files then later converted to EPUB format and saved out to disk.) When necessary for wishlist items, I add a group of 10 of those "empty" book formats and change the metadata of one or several appropriately, assigning a tag "_q0" which indicates it's a wishlist item.

Metadata Work Habits

Spoiler:

Metadata Loss.

Spoiler:

I don't want to make any decisions that result in losing metadata until becoming aware of potential ramifications by asking about it on MobileRead or learning for myself through further experience. I don't want to simplify or streamline things to the extreme of losing data. Examples of what I did that needed considerable work later to fix:

• Deleted the articles "The", "An", and "A" from Title.
• Changed Spectra (an imprint of Bantam) to Bantam so publishers were more consistent.
• Used only one of the 3 or 4 co-authors of an anthology available, rather than taking the trouble to use all the names including the editor if available.

Custom Columns.

Spoiler:

I try to minimize my use of custom columns. These are what I use now:

• Act, for temporary working tags to batch process groups of books (comma separated text like tags).*
• Note, for variant titles, pseudonyms, misc. - used rarely for those (text, show in tag browser).*
• Source, for origin of book (text, show in tag browser).
• Pages, (integers, format for numbers {0:,}, for use by Count Pages plugin.)
• Formats, to see at a glance (computed from other columns).
• ISBN, to see at a glance (computed from other columns).

*Footnote: Act and Note also are useful occasionally as temporary storage places for column information, particularly when restructuring any other library columns.

Metadata Workflow Summary.

Spoiler:

• I download some books, and leave them either in Downloads folder or in my Raw Books/Pending folder, noting source and any other metadata I want at the time in a folder-name, filename, or a new associated text-file. I usually don't do much of that at download time except noting source in a containing folder's name.
• When I have time I want to Add them to calibre a few at a time, usually everything I have by one author in a particular Download Source Group in Pending folder, or sometimes by author including everywhere on my computers that are not yet in a calibre library. Since this part is new to me I'm not sure yet how all the details will shake out - my prior habit was to throw the originals of whatever I added into the Raw Books folder without distinguishing Pending and Calibre'd and not tracking particularly consistently by Download Source. After Adding each group, I want to remember to move those original files from Downloads folder or Pending folder to the Calibre'd folder in Raw Books. If this doesn't work I'll go back to doing it just by DownloadSource folder in Raw Books, and do away with Pending and Calibre'd.
• I do metadata for that small group of books immediately after adding. As part of doing the metadata, I convert nearly everything to EPUB except complex PDFs and I also do an evaluation of the format quality and tag that with a rating. If I decide to delay cleaning I tag it "needs cleaning," but if it looks like I can successfully fix it, I usually do initial clean-up at that point then rerate the format quality rating. If the eBook's clean-up needs are beyond my present skill level, I tag it as a wishlist/placeholder item and go looking for a better format for that title, either soon after finishing the metadata for the others or later.
• After that initial format quality evaluation (and possible cleaning) I work on filling in, downloading, and correcting the remaining metadata on that group of books for that particular author, trying to make everything good and consistent across that author and consistent with the other books in the library.
• The order of my columns doesn't match my workflow sequence exactly. I prefer to leave the columns in my preferred order for browsing books in library view. The View Manager plugin allows switching to different column orders and sorts at the click of a button. I've tried it but don't use it often because I like leaving my columns in one place after I get comfortable with my preferred order for doing everything.
• The trade-off between being out of order of an exact workflow sequence and never getting startled or confused where columns are located balances positively for me on the side of being comfortable and unconfused.
• As I learn more, the workflow sequences change more often than the preferred column order or the library structure.
• I also try to remember this rule of thumb while entering metadata in tags or columns. Generally "less is more". It's better for me to have fewer tags and columns that are relatively consistent and standard than a whole lot of tags and columns in which the information is an unstandardized and inconsistent mess. Making the metadata consistent across a library requires a lot of work, steadily increasing in time required as the library grows and I want to make changes, and the more metadata there is per record, that requires even more time. So less is more. Less time for maintaining the library. More time for reading.
• Note: I integrated some of the workflow described by kiwidude into my workflow, some I'm still testing, and some I prefer doing my own way. I put a link to his workflow posts in the Progress/Other Resources section as an alternative map with examples of how another user deals with strategy, methods with metadata, and workflows. ((Edit 2011-09-04))

Metadata Workflow Sequence in Detail.

Spoiler:

Moving from left to right, my columns are in this order: Authors, Series, Title, Tags, Act, Note, Source, Pages, Size, Formats, Published, Publisher, ISBN, Date, Rating. I don't use Modified or Languages. The sequence for editing metadata is similarly left to right, with a few exceptions in the progression from the exact order of columns.

• Enter Source. I enter the source of the book first so I don't forget. If necessary I look to see what folder I put it in at download time, where I indicated source information.
• Do conversion to EPUB. That's my preferred format. I convert everything except PDFs, which I evaluate using Acrobat Pro - set to open it automatically from double-clicking in calibre by unchecking PDF in Preferences Behavior and setting Acrobat for PDF in the operating system. For most PDFs I convert them to EPUB immediately after that initial Acrobat evaluation. That EPUB is usually a good enough quality conversion for my fiction-reading purposes, after clean-up, excepting certain PDFs noted in the Conversion section below. I would guess this would hold true for people converting most formats to MOBIs too, though I haven't tested that much.
• Do format evaluation. Including EPUBs or PDFs. Format quality evaluation is combined with getting correct authors, series, title from the book itself, as well as other metadata like editor and edition. I do this using calibre viewer for EPUBs, Acrobat for PDFs, other tools using the Open With plugin for other formats when necessary, such as Kindle Previewer for accurate rendering on display of MOBI.
• Enter format quality rating. I put it in Tags. It's defined below under Metadata/Tags/Format Quality Rating heading.
• Correct Authors.
• Correct Series Name.
• Correct Series Index. Including any omnibus information that fits into a Series Index convention, such as starting all omnibus numbers in a series with 500 and incrementing that by 1 for each omnibus, as opposed to collections by one author that might be 600 series and anthologies containing stories by many authors which might be 700 series by my defined convention. In most cases I like to have novels in series numbered from 1 to 299 without the sequence being messed up by omnibuses, collections, or anthologies in their midst.
• Correct Title. Do necessary corrections, including edition information and variant titles in parentheses as well as titles of major elements of omnibuses (Title of E1; Title of E2; Title of E3). So a search of Title field later will find that information. Temporarily the extra metadata meant for Title column goes into the Note column leaving Title pure for the purpose of metadata download, then appended to Title after the metadata download.
• Extract ISBN. I use the plugin Extract ISBN to get an ISBN when possible, facilitating a more specific and accurate metadata download.
• Do metadata download. Prior Authors and Title corrections are vital for this, and ISBN narrows it down. I do a limited Metadata Download making sure not to overwrite any of the columns I've just filled in, by checking only the desired fields for download in Preferences/Metadata Download. My download choices usually are: Published date, Publisher, Comments, Cover. Calibre grabs ISBN automatically if it's not already in the record. I always grab a cover, even when book has internal cover already. I keep only a few sources checked (figuring the more checked, the slower the grab). Amazon's seemed more consistently accurate with broader item availability than others. By default I also use ISBNdb and Open Library. Others I keep unchecked and only use on a case by case basis when needed.
• Optionally, do metadata download testing. Presently I'm testing Goodreads. When I'm done with testing Goodreads, I'll test Barnes & Noble, then Fantastic Fiction. The tags and ratings I sometimes download from Goodreads are a temporary thing while testing. I delete their tags after using them to help me decide on my own tags, and I'll probably eventually hide the ratings column and stop using Goodreads tags and ratings because Ratings needs too much screen width for me and I prefer my own tag and rating schemes. My Tags column has only Format Quality Rating and possible Needs Cleaning tag at this point except for tags from testing Goodreads.
• Enter my tags. Now I know enough from the format evaluation and downloaded metadata such as comments to decide on what tags I want to use for genre, book-type (omnibus, collection, anthology), and so on. I use my own tag scheme for those. So next I delete any temporary tags I may have downloaded unless I'm keeping them longer for testing purposes. Then I move the extra Title metadata from Note column, appending to Title.
• Repeat all of above one by one for each newly added book by that author until last one is done.
• Do final additions and corrections. I double-check all metadata for that author and add more or correct it manually if necessary by using good web sites that use relatively standard conventions across metadata categories. I get better covers, correct ISBN13s for editions, edition metadata, Series Name and Series Index, Published-dates, or other relevant metadata from relevant internet sites.
• Choosing relevant internet sites. I like Internet Speculative Fiction Database (ISFDB) best for metadata standardization and consistency (except for genre tags, where they're not) and it's great for series name standardization, awards, covers, ISBN13s. Sometimes I also use the original download site (indicated in Source), Amazon, WorldCat, or Wikipedia. Most of my books are speculative fiction so using ISFDB where possible is good and they even include some paranormal romance. For thriller or suspense novels Wikipedia is often good if it's a popular author.
• I use the plugin Search the Internet sometimes when I remember to, but often it's just as easy to go to a site navigating by browser then searching for author when I'm not looking for one specific book title.

Series Details.

Spoiler:

• I tried multiple series columns but eventually went back to using just the one default series column.
• When it's important to get several sub-series in the right order, I handle multi-level series like this: SeriesUniverseAbbreviation; Series Name (a); SubseriesName. For example for Star Wars, as follows. SW; Clone Wars (b); SubseriesName. It's easy to search on "SW;" while ANDing any other desired keywords. If I use (a)'s, (b)'s, etc correctly, sorts by series come out in chronological order per original pubdate or recommended reading order, whichever I initially preferred it to be.
• For most books in most series I don't worry or care about all that, and just use the smallest/lowest level subseries name.
• When I'm certain of reading order and a series is complete, I often change the series to the broadest series name only such as "Valdemar" or "Elves on the Road." I put those into reading order.
• When a series is up to date, in Tags I add one that means no books are missing and it's up to date through the most recent series member in the library - a particularly useful tag in the case of a multi-author series.
• For all series I use author names rather than series name in Author(s) column; if I want to see all series members in a list I do a search or just sort by Series.

Tags.

Spoiler:

Tags Discussion.

• Over time using calibre I gradually used the tag browser less and the search box more.
• I don't use tag browser partitioning and I'm trying to wean myself off the tag browser and use searches more often.
• I don't use tags in a hierarchical structure within tag type except genre in the cases of genres that are relatively new to me that are similar to one I'm familiar with. I know some Urban Fantasy but not much about Paranormal Romance, so I've been using secondary genre tags to help get a feel for their similarities and differences. So I have Paranormal or Urban Fantasy as primary genre and one or more secondaries such as Vampire, Shifter, Demon, Fae, Magic. After I'm more familiar with the paranormal romance genre in general, I'll stop using genre secondaries on those books.
• I prefer the other tags all flat structure except for the tag category prefix so I can assign more than one on the fly without thinking about it.
• At about 3 or 4 months in I stopped using a lot of columns for genre, booktype, series-status and so on, and started using the default tags column, with tag prefixes to designate tag type, accompanied by abbreviations. I'm overall happy with the way I do tags except for the myRating tag ("r3") needs a prefix symbol or better method to enable faster searching on my ratings, and Missing ISBN13 and Missing Cover need a unique prefix. I'm still thinking about what to use and what else I'd have to change to use them.

Tag Abbreviations versus Human Readability.

• I don't suggest that anyone use tags in abbreviated fashion the way I do. I like doing it this way - it's a sort of personal quirk. The abbreviated tags became a personal language-slang after a couple weeks using them, good for myself but not good for communicating tags to others who don't know that slang. But there is some reasoning behind why I started using them in the first place.
• I use abbreviations for tags to see them all in a small space in narrow columns without needing to scroll through long wide columns or scroll back and forth through a sequence of different columns when I'm browsing the booklist and don't want to do a search to limit the display of books or highlight certain ones.
• They're also helpful in that dense way in some uses for Catalog and Content Server on small devices.
• Many experienced users say it's better for them to use fully expanded "human readable" tags, but I've found the abbreviations work much better for me at least until I become more adept at complex searches on the fly. Even then I probably will continue using them.
• Another important factor is that I wasn't importing outside tags from anywhere when I began (and still don't except when testing a tag source.) I wanted to use my own tag scheme which is generally simpler and more consistent, easier for me to remember, harder for me to make mistakes with, and quicker to type in search boxes or in metadata entry.
• For people who use tags that come in with the books or are downloaded from outside, those tags usually arrive in human-readable form as expanded rather than abbreviated words, and it takes time to change, standardize, or even just delete those tags. I got tired of doing that with tags that weren't standard or very consistently applied to similar books in one quality or another.
• I didn't like using abbreviations until after the first few days using them. I originally started using them due to a series of blunders I made by focusing calibre instances on different computers at the same Library files successively, using either computer to make changes, before I realized I'd ruined the metadata in metadata.db, OPF files, and formats by doing that. By the time I tried to restore the database from OPFs they were bad too. My custom columns metadata would no longer save out consistently, while tests of default tag column did save out. So while frantic and in a hurry, I manually moved all that metadata out of custom columns into the default tags column and abbreviated things on the fly. (I didn't know enough regex yet to do it the easy way in Bulk Metadata Search and Replace.) Saved everything out. Added into a new Library with no custom columns. It worked.
• Now I love my abbreviations. I tried to give them up for a week while testing Goodreads tags. Couldn't do it. I'm addicted.

How I Use Tags on a Book.

• Tag Types: _FormatQuality, ((GenrePrimary, (GenreSecondaries, [Type, %status, miscellaneous
• Example Tags: _q4, ((pn, (vmp, (ya, [om, %sma, %su, mi, r3
• That means: Format Quality Rating = 4, Paranormal, Vampire, Young Adult, Omnibus, Series-Multi-Author, Series-Up-To-Date, missing ISBN13, myRating = 3 and I've read it.
• I use a bunch of other tag types and abbreviations where relevant, such as for awards ("@h" for Hugo), format-fix-problems ("$xHFP" for header/footer/page#), etc.

My Format Quality Rating Tags.
After evaluating the format quality of recently added books I assign each book a Format Quality Rating tag. For example, in tag "_q4" the "_" sorts it to the beginning of the list of comma separated tags in the Tags column. The "q" reminds me that it's a format-Quality rating, and the "4" is the rating.

• _q0, wishlist item or bad format - a useful placeholder.
• _q1, indicates "needs clean-up" if I delay clean-up until later, formerly I didn't use it.
• _q2, rare cases where it's more than minor annoyance, not fixable, but retained anyway.
• _q3, okay, readable with only minor annoyance.
• _q4, good, readable with no annoyance.
• _q5, excellent. I don't bother with this, except for a few examples.
• _q0, also I use it to color text red in Authors, Title, Series, and Tags. For bad formats, it saves the trouble of using an empty book or empty book placeholder format. When using catalogs or content server with devices, it indicates wishlist items.

Conversion Work Habits

Spoiler:

Conversion Settings. I leave calibre's conversion settings at default except for testing a setting enough to know what it does before implementing it on a wider scale. This applies to Preferences/Conversion Options and to Conversion dialog box that comes up before converting.

Preferred Format. I prefer EPUB format (set in Preferences/Behavior) because it opens fast in the calibre viewer, works on my iPad without conversion, usually easily converts to MOBI format for my Kindle, and is useful for clean-up purposes.

Formats To Keep.

Spoiler:

• EPUB (initial evaluation, possible clean-up, reading on iPad, conversion for Kindle). if the original incoming format converted well to EPUB, after cleaning up any format problems it has if necessary, that EPUB becomes my "master" copy to generate conversions for various reading devices when I want to read something. This works for me for most fiction books.
• Original incoming format. If it wasn't an EPUB to begin with, and it converted well to EPUB, I delete the original from calibre's book record. I still have the downloaded original copy in Raw Books out in the operating system and I keep that there.
• Original incoming format. If it didn't convert well to EPUB, I keep it in the book record alone and delete the EPUB that was generated for initial evaluation. This allows easily double-clicking it to open it in the appropriate application. For example, this happens often with PDFs with complex graphics, old image-based PDFs, newer technical PDFs with complex layouts, or many textbooks. In these cases I evaluate it and read it in its native format using the appropriate application.
• Nearly all of my book records, after updating metadata and format cleaning, contain just one format, with most as EPUB and only a few as PDF.
• Most formats that I keep in calibre are readable with only "minor annoyance" or "better" on my reading devices, once metadata has been updated and corrected and any format problems cleaned up. If a format is not readable without major annoyance, I either delete it completely or tag it as "bad format" ("_q0") and keep it as a placeholder and don't read it yet, hoping I'll find a better format for that book some day.

Format Clean-Up Initial Conversion.

Spoiler:

I want to use the cleanest and least converted format available as the conversion Input Format. Potential choices for Output Formats:

• EPUB for fix in Sigil or other EPUB editor.
• RTF for fix in Open Office, Word, or other editor.
• HTMLZ for fix in any HTML editor, might be useful after I learn HTML.
• PDF for fix in Acrobat or other editor, sometimes productive for unlocked PDFs.
• MOBI for fix in PRC/MOBI editor.
• TXT or TXTZ which would lose formatting such as Bold/Italic.

Format Clean-Up Conversion Sequences.

Spoiler:

Here are a few of the conversion sequences I've tried. I like the top 3 as easy to do. I like the top 2 for quality of results. So far. But I'm not yet comfortable enough in HTML to edit HTML directly most of the time.

• OriginalFormat --> EPUB --> Sigil fix, EPUB --> PreferredFormat (EPUB for me).
• OriginalFormat --> HTML --> Open Office fix, HTML --> PreferredFormat.
• OriginalFormat --> RTF --> Open Office fix, ODT --> PreferredFormat.
• OriginalFormat --> RTF --> Open Office fix, RTF --> PreferredFormat.
• OriginalFormat --> RTF --> Word fix, DOCX --> Open Office ODT --> Preferred Format.
• OriginalFormat --> RTF --> Word fix, RTF --> Open Office ODT --> Preferred Format.
• OriginalFormat --> RTF --> Word fix, RTF --> PreferredFormat.
• OriginalFormat --> RTF --> Word fix, HTM --> PreferredFormat.
• Note 1: Calibre supports ODT as input format but not as output format.
• Note 2: Calibre does not support Word DOC and DOCX as input or output formats.
• Note 3: Writer2ePub extension to Open Office is another option to convert to simple EPUB after fix in Open Office.

Format Clean-Up Discussion.

Spoiler:

• Extremes. Regarding fixing problem formats, there are two extremes for eBook people. One is to just want to read eBooks and not care much about the formatting and any format problems. The other is to want to make each eBook format as perfect as possible at that person's current skill level (which tends to keep increasing) sometimes generating a need to spend a lot of time going back to re-fix older books to bring them up to par across the library. But this is also the main reason I decided to go slow adding eBooks. I'm trying to sit the fence between those extremes. I don't worry about all format problems, just the ones that annoy me the most that I currently know how to fix.
• Technology Changes. That's another point to consider for how I fix books and how much time I spend doing it. Format and device technology change over time, so in three years all that work on older formats may become relatively obsolete after buying the new superduper device that supports the new improved version of an older format, or that even supports a new super-duper-best-format-ever-invented. In ten years (if not three) those old formats in the library might look pretty poor in comparison and in that case, changing 20,000 books over in an effort to upgrade them wouldn't be pleasant and probably still wouldn't compare in quality. It's like old 78 rpm audio to 45 to 33 to tape cassette to CD to mp3 (or a better lossless audio format). Listening to an audio piece that went through all those upgrades - which took a lot of work to do at each stage - is no longer satisfactory in comparison to going out and buying an audio piece online for couple dollars per track or less and listening to that.
• It's Not My Job. I'm not a publisher, distributor, or editor. I'm an eBook reader. Cleaning up eBook formats takes valuable time away from reading. My goal as a reader isn't a perfect eBook, but to spend the least amount of time to make it "readable by me with as little annoyance as possible." I examine all new incoming formats for format quality. If it looks like I won't be able to clean it up in five minutes or less, I scrap it as not worth it or tag it "_q0" and add a tag for the type of format problems it has. I always work on a copy saved out of calibre rather than on the format in calibre, and that will have to be more flexible when I learn enough regex, CSS (Cascading Style Sheets), and mechanical details of various formats to fix any problems directly during conversion, but calibre will still leave an original format unchanged automatically, renaming it to OriginalXXXX where XXXX is the format type.
• Won't Fix Everything. I haven't worried about or cleaned up most Table of Contents (TOCs) yet because for most books except big omnibuses I don't use or care about TOCs. I do strip out headers, footers, and page numbers when I can without causing a lot of split paragraphs, because those annoy me by interrupting text flow on my reading devices. Except for problem PDFs (graphics, technical), if I can't fix that I scrap it or code it "_q0" meaning "bad format, replace with better someday." If in the meantime I learn how to fix that kind of problem, I may end up fixing it rather than replacing it.
• Comfortable Ways Work. I'm comfortable in Word so I had been using the conversion sequence discussed above that includes Word DOCX. The conversions from RTF to DOCX to ODT each reduced size considerably but I'm not experienced enough with evaluating formats to know much about their resulting quality except that it "looks okay for me to read now." During this next iteration of calibre use, I want to reduce the number of conversions and simplify that process so I'm now looking to learn better and simpler sequences. I'm starting to use that simpler sequence of Open Office for clean-ups into ODT into calibre.
• Learning Better Ways Works Better. After I'm more sophisticated using regex, I'll switch to using calibre's conversion search and replace to remove headers, footers, and page numbers. Once I learn enough HTML to be comfortable, I will switch to doing clean-ups of most problems using the simplest path available, either Sigil or HTML editor or calibre conversion's Search/Replace. I'm making it a high priority to learn regex, Sigil, and HTML.
• Reduce Conversions. I want to reduce the number of conversions in sequence for several reasons. Save time. Simplify workflow. And most important, achieve higher quality of format. Like photocopying copies of copies of copies, or successively converting audio files through "lossy" compressions or types of recording media, each step loses more formatting and content information while introducing more errors. Experts in conversion highly recommend minimizing the number of conversions and starting with as original and "unlossy" a format as possible.

Devices Work Habits

Spoiler:

Device Choice for Type of Reading. I've noticed different reading devices have different characteristics and capabilities (such as weight, color display or not) that make them suitable for different types of reading. Generally I use Kindle for fiction and iPad for technical and graphics reading. Due to my iPhone's display being small, I'm not comfortable using it for long-term reading.

Conversion Before Device Loading.

• I want to delay conversion until just before reading, convert on-the-fly to whatever format is necessary for the desired device.
• Preferred conversion input format for me is the EPUB I converted to originally after Adding, or a cleaned up version of EPUB. In the case of PDFs or any other problem or complex formats, I'll try reading it without conversion first on Kindle for fiction or iPad for graphical or technical in PDF or other format the device can read, and where that doesn't work well try determining how to convert it or fix it in a way that does work well.

Device Loading. I load my devices only 2 or 3 books at a time then when finished reading, delete off the device using calibre directly (preferred if possible), directly from iTunes (haven't tried that), or directly from device. If I travelled more frequently, then I'd load more books. After reading I assign a content rating in calibre metadata, then decide if there's even a slight chance I'll read it again. If not, I'll delete it from my library unless it can be useful as a placeholder.

Project Information

Spoiler:

Project. KISS my use of eBooks. KISS my use of calibre.

Definition of KISS. I use the "verb" form of the principle "Keep It Simple Stupid" as meaning "to simplify a complex project or series of tasks in order to improve results." The word "Stupid" in the principle is not used or intended in a pejorative manner against me or anyone else. When I say "to KISS" I mean "to simplify and improve." A brief Wikipedia explanation is here. I want to KISS my use of eBooks. I want to KISS my use of calibre.

Purpose of Project. Determine strategies and methods for gathering, managing, and cleaning eBooks. Gradually learn relevant "best practices." Learn to use calibre and associated software better. Manage eBooks better to facilitate the ultimate purpose, which is reading eBooks.

Reasons to KISS. When I started out new to eBooks and calibre in January 2011, I frequently felt overwhelmed. Paper books are fundamentally different than eBooks, generating a need to determine different strategies and methods for managing and using eBooks, which I hadn't done yet. Calibre is eBook library management software that allows new users to use it in simple ways while also accommodating more advanced users with many features and complexities. I was overwhelmed at first because I didn't know much about eBooks in general and I let myself get tangled in calibre complexities. So I sidetracked into tangents before settling in. Later I noticed that the more I learned about eBooks in general and the more I consciously simplified my use of calibre, the more I was successful in managing eBooks.

Project Summary. At seven months into using calibre, I wanted some discussion on my then-current strategies, methods, and work habits, so I laid out what I was doing hoping for response as KISS for New calibre Users Post #1. After subsequent discussion I later recast the KISS version posts from "giving advice to new users" to "documenting what I'm doing, as one slightly experienced new user." I also re-started at baseline zero each for eBooks and calibre. I intend to use better strategies, methods, and practices during this next iteration cycle of my calibre use.

Baseline Configuration.

• 2011-08-14. To achieve baselines of zero: Deleted calibre application, library files, configuration directory, and all associated system files manually and with aid of de-installer on both Macs. Deleted books from all devices and Raw Books folders. Reformatted all backup drives (deleting everything quickly) and then completely backed up both computers with fresh backups.
• My devices Kindle, iPad, and iPhone are up to date with operating systems and firmware upgrades.
• My computers MacBook17 and MacBook are up to date on firmware and running ESET Cybersecurity on latest Snow Leopard OS X (will upgrade when OS X Lion is more stable). The computers, Kindle, and iPad all use my local wireless network on an Apple Airport router. The iPhone I don't use for eBooks. The Macs have firewalls set and the router has security too.
• Installed latest binary version of calibre with its one library containing one eBook as a clean new calibre installation on MacBook17 only.
• Configured preferences, basic user info.
• Added custom columns.
• Added one template in metadata plugboards.
• Added one tweak to format PublishedDate as year only.
• Left all conversion settings at default except to not convert comic book images to black and white.
• Set a tag in Look and Feel/Column Coloring to red for book's metadata to indicate a wishlist item or bad format.
• In Behavior, I selected EPUB as preferred, and I checked yes for calibre's viewer to open all listed formats.
• At that point I had only a few miscellaneous eBooks scattered around: calibre Quick-Start Guide (in calibre library), Kindle User Guide, various user guide PDF formats, and a few other technical and reference PDFs.
• That was my starting situation and baseline for gathering and managing eBooks as well as helping me move closer to a situation somewhat similar to that of a starting-out calibre user, with the addition of some experience-based perspective.

Baseline Configuration Changes.

• 2011-08-15. Behavior. Unchecked: PDF.
• 2011-08-15. Add Books. Set Tags to Add: "_New" (without the quotes).
• 2011-08-24. Tweak. Library sort order (title and series): "no".

Request Feedback and Input on KISS post revisions after each is posted. All strategies, methods, work habits, workflows are offered not as advice but as examples of what one new user is doing, struggling with, or trying to do. I hope this may be useful to other new users, increasing in usefulness over time as it is refined in successive iterations. Feedback, input, and discussion will be helpful in correcting or improving any assumptions, strategies, methods, workflow, or practices. Please post any feedback, input, or discussion at end of this thread: Post #195. That link goes toward end of thread, there may be posts after it.

Thank You. Thanks to everyone who's posted on MobileRead, which is where I learned most of the strategies, methods, tools, and much else contained in this post. Particular thanks to those of you who posted in this project development thread and helped me try to internalize, integrate, and improve all this content or format the posts better. You know who you are.

Progress

Spoiler:

eBooks Gathered

Spoiler:

For eBooks, I'm more discriminating about what I want in my library than I was previously, regarding quality of both content and format.

• Amazon. Prior purchases and a few free downloads I specifically wanted.
• Baen Free Library. A few of them I specifically wanted.
• Baen free CDs. Got all of them so I can compare between specific eBook formats, and maybe learn a bit of HTML from the HTML web versions from the most recent CDs.
• Joshua Tallent's "Kindle Formatting: The Complete Guide to Formatting Books for the Amazon Kindle," for additional information about formatting eBooks. I bought it because I thought it would also apply in general if not all specifics to formatting content for EPUB.
• Some of the User Guides and Manuals and a few odds-and-ends old PDFs lying around on my computers.
• Most of those eBooks, not yet added to my calibre Library(ies).
• Others planned in near-term: I want a few more eBooks on a highly selective basis from sites that have large catalogs such as MobileRead (MR) and Project Gutenberg (PG). The former are usually better formats than the latter.

Tools Gathered and Tools Noted

Spoiler:

For tools to use in association with calibre, I try to find highly recommended ones even if they're not popular. And good tools have their own learning curves. For any tools I haven't tried enough myself, my comment includes "Noted".

• EPUB editor. Sigil. Uses regex, has regex help. (Open source, Multi-platform, recommended).
• EPUB generator. Writer2ePub. Extension to Open Office. (Free, Multi-platform, recommended)
• HTML editor. BBedit. Great text manipulation, not free, uses regex, has regex help (Mac only, recommended). Noted and recommended on MR: TextWrangler (Mac only, free, baby brother to BBedit), Vim (open source, multi-platform, mostly keyboard driven), Notepad++ with WebEdit plugin (both open source, Windows only).
• HTML editors (IDEs). Noted and recommended on MR: Aptima Studio (based on Eclipse, multi-platform), or Eclipse with Web Tools Platform (multi-platform). These are Integrated Development Environments (IDEs) which can be intimidating to non-programmers.
• MOBI viewing. Kindle Previewer. For accurate rendering of MOBI formats on display. (Mac, Windows.)
• MOBI generator. Kindlegen. Amazon's command-line tool for generating Kindle formats from input documents (free, multi-platform.) Noted: I'd also try Mobipocket Creator (for Windows) if it were available for Mac users.
• Word Processors. I've been using MS Word for years, started this year with Open Office (open source, Multi-platform, recommended).
• Browser alternative. Firefox. Heard there are helpful plugins, haven't investigated yet. (Free, multi-platform.)
• Image manipulator. GIMP. Has a learning curve. (Open Source, multi-platform.)
• File Renamer. A Better Finder Rename, for Mac. Uses regex, has regex help. Noted and recommended on MR: Flash Renamer (Windows) looks good, and there are many others.
• Scripts and regex. File renaming script for pre-processing books before Adding to calibre is available from penguinaka (Windows). He also lists good regex for various types of filename structures to be used in file renamers. That thread is in Calibre Forum's Library Management Sub-Forum. Links are provided below in Other Resources.
• Adobe Acrobat X Pro. Not recommended for new users. Bought it recently, still learning it. It's expensive, has high learning curve, and isn't helpful in most cases for cleaning up PDFs before conversion in calibre, since many PDF books arrive "locked" to prevent anything but copying. Conversion to EPUB usually causes some problems when successful, but it's easier if time consuming to clean those after the conversion by using a word processor or some other type of editor (Open Office, Sigil, BBedit, etc).
• I just learned today that Acrobat X Pro can save out as HTML. So maybe clean-up of those PDFs would work best going Acrobat PDF --> Sigil fix HTML --> EPUB. I'll test that. ((Edit 2011-09-03))

Other Resources

Spoiler:

• CSS Introduction by jackie_w (Recommended), for getting started with Cascading Style Sheets (CSS) in KISS post #190 here. I intend to follow this up soon for learning CSS.
• Metadata Workflow by kiwidude (Recommended), as an alternative map with examples of how another user deals with metadata and its workflow, described in post here and a few posts later here. I integrated some of that into my strategy and workflow, still testing some things. ((Edit 2011-09-04))
• File Renaming Regex list, by penguinaka (Recommended), for use in a File Renamer application: here.
• File Renaming Script, by penguinaka (Noted, Recommended on MR), BookCase (Windows): here.

Plan

Spoiler:

• Add tools and resources to lists as I come across them and have tested them where possible, and at least researched them and gathered other MR users' recommendations.
• Add books very slowly (short-term mostly for testing rather than reading purposes).
• Learn CSS, Sigil, and HTML (short-term ongoing through long-term).
• Continue KISS work and revisions corresponding to my own experience, trying to integrate suggestions made by others into my use of eBooks and calibre and then into the KISS posts (short-term ongoing).