Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Closed Thread
 
Thread Tools Search this Thread
Old 08-23-2011, 09:09 AM   #151
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by unboggling View Post
Yes. I've been trying to train myself out of relying on the tag browser.
I don't use it at all. It just takes up space and I've got enough books that almost all headings in the tag browser would take far longer to scroll through than just typing part or all of the relevant tag, author, title, etc. that I'm looking for. I also like finding related books, so if I type "Asimov" in the search box, I get all books written by Isaac Asimov, but I also get books where the comments say inspired by Isaac Asimov's Caves of Steel, etc. I like finding the cross links that I wouldn't find with a perfect author search produced by clicking on the tag browser.
Starson17 is offline  
Old 08-23-2011, 10:08 AM   #152
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
Quote:
Originally Posted by Starson17 View Post
I don't use it at all. It just takes up space and I've got enough books that almost all headings in the tag browser would take far longer to scroll through than just typing part or all of the relevant tag, author, title, etc. that I'm looking for. I also like finding related books, so if I type "Asimov" in the search box, I get all books written by Isaac Asimov, but I also get books where the comments say inspired by Isaac Asimov's Caves of Steel, etc. I like finding the cross links that I wouldn't find with a perfect author search produced by clicking on the tag browser.
I like that, about wanting to see the cross links, hadn't considered it before.

Speaking of work habits, even when newer and the tag browser was vital to me, I always kept Tag Browser and Cover Browser hidden, wanting more room to see more fields at once. And the only time I opened one or the other was when I specifically needed to do something with covers or links or something useful in Tag Browser.
unboggling is offline  
Old 08-23-2011, 11:34 AM   #153
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
BTW, please take as a given that I'd like to get feedback from people setting me straight about anything I've posted containing any bad assumptions, strategies, methods, workflows, work habits, etc. If you suggest an alternative I'll be happy. And any discussion is welcome, corrective or not.

Last edited by unboggling; 08-25-2011 at 04:50 PM.
unboggling is offline  
Old 08-26-2011, 03:54 AM   #154
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
---

Link to latest version: Workflow with Examples for New calibre Users, Version 0.90, 2011-09-24, ThreadPost #288.

---


KISS for New calibre Users (v 0.50) 2011-08-26

I'm using the "verb" form of the principle "Keep It Simple Stupid" as meaning "to simplify a complex project or series of tasks in order to improve results." The word "Stupid" in the principle is not used or intended in a pejorative manner against myself or anyone else. "To KISS" means "to simplify and improve." Saying someone KISSed something has a connotation of praise. A brief Wikipedia explanation is here. I want to KISS my use of eBooks. I want to KISS my use of calibre. This is how I'm trying to do that.

KISS Post Revision History
Spoiler:
  • Version 0.50, 2011-08-26, KISS Post #154. Added "Progress" section. Moved "Project Information" section down. Other changes minor.
  • Version 0.40, 2011-08-23, KISS Post #145. Restructured. Added version history, metadata entry workflow sequence. Expanded conversion section. Other changes minor.
  • Version 0.31, 2011-08-21, KISS Post #119. Added formatting and minor changes.
  • Version 0.30, 2011-08-20, KISS Post #91. Restructured as project at zero baselines.
  • Version 0.21, 2011-08-12, KISS Post #52. Restructured as advice accompanying work habits.
  • Version 0.10, 2011-08-08, KISS Post #1. Casually proposed KISS, with tips and work habits.


Strategy for eBooks
Spoiler:

Overall. Obtain, clean-up, and read "on demand." I plan to make exceptions where warranted for good reasons such as "found a great edition that's hard to find" or "it's temporarily on sale" and "I want it enough to break the rule." But my primary purpose is to read eBooks, not to gather, clean, and hold them.

Reasons for Overall Strategy. I want to "go slow" this iteration of my calibre use for several reasons. As a new user of eBooks and calibre I'm still in steep learning curves for converting and cleaning-up eBooks. If my library grows fast, most of those books will have been previously format-cleaned at lower quality levels as I rise up the learning curves. The greater the quantity of older books there are at any point, the more time it will take to re-fix them to make them more consistent in quality across the library. Re-fixing previously fixed books adds more conversion errors and problems into the mix. So it makes sense to go slowly and build carefully at least until those steep learning curves level out toward a slow and gradual learning process.

Gathering. Browse the internet to determine what I want to read shortly - in the next month or less. Acquire books for that read-on-demand strategy as possible. Primarily by browsing the internet. Places like Amazon, Barnes & Noble, Baen, Mobile Read, and so on.

Cleaning. Lean and mean according to my skill level. Minimize multiple conversions in sequence. Retain original incoming format. Always do major clean-up work on copies rather than originals just as a matter of habitual hygiene. Calibre now automatically saves an Original format when a user tries to convert it, which is convenient.

Managing. Learn and use the calibre eBook library manager.

Learning. Learn Cascading Style Sheets (CSS) and the Sigil EPUB editor soon. Gradually learn Regular Expressions (regex). Gradually learn Hypertext Markup Language (HTML). Learn better strategies and methods on the Mobile Read website, and other websites following links or tips from Mobile Read. Start improving other eBook-related skills in an ongoing manner.

Changing Habits. All work habits described below are "proposed" habits for me to meet my needs for managing and reading eBooks. They are what I've been doing, am in the process of changing, or want to do from now on. They change after determining a new strategy or learning a new method or skill.



General Work Habits
Spoiler:

Cybersecurity. My antivirus software is set to auto-scan all volumes, with specific settings to exclude calibre libraries from scans. That prevents the security software from causing performance slow-downs when calibre frequently accesses the book files for one reason or another. The books that were added to calibre had been scanned at download, then scanned again if they were accessed by any other application such as a compression expander or a reader application, then scanned again when calibre accessed them to copy during Add Books.

Backups. I use backup software to automatically backup my internal disk on an hourly basis to an external drive. The calibre application and all associated files are on my internal disk. I have file hosting/syncing services but haven't used them with calibre because I didn't want another layer of complexity yet. In the future if I do use a file hosting/syncing service, I will continue to do an automated backup of my own rather than depending on a server owned by someone else.

Raw Books. I keep all downloaded files that got copied by calibre when I Added them. After Adding to calibre, I put the original downloaded raw eBooks in folders in this structure: Raw Books/Source/DL Group/Original Filename. DL Group refers to identifying a group of files downloaded at once, or to a broader category such as Baen Free CDs. I've found myself searching Raw Books numerous times and re-adding books into calibre for one reason or another. They have bad metadata or haven't been cleaned up but at least they are the original incoming formats; keeping them available is an insurance policy against future need. It functions as a second kind of backup, but raw.

Tangents. I want to avoid running off on complex time-consuming tangents that take me away from my main purpose, reading eBooks. I wasted time doing that merely to avoid feeling overwhelmed. For example, writing scripts to print booklist, worrying about other database interfaces, and doing other things that had little direct bearing on reading eBooks or using calibre to manage eBooks and associated reading devices.

Automation. I like to do a process manually for awhile until I'm familiar with it before trying to automate it with scripts, regular expressions, templates, or computed columns. Attempted automation without prior innate understanding of the manual mechanics usually wastes lots more of my time than it saves. Not knowing what I was doing but trying to combine different types of automation all at once anyway led to me feeling frustrated and overwhelmed.



Calibre Overall Work Habits
Spoiler:

Mouse Tips. I pay close attention to Mouse Tips. These little boxes that come up when hovering the cursor over something contain important help messages about how calibre works. They are more up to date than the manual and tutorials due to the calibre software improving so rapidly through revisions, additions, and updates.

Stickies. I pay close attention to the Stickies at the top of the calibre forum, each calibre subforum, and other forums here. They contain important information.

Workflow Timing. Using the calibre viewer I evaluate an incoming eBook's format shortly after it's added then assign a format quality rating tag. If it looks like I can successfully fix it, I delay any clean-up or conversions until shortly before reading on the reading device. If the eBook's clean-up needs are beyond my present skill level, I tag it as a wishlist/placeholder item and go looking for a better format for that title, either right away or later. After that initial format quality evaluation I work on filling in the remaining metadata.

Regular Expressions. Calibre uses Regular Expressions (regex) in several areas to manipulate strings of text. The only regex I used for 6 months was menu-supplied for detecting metadata while adding books by filename. I'm not yet familiar enough with regex to use the Conversion Search and Replace regex on a book format. At about seven months into using calibre I began to incorporate simple regex into my workflow using Edit Metadata in Bulk, Search and Replace in regex mode.

Templates. Calibre uses a template language to assign a column name to substitute metadata from that column. I currently use only one template. It's in Metadata Plugboards and adds series and series index to title for Kindle.
Code:
{series}{series_index:0>2s| - | - }{title}
Libraries. Presently I use only one library named "Main". When I first started with calibre I used an additional library named "Add" for evaluating and cleaning books and working on the metadata. Now I'm going to try doing it all in "Main" for awhile.

Catalogs. I rarely use this feature. When I do need to put a catalog on one of my devices, I check only "Books by Authors" and indicate wishlist item with "_q0" (defined in Metadata section below).

Plugins. I use these Plugins frequently: Find Duplicates, Open With, Search Internet, Quality Check. I recently added Count Pages.

Other. I've not used these parts of calibre due to no need to use them yet: Fetch News, Get Books, Content Server, or Command Line Interface. I've not used collections in columns or on devices. I've not yet used these parts of calibre but want to learn and use during conversion: CSS (style sheets), Search and Replace regex.



Adding Books Work Habits
Spoiler:

Methods. Methods for adding books depend on the source of a group of books and their filename structure.
  • Normally I leave Automerge turned off and try to set up the Add operation in a sequence that puts different formats for identical author/title matches into separate book records, because I prefer to work with different formats in separate records when necessary to evaluate which ones to delete or keep, as opposed to trying to evaluate several formats of differing qualities all crammed into one new book record.
  • Reading metadata from file contents. If the eBook source provides good metadata in the formats, it's easiest for me to add them to calibre by reading that metadata from internal file rather than filename.
  • Reading metadata from filename. Manually fix all the author/series/title info in the filename out in the operating system (OS), or do it there using renaming tools (which use regex also) or scripts by batching together files with similar structures to fix, then Add Books to calibre once filenames are fixed to match one of the Add Books by filename regexs in the menu.*
  • Reading metadata from filename. If I knew enough regex I could write and use successively a different regex to match each varying file structure of those books and import the appropriate metadata directly into the appropriate column during Add Books - but I don't know regex enough yet.
  • Reading metadata from filename. I could add that entire group of books as they are, as a mess, without fixing names in the OS, then fix everything selectively using regex in the Edit Metadata in Bulk Search/Replace window.
  • I've tried it all those ways except with renaming tools or scripts, and they all require work.
  • *Note. There are scripts available on this forum to help solve this problem, standardizing names before import. I haven't used them because I'm still learning and want to avoid scripts for awhile longer. And as far as I've noticed, the better scripts aren't available for Mac users without revision.

Empty books. I don't use Empty Book command or Empty Book records. I created a folder containing empty files (originally text, later converted to EPUB) titled Empty01 through Empty10 by author AAA, TBD. When I need to for wishlist items, I add a group of 10 "empty" book formats and change the metadata of one or several appropriately, assigning a format quality tag "_q0". The reason I do this is because empty books don't get included when I Save To Disk a selection of books out. When I want them included in saves that eventually get added to a different library, book records indicating wishlist items need to hold a format. If I want to avoid all that I can just use Copy To Library instead which does copy empty books without formats.



Metadata Work Habits
Spoiler:

Metadata Loss. I don't make any decisions that result in losing metadata, until I'm more experienced and become aware of potential ramifications. Examples of what I did that needed considerable work later to fix: Deleted the articles "The", "An", and "A" from Title. Changed Spectra (an imprint of Bantam) to Bantam so publishers were more consistent. Used only one of the 3 or 4 co-authors of an anthology where the actual editor's name wasn't available, rather than taking the trouble to use all the names. I don't want to KISS things to the extreme of losing data.

Custom Columns. I try to minimize my use of custom columns. These are what I use now (with recent addition of "Source" and "Pages"):
  • ISBN, to see at a glance (computed from other columns).
  • Formats, to see at a glance (computed from other columns).
  • Act, for temporary working tags to batch process groups of books (comma separated text like tags).
  • Notes, for variant titles, pseudonyms, miscellaneous (text, show in tag browser). I use it rarely.
  • Source, for origin of book (comma separated text like tags).
  • Pages, (integers, format for numbers "{0:,}" for use by Count Pages plugin.)

Workflow Sequence. For newly added books, the workflow order for editing columns can be important. Moving from left to right, my columns are in this order: Authors, Title, Series, Tags, Act, Notes, Source, Pages, Size, Formats, Publisher, Published, ISBN, Date. I don't use Modified, Rating, or Languages. The sequence for editing metadata is similarly left to right. I correct Authors and Title first because they are vital later for Metadata Download. I get series information by using browser at relevant sites. I then evaluate the format and learn enough about the book to assign tags into Tags, using my own tag scheme, and not using downloaded tags. If the format needs clean-up I assign a "needs clean-up" tag. I enter the origin of the book in Source. Then I do a limited Metadata Download making sure not to overwrite any of the fields I've just filled in, by checking only the desired fields for download in Preferences/Metadata Download. After the Metadata Download I doublecheck the new downloaded metadata and correct it manually if necessary by finding better covers, ISBN13s, or other metadata from relevant internet sites such as Amazon, Internet Speculative Fiction Database (ISFDB), or WorldCat.

Metadata Downloads. I grab only Published (date), Publisher, Comments, Cover, and ISBN (ISBN is calibre-automatic). I always grab a cover, even when it has an internal cover already. I keep only a few sources checked (figuring the more checked, the slower the grab). Amazon's seemed more consistently accurate with broader item availability than others. By default I also use ISBNdb and Open Library. Others I keep unchecked and only use on a case by case basis when necessary.

Series. I tried multiple series columns but eventually went back to using just the one default series column. When it's important to get several subseries in the right order, I handle multi-level series like this: SeriesUniverseAbbreviation; Series Name (a); SubseriesName. For example for Star Wars, as follows. SW; Clone Wars (b); SubseriesName. It's easy to search on "SW;" while ANDing any other desired keywords. If I use (a)'s, (b)'s, etc correctly, sorts by series come out in chronological order per original pubdate or recommended reading order, whichever I initially preferred it to be. But for most books in most series I don't worry or care about all that, and just use the smallest/lowest level subseries name. Sometimes I use the broadest series name only, put in reading order, such as "Valdemar" but I prefer to do that only when I'm certain of reading order and a series is complete. When a series is up to date, in Tags I add "%su" which means none missing and up to date through the most recent series member in the library - a particularly useful tag in the case of a multi-author series. For all series I use author(s) names rather than series name in Authors column; if I want to see all series members in a list I do a search or just sort by Series.

Tags. Over time using calibre I gradually used the tag browser less and the search box more. I don't use tag browser partitioning and I'm trying to wean myself off the tag browser and use simple searches more. I don't use tags in a hierarchical structure, I prefer them all flat so I can assign more than one on the fly without thinkling about it. At about 3 or 4 months in I stopped using a lot of columns for genre, booktype, series-status and so on, and started using the default tags column, with tag prefixes to designate tag type, accompanied by abbreviations. Example of how I use tags now on a book:
  • _FormatQuality, ((GenrePrimary, (GenreSecondaries, [Type, %status, miscellaneous
  • _q4, ((fn, (mgc, (ya, [om, %sma, %su, r3
  • Translation: Format good, fantasy with magic, young adult, omnibus, in multi-author series, series up to date, myRating=3stars (and I've read it, otherwise it wouldn't be rated).
  • I use a bunch of other tag types and abbreviations where relevant, such as for awards, format-fix-problems, etc.

Tag Abbreviations versus Human Readability. I use abbreviations for tags to see them all in a small space in narrow columns without needing to scroll through long wide columns or scroll back and forth through a sequence of different columns. Some experienced users say it's better for them to use fully expanded "human readable" tags, but I've found the abbreviations work much better for me at least until I become more adept at complex searches on the fly. Another factor in my decision to use abbreviations is that I don't import outside tags from anywhere; I'd rather use my own tag scheme which is generally simpler and more consistent. For people who use tags that come in with the books or are downloaded from outside, those tags usually arrive in human-readable form as expanded rather than abbreviated words, and it takes time to change or standardize those tags.

Format Quality Rating Tag Abbreviations. I use the calibre viewer to evaluate the format quality of recently added books (except I use Acrobat for PDFs), and then I assign each book a Format Quality Rating tag. For example, in tag "_q4" the "_" sorts it to the beginning of the list of comma separated tags in the Tag column. The "q" reminds me that it's a format-Quality rating, and the "4" is the rating.

My Format Quality Rating Tags.
  • _q0, wishlist item, placeholder, or bad format.
  • _q1, indicates "needs clean-up" since I'm now delaying clean-ups, formerly I didn't use it.
  • _q2, rare cases where it's more than minor annoyance, not fixable, but retained anyway.
  • _q3, okay, readable with only minor annoyance.
  • _q4, good, readable with no annoyance.
  • _q5, excellent. I don't bother with this, except for a few examples.
  • _q0, also I use it to color text red in Authors, Title, Series, and Tags. For bad formats, it saves the trouble of creating an empty book or empty book placeholder format. For catalogs it indicates wishlist items.


Conversion Work Habits
Spoiler:

Preferred Format. I prefer EPUB format (Preferences/Behavior) because it opens fast in the calibre viewer, works on my iPad without conversion, usually easily converts to MOBI format for my Kindle, and is useful for clean-up purposes.

Formats To Keep. EPUB (for my iPad, initial evaluation, and possible clean-up), MOBI (for my Kindle), original incoming format (for clean-up related or other conversion).

Formats To Read. When the device supports it, I want to read the book in it's native incoming format if it doesn't need clean-up, otherwise I'll read a conversion after any necessary clean-up.

Format Quality. I want to achieve the highest level of format quality my skills allow, but I also want to minimize the time I spend fixing things. My clean-up skills gradually improve over time so older "cleaned" books in my library tend to have lower format quality levels than newer books. That's why I want to delay obtaining new books and cleaning them up until before reading, as shortly before as feasible.

Format Evaluation. For each new added book I examine it once then assign a format quality rating tag. In the case of a book with multiple formats each with possibly differing quality levels, that one format quality rating applies (using deduction): to the "Preferred" format I read the most if it's there, if not then the next format I read the most if it's there, if not then apply judgement on case by case basis. When I find and compare book duplicates, the main criterion I use is the format quality rating and retain the record holding the higher quality format. I may have a "_q3" sitting in the library for 6 months and then a "_q4" for that title shows up, so eventually I replace the "_q3" format with the "_q4". I rarely used anything except "_q0", "_q3", or "_q4" and now that I'm delaying cleaning I'll start using "_q1" to indicate it needs cleaning. Doing it this way, worse formats have a chance of getting replaced by better later. That applies to anything that has something wrong with it: Advance Reader Copies, incomings in any format that have no bold/italics (usually caused by it previously having been a text format), problem formats of any kind, all of which are convenient to keep as placeholders.

Format Clean-Up Initial Conversion. I want to use the cleanest and least converted format available as the conversion Input Format. Potential choices for Output Formats:
  • RTF for fix in Open Office, Word, or other editor.
  • EPUB for fix in Sigil or other EPUB editor.
  • HTMLZ for fix in any HTML editor, might be useful after I learn HTML.
  • PDF for fix in Acrobat or other editor, sometimes productive for unlocked PDFs.
  • MOBI for fix in PRC/MOBI editor.
  • TXT or TXTZ which would lose formatting such as Bold/Italic.

Format Clean-Up Conversion Sequences. Here are a few I''ve tried:
  • OriginalFormat --> EPUB --> Sigil fix EPUB --> PreferredFormat (EPUB for me).
  • OriginalFormat --> RTF --> Open Office fix RTF --> PreferredFormat.
  • OriginalFormat --> RTF --> Open Office fix ODT --> PreferredFormat.
  • OriginalFormat --> RTF --> Word fix DOCX --> Open Office ODT --> Preferred Format.
  • OriginalFormat --> RTF --> Word fix RTF --> Open Office ODT --> Preferred Format.
  • OriginalFormat --> RTF --> Word fix RTF --> PreferredFormat.
  • Note 1: Calibre supports ODT as input format but not as output format.
  • Note 2: Calibre does not support Word DOC and DOCX as input or output formats.
  • Note 3: Writer2ePub extension to Open Office is another option to convert to simple EPUB after fix in Open Office.

Format Clean-Up Discussion.

I'm not a publisher, distributor, or editor. Cleaning up eBook formats takes valuable time. My goal as a reader isn't a perfect eBook, but to spend the least amount of time to make it "readable by me with as little annoyance as possible." I examine all new incoming formats for format quality. If it looks like I won't be able to clean it up in five minutes or less, I scrap it as not worth it or tag it "_q0" and add a tag for the type of format problems it has. I always work on a copy saved out of calibre rather than on the format in calibre, but that will have to be more flexible when I learn enough regex to fix during conversion.

I haven't worried about or cleaned up most Table of Contents (TOCs) because for most books except big omnibuses I don't use or care about TOCs. I do strip out headers, footers, and page numbers when I can without causing a lot of split paragraphs, because those annoy me by interrupting text flow on my reading devices. If I can't fix that I scrap it or code it "_q0". I'm comfortable in Word so I had been using the conversion sequence discussed above that includes Word DOCX. The conversions from RTF to DOCX to ODT each reduced size considerably. During this next iteration of calibre use, I want to reduce the number of conversions and simplify that process so I'm now looking to learn better and simpler sequences. Starting now, I will switch to using that simpler sequence of Open Office for clean-ups into ODT into calibre. After I'm more sophisticated using regex, I'll switch to using calibre's conversion search and replace to remove headers, footers, and page numbers. Once I learn enough HTML to be comfortable, I will switch to doing clean-ups using the simplest path available, either Sigil or HTML editor or calibre conversion's Search/Replace. I'm making it a high priority to learn regex, Sigil, and HTML.

I want to reduce the number of conversions for several reasons. Save time. Simplify workflow. And most important, achieve higher quality of format. Like photocopying copies of copies of copies, or successively converting audio files through "lossy" compressions and file types, each step loses more formatting and content information while introducing more errors. Experts in conversion highly recommend minimizing the number of conversions and starting with as original and "unlossy" a format as possible.

I leave calibre conversion settings at default until I've tested a setting enough to know what it does.



Devices Work Habits
Spoiler:

Device Choice for Type of Reading. I've noticed different reading devices have different characteristics and capabilites (such as weight, color display or not) that make them suitable for different types of reading. Generally I use Kindle for fiction and iPad for technical and graphics reading. Due to my iPhone's display being small, I'm not comfortable using it for long-term reading.

Conversion Before Device Loading. I want to delay conversion until just before reading, convert on-the-fly to whatever format is necessary for the desired device. Preferred conversion input format is the original format that first arrived in calibre, if it didn't need fixing, otherwise a cleaned-up format.

Device Loading. I load my devices only 2 or 3 books at a time then when finished reading, delete off the device using calibre directly (preferred if possible), directly from iTunes (haven't tried that), or directly from device. After reading I assign a content rating in calibre metadata, then decide if there's even a slight chance I'll read it again. If not, I'll delete it from my library unless it can be useful as a placeholder.



KISS Project Information
Spoiler:

Project. KISS my use of eBooks. KISS my use of calibre.

Purpose of Project. Determine strategies and methods for gathering, managing, and cleaning eBooks. Gradually learn relevant "best practices." Learn to use calibre and associated software better. Manage eBooks better to facilitate the ultimate purpose, which is reading eBooks.

Baseline. My baseline is similar to what a brand new user to eBooks and calibre might have after installing calibre and then accomplishing these important steps.
  • Soon, read Quick-Start Guide thoroughly.
  • Soon, set system security auto-scan antivirus to exclude calibre libraries.
  • Soon, choose a "Preferred" Format in calibre/Preferences/Behavior with considerable forethought.
  • Soon, learn and set how to get the specific metadata desired onto current primary reading device.
  • Soon, learn and set how to deal with Digital Rights Management (DRM) as it relates to converting eBooks to format of choice for reading device of choice.
  • At leisure, install plugins and add custom columns.
  • At leisure, learn how to color text in book records and what to base that coloring scheme on.

Request Feedback. All strategies and work habits are proposed and are not offered as advice, but as examples of what one new user is doing, struggling with, or trying to do. I hope this may be useful to other new users, increasing in usefulness over time as it is refined in successive iterations. Feedback and discussion will be helpful in correcting or improving any assumptions, strategies, methods, workflow, or practices. Please post feedback and discussion in this thread "KISS for New calibre Users" or send me a private message.

Reasons to KISS. When I started out new to eBooks and calibre in January 2011, I frequently felt overwhelmed. Paper books (pBooks) are fundamentally different than eBooks, generating a need to determine different strategies and methods for managing and using eBooks, which I hadn't done yet. Calibre is eBook library management software that allows new users to use it in simple ways while also accomodating more advanced users with many features and complexities. I was overwhelmed at first because I didn't know much about eBooks in general and I let myself get tangled in calibre complexities. So I sidetracked into tangents before settling in. Later I noticed that the more I learned about eBooks in general and the more I consciously simplified my use of calibre, the more I was successful in managing eBooks.

Project Summary. At 7 months into using calibre, I wanted some discussion on my then-current strategies, methods, and work habits, so I laid out what I was doing hoping for response. That's available in the forum thread "KISS for New calibre Users" in the original post and subsequent responses. I then recast the KISS version posts from "giving advice to new users" to "documenting what I'm doing, as one slightly experienced new user." I also re-started at baseline zero each for eBooks and calibre. I intend to use better strategies, methods, and practices during this next iteration cycle of my calibre use.

Configuration Status. My devices Kindle, iPad, and iPhone are up to date with operating systems and firmware upgrades, as is my primary computer. I have one Library containing one eBook in a clean new calibre installation. After installation I configured preferences. Added custom columns. Added one template in metadata plugboards. Added one tweak to format PublishedDate as year only. Left all conversion settings at default except to not convert comic book images to black and white. In Look and Feel, I set a tag to color a book's metadata red to indicate a wishlist item or bad format. In Behavior, I selected EPUB as preferred, and I checked yes for calibre's viewer tp open all listed formats except PDF. At that point I had only a few miscellaneous eBooks scattered around: calibre Quick-Start Guide (in Main), Kindle User Guide, various user guide PDF formats, and a few other technical and reference PDFs. That became my starting situation for gathering and managing eBooks.

Thank You. Thanks to everyone who's posted on Mobile Read, which is where I learned the strategies, methods, tools, and nearly everything else contained in this post. Particular thanks to those of you who posted in the KISS thread and helped me try to integrate all this stuff or format the posts better. You know who you are.



Progress
Spoiler:

Writing the KISS posts forces me to think out and reconsider everything I'm doing. I've also been gathering some eBooks and software tools. For tools, I try to find highly recommended ones even if they're not popular. And good tools have their own learning curves. For eBooks, I'm more discriminating about what I want in my library than I was previously, regarding quality of both content and format.

Tools Gathered:
  • File renamer. A Better Finder Rename, for Mac. Uses regex, has regex help.
  • EPUB editor. Sigil.
  • EPUB generator. Writer2ePub.
  • HTML editor. BBedit. (Mac only). Great text manipulation. Uses regex , has regex help.
  • 2nd Browser. Firefox. Some plugins may be helpful.
  • Image manipulator. GIMP.
  • MOBI viewing. Kindle Previewer. For accurate rendering on display of MOBI-based formats.
  • MOBI generation. Kindlegen. Amazon's command-line tool for generating Kindle formats from input documents.*
  • Adobe Acrobat X Pro. Bought it recently, still learning it.**

    *Note 1: I'd also try Mobipocket Creator if it were available for Mac users.

    **Note 2: I don't recommend Acrobat Pro to new users unless they can afford the cost and time learning it. I'm not sure yet how helpful it may be for cleaning up PDFs, since most newer PDF books arrive at least "locked" to prevent anything but copying, explicitly different from DRM. So it doesn't seem like a good tool for me for editing headers, footers, typos, or bad formatting out of many eBooks in PDF format.

eBooks Gathered:
  • Amazon. Prior purchases and a few free downloads I specifically wanted.
  • Baen Free Library. A few of them I specifically wanted.
  • Baen free CDs. Got all of them so I can compare between specific eBook formats, and maybe learn a bit of HTML from the HTML formats on the more recent CDs.
  • Bought Joshua Tallent's "Kindle Formatting: The Complete Guide to Formatting Books for the Amazon Kindle," for addional information about formatting eBooks. I thought it would also apply in general if not specifics to formatting content for EPUB.
  • Most of those eBooks, not yet added to calibre Library(ies).
  • Others planned in near-term: I want a few more eBooks on a highly selective basis from sites that have large catalogs such as Mobile Read and Project Gutenberg. And from my computers, gather User Guides and Manuals and a few odds and ends old PDFs lying around and add them too.

Last edited by unboggling; 09-24-2011 at 12:10 PM. Reason: Link to newer version.
unboggling is offline  
Old 08-27-2011, 01:01 PM   #155
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
Some major work habits of mine are not so simple and streamlined as I thought.

Finally tried the Goodreads metadata download plugin today now that I've added about 200 books from Baen and Amazon and have enough books to test results on. I generally like the Goodreads metadata and the simple, consistent way they do genre tags. And I realized my workflow sequence and methods for entering metadata as described in any KISS versions to date aren't very KISS after all. And same for my use of abbreviations and avoiding downloaded tags.

So now I'm considering a major revision of how I work that includes:
  • Additional custom columns for metadata items that Goodreads doesn't cover, such as Format Quality, Award, and BookType (they do Anthologies but not Collections or Omnibuses per the genre list). This will allow moving columns around to better accomodate actual workflow sequences, as well as making searches easier and more precise.
  • Not cram so much metadata into one narrow tags column, and expand abbreviations to be more human readable. Yes, I know that I can map some of their metadata to my abbreviations if I want to, but I'm not sure I want to anymore.
  • Possibly even use Goodreads' Title to overwrite my entered Title if they're consistent about the way they append metadata such as Omnibus and Volume info to the title, and maybe they also append other types of metadata too? (Haven't tested that enough yet.)
  • Use their ratings in addition to my own.
  • Only use Amazon in a second Metadata Download pass focused on those selected books that Goodreads didn't have.
  • Possibly use Goodreads Sync plugin after I learn more about the Goodreads site.
  • Use View Manager plugin to manage different views of columns for different needs, such as metadata entry for newly added books vs browsing my library for something to read.
  • And so on (there's more)….
I'd overcomplicated stuff (again) and deluded myself it was for KISS reasons. Hope I didn't lead anyone astray.

-unbo

Last edited by unboggling; 08-27-2011 at 02:29 PM. Reason: clarify
unboggling is offline  
Old 08-27-2011, 03:03 PM   #156
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@unboggling - I've not followed this thread in detail at all, but as you asked for feedback I will say that your "Metadata work habits" above has very little that is "simple" about it imho. As a personal choice you can do what you like of course, but if you are trying to come up with a "here's a simple way to go about organising your library and working with Calibre" I think it is way too OTT. i.e.

_q4, ((fn, (mgc, (ya, [om, %sma, %su, r3

I wouldn't recommend inflicting that on anyone - seriously!

I guess it depends on what your goal is. If you are wanting to show people what you are doing to get their feedback, of course you should write (and do) what you like. If it is more focused on the "simple" side of things to help others get started with some "best practices", I think you should reduce the scope in some areas and push them into a more "Advanced usage" type section.

You are going through the Calibre discovery process as we all have, and only through experimentation will you find exactly the balance that suits. On past threads that have looked at what people use custom columns for, it is quite astounding the amount of detail some users store in their library.

I'm glad you are using some of my plugins, most of them came about because of ways that I wanted to work. I have spent more time "using" than "developing for" Calibre the last few months, and it has given me a chance to refine my own workflow and plugin usage. And with thousands of books in Calibre I have had to fine tune things for my needs to get the balance between what is "really important" versus just data for the sake of it.

For instance things like quality - I just use a yes/no column called #retail that tells me whether I have a retail quality version or something that isn't. I only keep books that are readable - if it is a poor quality OCR scan or whatever then I trash it. I mean, why keep crap? I have too many books to read as it is - and if I really want a book that I only have crap versions of I will just buy the retail version when it is available one day.

Just in case anyone was interested, this is my rough workflow/setup (I'm not suggesting in any way it is better, just publishing it for your info). I make no apologies for the fact that it sounds like an extended plugin advert - I wrote them mostly for my own usage and here is most of them I use daily.
Spoiler:
  • ePub rocks my world and is my "master", from which I convert to MOBI for my Kindle. If I had no Kindle, I would just keep ePub. I usually only buy ePub, or if it is cheaper I buy MOBI, convert to ePub and sometimes back again.
  • The only other format I ever keep (in the absence of ePub) is PDF for some childrens books full of images (or technical books). PDF conversions suck, and I only read those types of books on a tablet anyway. Well not that I read childrens books myself, but I do want to spoil my nephews one day with a tablet.
  • Once converted I chuck every other format outside of my ePub/MOBI. I've already made my choice to get the best quality version that is readable, why keep what I know for a fact I will never use?
  • When adding a book, I use hotkeys associated with the Quick Preferences plugin to switch between different regexes based on the filename. I only add books one or a few at a time using drag/drop. If the filename will not be a good match to my couple of regexes I just create an empty book for the author and type the title manually then drag/drop it.
  • After adding a book, I use the Extract ISBN plugin to give the metadata download a better chance of a match. If I remember, depends on how simple I think the title is. Only if it is a non fiction technical book do I allow Calibre to overwrite title/author.
  • I only use my Fantastic Fiction, Goodreads and B&N metadata plugins. Eating my own dogfood, and the best sources for cover images and series info imho.
  • Depending on how good the B&N image sizes returned are, I use Google images via the Search the Internet plugin to drag/drop a better quality cover.
  • I only use a simple set of genres (into the Tags column) as from the Goodreads metadata plugin. Dwanthny has a more complete set mapped but seeing a book as "Fantasy" or "Science Fiction" is enough for my needs for most books.
  • I don't make hard and fast rules about number of tags etc, you make it too hard for yourself. Sometimes a book screams at you for being cross-genre, like "Fantasy, Romance". And if that book is targeted at younger readers, it might be "Fantasy, Romance, Young Adult".
  • I do have a couple of other tag values I put in Tags - like "-" to indicate a wishlist item (empty book), just to use less space on screen than a column. But there only a couple, I've trashed others along the way.
  • I convert/cleanup books on an author by author basis, not "as I need them". But it is an ongoing time consuming mission. I can understand Jacki's points, but for me between the MOBIs I have to read on my Kindle now, and the ePub which will do me for years to come I have something that is perfectly readable.
  • In terms of cleanup I do have certain indents, margins, stripping fonts inserting page breaks things I do on the master. The #1 priority is that it is the best possible of the versions I have if I have multiple in terms of making sure it is not someone else's crappy conversion with lost quotes etc.
  • Any editing of books is only ever on the ePub, either with Tweak ePub, Sigil (via the Open With plugin), or the Modify ePub plugin to replace covers/strip jackets.
  • After the cleanup and mobi conversion, I use the Count Pages plugin to populate my #pages and #words custom columns. I like to see the relative size of books.
  • When I have read a book, I tick my #read yes/no column. Then I use the Goodreads Sync plugin to add it to my Read list. I then go to the website to specify the date read and my rating for it. One day I will enhance the Goodreads Sync pugin to allow doing all of that in one step from Calibre. I don't pull data from Goodreads into Calibre.
  • If it is a short story, I add a ", Short" to whatever the genre tag is. And I use the "Generate Cover" plugin to make a cover for it. I have a whole bunch of prefdefined images/fonts for different genres.
  • If I am keeping track of authors for friends/family, I use the "User Category" plugin to add the author to a category for each person.
  • In particular for sharing books with my parents I keep a #sent custom column to record which books I have sent them.
  • In terms of general day to day usage, I actually only use the View Manager plugin with a single view configured. However it has my sort order configured of by author/series/published/title with a hotkey. So I randomly click on column headings to sort data as I like it for cleanup purposes, then can hit my hotkey to get my preferred sort back in one keypress.
  • If you have a mishmash of series data and a lot of titles in the series, or I want to rename a series to shorten it, I use the Manage Series plugin
  • I also use the Search the Internet plugin mainly with FantasticFiction to create empty books for other books I want from the author or are upcoming etc.
  • Periodically I make use of a plugin like Quality Check to check the state of the library. However as I am pretty disciplined with this particular library in terms of not just "throwing thousands of books at once" in there, I'm happy to only do this every now and then.
  • I use meme's Kindle Collections plugin (brilliant) to organise my Kindle. However I use only a fraction of it's functionality. I organise my books by a single author_sort criteria, and don't manipulate my Kindle collections using it.
  • I don't stick my whole library on the Kindle, but I do put several hundred books on there across a range of genres so I can have a wide choice wherever I happen to be when I finish reading a book on it. Periodically I top it up and remove any books I have read off it.
kiwidude is offline  
Old 08-27-2011, 03:40 PM   #157
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
@kiwidude - that was exactly what I was hoping for, and you delivered above my hopes about 2000%. You've given me ideas and suggestions I already want to integrate into my own use of calibre, and that's without yet having it all fully sink in and considering further implications. Thank You !!!!

Re goals for the thread, yeah, I had conflicting motives and a hard time with being confused between my own relatively new user needs, and a completely starting out new user's needs. Theducks also proposed the idea of separating stuff out for new users and intermediate level users. Good idea, but difficult for me while I'm still so new, confused, and experimenting.

I consider your post a gentle nudge in good directions and appreciate the time you spent doing it.

Last edited by unboggling; 08-27-2011 at 03:45 PM. Reason: spelling
unboggling is offline  
Old 08-27-2011, 04:00 PM   #158
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Glad you found it useful - not many people are brave enough to ask for critique on how they are organising their library and even less to spend time turning that into a tutorial/guide so along with others I applaud your efforts.

There is no "one true way", and as probably 95% of Calibre users have never downloaded a plugin in their life my workflow obviously has a bunch of steps in it others don't bother with (or do manually with more effort). For every point on my list you are guaranteed to find someone who will argue the opposite. But if something in there sparks an idea for you or someone else then it's all good...
kiwidude is offline  
Old 08-27-2011, 04:16 PM   #159
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
More like a firestorm of ideas than a mere spark…. So I'll sleep on it all. Then the wise thing would be for me to let it all cook at slow simmer for a few days while I add more books from Baen CDs do non-eBook/non-calibre things. Heh, I don't think I'm that wise, I'll probably start implementing changes without enough forethought tomorrow.

Last edited by unboggling; 08-28-2011 at 04:20 AM. Reason: (a little wiser, less to change later)
unboggling is offline  
Old 08-29-2011, 01:52 AM   #160
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
Aaaaarrrrrrrggggg!

I'm trashing the proposed strategy to "obtain on demand and clean up shortly before reading". I didn't consider how much time it takes to find books, get their metadata accurate up to my library standard, clean-up formats if necessary. The strategy sounded good on paper screen, doesn't work in real life.
unboggling is offline  
Old 08-29-2011, 03:15 AM   #161
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
Also at a similar level of overall strategy, kiwidude's point about using ePub as the "master" and generating formats for various devices from that makes sense to me too and I plan to do that for nearly all formats except certain technical, graphic-laden, or otherwise complex PDFs which I'll keep in the record and not bother to convert. And not keeping the original incoming format in calibre in most cases makes sense to me too, if I keep those outside calibre in a "Raw Books" folder. If in the future I need it back in calibre, I can import it as a new record and work with it from there, plus easily merge it with the ePub/Mobi record if necessary. Personally, keeping those uncleaned formats in the same record as cleaned formats confuses me as to which is which. (That is why I originally started out using ePub as "master" and deleting all the incomings from calibre.)

I generally like how kiwidude does his library organization and workflow too and am presently stealing borrowing adopting as much of it as fits my needs - which is a lot of it - and will also incorporate a lot of it into the next KISS posts.

For the KISS posts I'm planning to separate into two separate posts, one for new users of calibre at the beginning level, with most of the conversion, format, and other technical discussion moved to one that's just documenting what I'm doing. Those will be titled something like "Keep It Simple for New calibre Users" and "My KISS Project". Those revisions probably won't be written and posted until sometime next month.

Last edited by unboggling; 08-29-2011 at 02:57 PM. Reason: Added PDF exception; removed comma from "Keep It Simple, for New calibre Users"
unboggling is offline  
Old 08-29-2011, 06:10 AM   #162
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@unboggling - re the metadata as you go thing. I will share one of my "failure" experiences here, which only applies to people who have a very large collection of books to begin with.

Like I am sure a lot of Calibre users my initial approach was to just throw all my years of books into my first Calibre library and then "clean as I go". However apart from performance issues when you have an enormous library (which subsequent Calibre releases have improved but not eliminated) it is a case of garbage in, garbage out.

If your initial load of Calibre is from a clean and structured source (say a Kindle using only books bought from Amazon) then there is no problem. However if like me you have numerous duplicate formats and editions from varying sources, I found it unmanageable to take the approach of "I will just clean it up as I go later". It became just too hard and "dangerous" in the adding to the library in terms of what to do about duplicates. If you have multiple of a format for the same book they could all be varying quality, done from different conversions of different sources etc. That is all assuming the file was correctly named as the right book in the first place! So either you have to make a decision at the time you import to Calibre as to which is the best format to have, or you import them all and end up with a mammoth library(slow) with a huge duplicate problem to resolve.

So what I ended up doing was almost starting again by having multiple libraries - the initial one that is the "partially clean mess", and my "clean one" that books get migrated into that is intended to contain only books that are cleaned up. Note that if I was installing Calibre for the first time today I would not have two libraries for this purpose - it is only the fact that I had already invested so many months of effort into partially cleaning the initial one that I have kept it.

So now what I do is just load books into my "real" Calibre library on an author by author basis. I do an indexed search in Windows Explorer of all my mish-mash source book folders which are under a common root folder, to find all books I have for an author, sorted by type. If I have already setup books for that author in my old library I use the Copy to Library (with Delete) to move into my "clean" one. Then for each book I update the metadata/cover and identify which is the best format of it I have, and if it isn't an ePub then (some PDF exceptions noted) I convert it to ePub and do any final cleanup. Then I delete every other format, in my source directories and from Calibre (my Calibre is of course backed up).

Now it didn't take very long to get enough books in my new library to ensure I would not be running out of things to read any time soon. Is every book I own in there? No. However I have just started with my most favourite/desired authors to read, and gradually chipping away at them every week. At some point I will get bored with doing this (or run out of time) as given the years of "to read" books I have already the likelihood of reading the rest is slim to none. However as an author name comes up mentioned by family or friend I can just repeat the exercise for that author to extract them from the source mess and add them to the clean library.

My so called "clean" library is however not perfect, it does have a backlog built up in it. For a prodigious author where the quality is so-so it takes a long time to do all the editing, and it is easy to get distracted into adding another "small" author you know you have the retail versions for. So I have a #done yes/no column I use with each book, which in combination with #retail gives me an easy way to see which books I still need some effort on. However at least I know that format is the best I have at that point for that book/author.

That was the balance that ended up working for me anyways. Like I said, if I was starting from scratch today I would just use the "author by author" gradual approach to add from windows explorer to Calibre. The risk for people who have multiple conflicting formats of a book is to try to add them all to Calibre - my advice is find your best format and only add that one in. You can associate the Calibre ebook viewer with the various file types like epub, mobi etc so it is just a double click to open them from the explorer search results.

Last edited by kiwidude; 08-29-2011 at 06:13 AM.
kiwidude is offline  
Old 08-29-2011, 08:00 AM   #163
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
Thank you for those ideas, too, kiwidude. I didn't originally have the problem of importing a lot of eBooks lying around on my computers, because I was relatively new to wanting eBooks at all. When I got my first reading device (Kindle) - that changed and I became an eBook addict who'd much rather read something on a device than a paperbook, brought on in part because I was tired of relocating with lots of bookcases and book-weight.

Eventually I dumped nearly everything I found out there on the internet into a library for cleaning-up and then moved all clean formats to my main library. And I began bogging down in the metadata-updating and format-cleaning processes. As a fresh beginning calibre user I couldn't handle putting it all into the same library as "clean" books after the first week or two.

So I like your ideas.

Handling metadata and/or format clean-up in small groups by author makes a lot of sense and I'll "adopt" that into the KISS posts and my own workflow too. The #done column idea (or at least specific tags for "evaluated" and "cleaned", if not separate columns) makes sense to me too.

I'm still thinking about whether I'm better off using a "NeedsCleaning" library or just doing it all in one library. The past week I've been doing it all in one library. Pros and cons either way, but I still don't like mixing up unevaluated books with evaluated books, and evaluated still-need-cleaning books with cleaned books. Having evaluated and cleaned-if-necessary books in a separate library gives me warm feelings of security and accomplishment.

And I still plan to be much more discriminating about content and format quality that I download in the first place.


Last edited by unboggling; 08-29-2011 at 08:27 AM. Reason: clarify
unboggling is offline  
Old 08-29-2011, 08:58 AM   #164
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by unboggling View Post
I'm still thinking about whether I'm better off using a "NeedsCleaning" library or just doing it all in one library. The past week I've been doing it all in one library. Pros and cons either way, but I still don't like mixing up unevaluated books with evaluated books, and evaluated still-need-cleaning books with cleaned books.
I keep them all in one and tag them according to whether they've been cleaned up. I use the content server a lot, and if I don't keep them in one library, I can't access them, or find out if I've got a book or all books in a series, etc.

Most of my books were in decent shape for reading as TXT format, and I had too many to wait until they were all converted.
Starson17 is offline  
Old 08-29-2011, 08:59 AM   #165
travger
Evangelist
travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.
 
travger's Avatar
 
Posts: 480
Karma: 270594
Join Date: Aug 2010
Device: palm tx, Windows7, Galaxy A5
Quote:
Originally Posted by unboggling View Post
I didn't consider how much time it takes to find books, get their metadata accurate up to my library standard, clean-up formats if necessary.
One reason why I started cleanup library. When I'm already on the page it's easier to find other freebies than go hunting for them later (and who says that they'll be free then). Also, it's easier to see if I already have the book and keep author's books together.
I toss them in and enter easily avilable data from download page - publication date, source, synopsis+some reviews...
When I'm looking for reading material, it's easy to browse the covers, read the blurbs and decide. I may make some cleaning of metadata in between also (like, when in isfdb, I can look up data (awards, series, first publication) for several books I have from the author), but can't know anything about formatting and such until I've read it.

Kiwidude advised to find your best format and only add that one in. I like to see at a glance what I have for the book.
So my Calibre shows 'formats' in book detail and just one look tells me what I did. If there's only mobi, it means I dl-d it and have no gripes (not happened yet). Mobi/prc + txt = some gripes (maybe not worth the cleaning, maybe some other comments). Pdf/epub + prc + ace + txt = converted and cleaned up, details in txt, materials for mobi creator in ace.
One click on format opens it - mobi in mobireader, pdf in acrobat, epub in Calibre reader...
When I've read it, then I delete all other instances of the book from my HD (exept Calibre backup of course).

Maybe in the future if I see lot of lag in Calibre (never knew that having several formats could cause it) I'll move everything exept mobi and txt to archive, but not now.
travger is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Two Users--One Calibre Library? jerrypettit Devices 3 06-24-2011 01:01 PM
Hi, Calibre users zaphod234 Introduce Yourself 6 07-22-2010 08:16 AM
The Success of the iPad and KISS ColdSun General Discussions 68 07-10-2010 10:33 PM
How do Calibre users.... Lanyon Calibre 24 01-01-2009 01:29 PM


All times are GMT -4. The time now is 12:27 PM.


MobileRead.com is a privately owned, operated and funded community.