Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 07-19-2012, 04:11 PM   #1
unboggling
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
How I Manage eBooks with calibre

Revised 2014-06-23. Long post. Strategy, workflow, and layout are mapped in detail and example.


Approach
Spoiler:

Overview
  • Purpose. Describe a practical approach for managing ebooks.
  • Topics. Needs and Strategies, Workflow and Layout.

Needs and Strategies
  • Identify Needs. Identify current personal needs for ebooks. Reevaluate needs periodically and revise the list of needs to reflect any changes. Some of my needs are:

    • Feed the reading habit primarily with fiction, usually science fiction or fantasy novels, and secondarily with text-based nonfiction and news. Read books that are text-based and simply formatted. Also read books with complex formats, image-laden content, or technical content.
    • Avoid reading books with annoying formatting problems that adversely affect readability.
    • Maximize time for reading. Minimize time fixing formatting problems and maintaining book libraries.
    • Read comfortably during short or long reading sessions, usually at home or occasionally while out and about. Avoid eye strain caused by glare, brightness, or too-small screen. Avoid sore wrists from heavy device.
    • Find books quickly and easily regardless of actual location.

  • Devise Strategies. Devise strategies to meet needs, within the constraints of available ebook-related hardware and software technology. Review strategies periodically and revise them to reflect changes in needs or technology. Some of my strategies are:

    • Keep it simple. (See KISS principle, Wikipedia.)
    • Use calibre as the hub of my ebook activities. Supplement calibre with satellites such as reading devices, reader apps, and other useful tools.
    • Process new books a few at a time, one author at a time, and finish processing them before adding other new books.
    • Keep metadata consistent within each library and across multiple libraries by comparing with metadata on a good bibliographic site, at first entry of metadata and in periodic author-level updates.
    • Be discriminating about content. Be selective about which authors to read, their overall writing skill, and which publication types and subjects/genres fall within areas of interest. Do not collect books just because they are free or cheap.
    • Be discriminating about source and quality of formatting. Some sources tend to have formats with higher quality formatting and fewer formatting problems than other sources. Track sources and use the better ones. Track format quality, fix formatting problems that are quickly fixable, and only keep formats that are readable with mild or no annoyance.
    • Standardize on EPUB as the format for most books. Make exceptions for PDF format, due to probable conversion problems and wasted time fixing them. Keep PDFs that are nonfiction with complex formatting to preserve. Avoid fiction PDFs.
    • Read image-laden, color-formatted, or complexly-formatted books on a tablet. Read simply-formatted text-based books on e-ink device.
    • Learn and adjust. Observe, orient, decide, and act in rapid cycles. (See OODA loop, Wikipedia.)

Workflow and Layout
  • Refine Workflow. Derive workflow from needs and strategies, in conjunction with layout, within the constraints of available technology and skill using it. Gradually refine each workflow, adapting to changes in needs, strategies, technology, skill, and layout. Keep it simple. Workflows differ between various people due to different needs, strategies, skills, chosen technology, and personal inclinations. (See Links > Workflow). I do three separate workflows, described in the sections Processing, Reading, and Maintaining.
  • Refine Layout. Layout includes library structure and associated metadata scheme. A metadata scheme defines metadata elements for each column and conventions for their use. Each library may need a different layout depending on the purpose of the library. Gradually refine each layout, adapting to changes in needs, strategies, technology, skill, and workflow. Keep it simple. Layouts differ between various people due to different needs, strategies, skills, chosen technology, and personal inclinations. The Library section describes my library structure. The Metadata section describes my metadata scheme.


Processing
Spoiler:

Overview
  • Purpose. Transform text-based books one author at a time into readable condition, with accurate metadata. My focus is on fiction, specifically novels. Generally the processing workflow also accommodates other types of content — such as nonfiction, shorts, comic books, magazines, or audio books — with little or no adjustment.
  • Topics. Browse Books, Add Books, Edit Metadata, Convert/Edit Formats, Update Metadata.

Browse Books
  • Choose Current Author. Choose an author. This author becomes the Current Author for the Processing workflow. Some ways I choose Current Author are:

    • Browse New Releases. Browse new releases and forthcoming books on a retail or bibliographic site. Fantastic Fiction has lists of books released or forthcoming for the current month, the prior 3 months, and the next 3 months. Internet Speculative Fiction Database (ISFDB) has lists of books released or forthcoming for the current month and the next several months. Amazon has lists of new books by genre. Browsing new releases provides an opportunity to investigate and potentially choose authors not yet in the library, or to update an author already in the library with newly available books. (See Links > Bibliographic Sites).
    • Browse Authors in Library. Check if any books are newly available for an author who has books in the calibre library. In a long-term library-update cycle, gradually work through all the authors in the library, updating each successive author in the library as the new Current Author.
    • Other. Recommendations from family, friends, social networking sites, book reviews.

  • Search for Owned Books. Search for books by Current Author that are already owned. To limit the number of places to search, first consolidate books into as few places as practical. I keep all processed books in one main calibre library, so I search it for Current Author. The first two years of using calibre I also did an operating system search of a Pending folder of books not yet added into a library, until eventually all Pending books were added to calibre, one Current Author at a time.
  • Find List of All Books by Current Author. On a bibliographic site with consistent metadata for many authors, find a list of all books written by Current Author. In a browser window I usually keep a tab open to Internet Speculative Fiction Database (ISFDB) and a tab open to Fantastic Fiction, then switch back and forth, pasting author name into the relevant search box as necessary. Occasionally I open tabs for other helpful sites, or use the plugin Search the Internet, which has a menu of link templates for various sites.
  • Add Empty Books. To avoid duplication and repetition of investigative work in the future, add empty book placeholders for wishlist books to get in future, and add empty book exclusion placeholders for books not to get at all. An empty book is a book record with no associated book format. For each new empty book, enter metadata for Authors, Series, Title. Enter format status in custom column FormatQ: f0 for exclusion, or f1 for wishlist. For exclusion placeholders also edit Title to specify exclusion at author, series, or title level. (See Metadata > Custom Columns > FormatQ.)
  • Evaluate Current Author Status. If there are no books by Current Author to acquire now, edit custom column Updated with the current year and month or the inactive code for any books in the library where Current Author is the first or only author in Authors, then go back to the first step in Processing > Browse Books to choose a different Current Author. (See Metadata > Custom Columns > Updated.)
  • Note, Digital Rights Management (DRM). I assume in this workflow that DRM is not an issue. Several years ago when DRM was an issue for me, I searched the internet for Apprentice Alf.
  • Compare Sources. Optional. If necessary compare sources to determine which to select to acquire books to get now. In the Get Books feature in calibre, choose the desired sources, then search for Current Author, sort by Store (source) to see all books there by that author, or sort by Title to compare prices between Stores (sources).

Add Books by Current Author, For Each Source
  • Choose Source. Choose a source. Ideally, choose a source that offers the desired format. I like EPUB better than AZW3. PDF is acceptable for complexly formatted books such as technical books, art books, or textbooks, only if good versions are not available in AZW3 or EPUB. Amazon does not offer EPUB format, but Amazon prices are often a little cheaper, and Amazon has a conveniently large selection in AZW3 format, which can be converted successfully to EPUB later.
  • Download Books. Download desired books from this source. When I have the choice, I prefer downloading to computer rather than directly to device, to avoid extra workflow steps later.
  • Pre-Assess Multiple Book Formats per Title. For any downloaded books where the download included multiple formats per title, to simplify processing after the Add choose only one format to add, either the preferred format or the best pre-assessed format. If necessary to pre-assess, open each format file in the suitable reader application, as set in the operating system for relevant file type.
  • Standardize Filenames in Operating System. Standardize the filenames in the operating system prior to Adding Books, into a structure compatible with the chosen regular expression for Adding Books. I use the structure "authors - series series_index - title"; series is optional; if there are coauthors I separate them with space ampersand space " & ". Manually editing each filename is not too time consuming because I add just a few books at a time. Otherwise I would use a file renamer tool to bulk rename a lot of files needing a similar filename structure changed to a different filename structure. (See Links > Filename Renamers; and Links > Filename Limitations).
  • Add Books. For each book in Download (or Pending) folder that has a preexisting wishlist placeholder, add the format by dragging it into the Formats box of Edit Metadata Individually or the Book Details pane of the wishlist placeholder, then delete the wishlist tag "f1" in custom column FormatQ. For other books in the Download (or Pending) folder, drag and drop them onto the Library View booklist to copy them into the library. For books directly downloaded to device, connect device, add them to calibre library from device, then remove them from device; for any with a corresponding wishlist placeholder, merge the record with the book format into the placeholder record and delete the wishlist tag "f1" in FormatQ; then Save to Disk into the Download folder all the books just added that had been downloaded directly to device. My preference settings for adding books are:

    • Preferences > Adding Books. All options are unchecked on both tabs, except Copy to Library to preserve date added. Automerge is unchecked because I prefer each format of each book added as a separate book record. Read metadata from file contents is unchecked because the results are unpredictable, often requiring more work later to correct bad metadata or delete unwanted metadata. The only metadata I want set during the Add is Authors, Series, and Title. So I prefer reading metadata from filename using a regular expression.
    • Note, Defaults. To see what the default settings are for any Preferences pane except Plugins, click the button "Restore Defaults". Click the Cancel button to return to prior settings, or the Apply button to keep the default settings.
    • Quick Preferences. The plugin Quick Preferences has a menu of regular expressions tailored for various filename structures. I use the Quick Preferences regular expression labeled "Author [- Series #]- Title", which is one of the defaults included in the plugin. This regular expression handles many variations correctly as long as the metadata in the filename is in the order author, series, title, with series being optional.

  • Move Raw Files. For the books just Added, move the corresponding original files from Download (or Pending) folder to the relevant source folder in Raw Books folder. Keeping these original downloaded raw files is a form of insurance, as a one to one raw backup (not converted or format-edited) of each book in the calibre library. The standardized and comprehensible filenames, edited previously to fit the regular expression for adding books, also will be useful in identifying any of these raw formats if necessary in future.
  • Enter Source. In calibre, enter source of format in Source. (See Metadata > Custom Columns > Source.)
  • Move Metadata for Authors, Series, Title. If necessary move metadata into the correct columns for Authors, Series, and Title.

Edit Metadata, For Each Newly Added Book by Current Author
  • Edit Authors, Author Sort, Series, Series Index, Title, Title Sort. For newly added books including placeholders, edit metadata as necessary for Authors, Series, Series Index, Title, Author Sort, Title Sort. Apply relevant conventions to the metadata. (See the Metadata section for details.) Authors and Title need to be clean prior to Metadata Download, because Metadata Download relies on Authors and Title for matching when there is not an ISBN in Identifiers.
  • Extract ISBN. Optional. If necessary to get specific metadata for a specific publication of a book, extract an ISBN using the plugin Extract ISBN, or in Edit Metadata Individually paste into Identifiers the ISBN of the desired publication copied from the bibliographic site or relevant publisher site.
  • Download Metadata. For each newly added book and wishlist placeholder, in Edit Metadata Individually, using sources Amazon, ISBNdb, and Fantastic Fiction (optional plugin), download metadata for the best choice of publication, fetching only Comments, Languages, Published, Publisher, Cover (automatic), and Identifiers (automatic). If dissatisfied with results of the metadata download, delete the Identifiers then download metadata again with different choice of publication or cover. If necessary, copy better cover or Comments from a bibliographic site. If necessary, trim cover manually using the Trim Image tool in the Edit Metadata dialog. If necessary, edit Languages, Identifiers, or Comments. (See the Metadata section for details.) Note: I do not download any other metadata because I already set Authors, Series, and Title; I prefer relying on my own rating system rather than an outside Rating; and I would rather manually enter other desired metadata per my own metadata scheme faster than editing or deleting a downloaded mess of unpredictable, disorganized, or inaccurate Tags.
  • Edit Metadata. In the booklist, edit or enter metadata, referring to the bibliographic site. If necessary edit Published and Publisher. Enter metadata in custom columns Genres and Keywords, gleaned from information in Comments. If relevant, enter metadata in custom columns Types, Awards, Read. If Title is not unique for different publications of one authors/title, append edition or other publication-level metadata to Title in parentheses. If necessary, enter alternate author names (also known as) in custom column aka. (See the Metadata section for details.)

Convert/Edit Formats, For Each Newly Added Book by Current Author
  • Assess Format. Open the format in the appropriate viewer, depending on the type of format. Scroll or page through the format, looking for formatting problems that interfere with readability or are annoying. Initial setup: in Preferences > Behavior > "Use Internal Viewer," check any format to be opened automatically by calibre Viewer, and uncheck any format to be opened automatically by an external application per the operating system setting for that file extension. I use the calibre defaults.
  • Convert Format. Convert format to EPUB, unless it is an EPUB with no formatting problems or a PDF. (I avoid adding PDFs to the library when possible, but would rather read a PDF as PDF than convert it to EPUB followed by wasting time fixing all the conversion problems.) If formatting problems were identified in a format other than PDF, in the Conversion dialog box tailor the conversion with settings that will fix as many of the formatting problems as possible, convert, assess the results, if necessary convert again with one or more different settings then reassess, repeating as necessary. (See Links > Conversion.) The settings initially depicted in the conversion dialog box before the first conversion are the corresponding settings in Preferences; mine are:

    • Preferences > Behavior. Preferred output format: EPUB. Job priority: Normal. Both are calibre default settings. I chose EPUB as preferred output format for two reasons: Calibre's ebook viewer opens fastest with EPUB because it is an EPUB viewer; so to view a non-EPUB format calibre has to partially convert it to EPUB behind the scenes, which takes time. And EPUB is an open standard, while AZW3 and other Amazon formats are proprietary.
    • Preferences > Input Options. All settings are calibre default.
    • Preferences > Common Options. Look and Feel, text justification: justify text. Page Setup, output profile: Kindle (initially set in Welcome Wizard dialog at first use of calibre). All other settings are calibre default.
    • Preferences > Output Options. All settings are calibre default.
    • Preferences > Tweaks. Save original file when converting/polishing from same format to same format: converting = True and polishing = False. The first is calibre default. Saving the original format when converting supports repetitive conversions of original format to same format with different settings until getting it right, and also serves as a backup in case of fatal mistakes while editing the format. When necessary, restore the format from original by right clicking the original format in Book Details or Edit Metadata Individually, then choosing "Restore 'format_name' from the original". I set the tweak for polishing to false to save time when removing unnecessary formats later, because Polish Books does not do a conversion and its limited options are not likely to be destructive.
    • Note, Defaults. To see what the default settings are for any Preferences pane except Plugins, click the button "Restore Defaults". Click the Cancel button to return to prior settings, or the Apply button to keep the default settings.

  • Balance Between Extremes. As an ebook consumer primarily reading books for enjoyment, my objective is to spend the least amount of time fixing formatting problems to make a book readable with no annoyance. I ignore formatting problems that do not affect readability, ignore formatting problems that are only mildly annoying but are not quickly fixable, and fix formatting problems that are annoying and quickly fixable. If that does not achieve readability with mild or no annoyance, I remove the format to make the book a wishlist placeholder. That is how I balance between ignoring all formatting problems, or striving for perfect formatting for every book. See Thoughts for those new to ebook reading, post by jackie_w.
  • Edit for Major Formatting Problems. If necessary use one of the following approaches to fix any major formatting problems that adversely affect readability and are quickly fixable. Then assess the results, and if necessary edit the format for major formatting problems again, better, then reassess. (See Links > Format Editing, and Links > Conversion.)

    • Fix Underlying Code in Editor. In this approach, directly edit the underlying code in the format. This requires basic knowledge and skills with regular expressions, HyperText Markup Language (HTML), eXtensible HyperText Markup Language (XHTML), and Cascading Style Sheets (CSS). Regular expressions are used in search/replace operations. CSS simplifies applying styles to text. HTML marks up text for construction of web pages. XHTML, largely based on HTML and extending it, also marks up text. AZW3 wraps EPUB. EPUB uses XHTML. MOBI uses HTML-like tags. Tools for fixing the underlying code are:

      • Edit Book (EPUB, AZW3). Select Book, open calibre's Edit Book feature, edit HTML/XHTML and CSS files with regular expressions in search/replace operations, clean up the format structure as desired, save. The Command Line Interface counterpart is ebook-edit.
      • Sigil (EPUB, HTML, TXT). Open in Sigil, edit HTML/XHTML and CSS files with regular expressions in search/replace operations, clean up the format structure as desired, save EPUB. Sigil is an open-source EPUB editor, no longer actively developed.
      • Text Editor (EPUB, AZW3). Unpack the format, in text editor edit the HTML/XHTML, and CSS files with regular expressions in search/replace operations, manually clean up the format structure as desired, repack the format. Some useful adjunct tools include calibre Unpack Book (EPUB or AZW3), and the optional plugin KindleUnpack (AZW3, MOBI, other Amazon formats). See list of some code-oriented editors in Links > HTML/XHTML.

    • Fix in Word Processor. In this approach, edit in a word processor, with calibre automatically dealing with the underlying code during conversion. This requires no knowledge/skills with HTML/XHTML and CSS, and little or no knowledge/skill with regular expressions. I use this approach with Word often, and have used it with OpenOffice Writer and LibreOffice Writer. The workflow steps are:

      • Convert to RTF. Convert EPUB to RTF.
      • Save RTF to Disk. Save the RTF to disk into a Fix Formats folder. My permanent preference settings for Saving Books to Disk include: save cover separately, update metadata in saved copies, save metadata in OPF file, convert non-English characters, show files in file browser after saving to disk, save template (not including the quotation marks): "{author_sort}/{title} ({id})/{authors}{series:| - | }{series_index:0>2s}{title:| - |}{#formatq:| (|)}"
      • Edit RTF. Open the saved RTF copy in a word processor, either Word or Writer. If markup characters are not visible, in the View menu turn on markup (Word) or nonprinting characters (Writer). Fix various annoying formatting problems using find/replace. Example problems:

        • Linefeed versus Paragraph. One frequent problem is that a calibre conversion could not add or remove the extra line between paragraphs or change the paragraph indent. This usually shows up later in the RTF in word processor as linefeed markers instead of paragraph markers, so replace the linefeed markers and any spaces before or after them with paragraph marker, and if there is no tab marker for indenting paragraphs, add one. The basic operation in Word (not including the quotation marks) is: find "^l" (that is lowercase L) and replace with "^p", or find "^l" and replace with "^p^t", with variations to eliminate any space or non-breaking space before the "^l" or after the "^l".
        • Page Breaks. Remove hard page breaks that interfere with readability, such as a page break after each ToC item, or a page break between "Chapter n" and an associated chapter title. In those cases, with Word eliminate the offending page breaks by replacing "^m" with nothing, keeping in mind that calibre may "insert page breaks before (at XPath expression)" automatically during later conversion to EPUB (as set in Preferences > Common Options > Structure Detection).

      • Anchor Images. In Writer, if there are images, anchor each image to a paragraph. This is not necessary in Word.
      • Save As. From Word save as DOCX, or from Writer save as ODT, into the Fix Formats folder. Saving as DOCX or ODT avoids the large file-size of RTF; the DOCX or ODT is significantly smaller. Do not save directly into the calibre library folder because calibre will not detect it, which is one reason why the RTF was saved to disk from calibre into the Fix Formats folder in the first place.
      • Add Format. Add the DOCX or ODT format to selected book record in calibre, by dragging from Fix Formats folder and dropping onto Book Details or dropping into the Formats Box in Edit Metadata Individually.
      • Convert to Preferred Format. Convert DOCX or ODT to EPUB.

  • Edit for Minor Formatting Problems. If necessary fix minor formatting problems that adversely affect readability and are quickly fixable. Then assess results, and if necessary fix minor formatting problems again, better, then reassess. Some useful tools are:

    • Edit ToC. Fix problems with the Table of Contents (ToC) using the calibre feature Edit ToC. I usually skip this. I never use ToC when reading fiction, and the little nonfiction I read usually already has a good ToC.
    • Quality Check and Modify ePub. Identify specific problems in the format with the plugin Quality Check, and fix many of the them with the plugin Modify ePub (EPUB only), without doing a conversion. Quality Check also can do a few small fixes to metadata. I use one or the other plugin if necessary to identify or fix a specific problem.

  • Enter Format Quality Rating. Enter the format quality rating and any other relevant format-level tags in FormatQ. (See Metadata > Custom Columns > FormatQ.)
  • Remove Unnecessary Formats. Remove all formats in wishlist or exclusion placeholder records. Remove all unnecessary formats for other books, such as ORIGINAL_EPUB, DOCX, AZW3, or MOBI, retaining just one format in the book record, the master format. The master format is either an EPUB or a PDF.
  • Remove Duplicates. If the book is one of a set of duplicate book records, compare metadata between the duplicates. Correct metadata if necessary for the duplicate with format that has best format quality rating, keep that record, and remove the other(s).
  • Count Pages. Use the plugin Count Pages to enter page count in custom column Pages. (See Metadata > Custom Columns > Pages.)
  • Polish Book. Optional. Polish the format in the library if the cover or important metadata fields (Authors, Series, Title) were changed since the last conversion. In the calibre feature Polish Books, I usually check the options to update metadata and update cover in the format, and uncheck the other options (embed referenced fonts, subset embedded fonts, smarten punctuation, add "book jacket", remove "book jacket", remove unused CSS rules). Polish Books (EPUB or AZW3 only) or the plugin Modify ePub (EPUB only) operate on the format in the library without doing a full conversion. Generally, depending on type of format, in conversion or export operations calibre automatically updates the cover and metadata in the conversion output format or the exported copy, but not in the conversion input format or the format copied for the export.

Update Metadata, For All Books by Current Author
  • Edit Metadata for Consistency. For all books where Current Author is sole author or a coauthor, review and edit metadata for consistency with the metadata scheme, and consistency with metadata on the bibliographic site. Doing this for each processed Current Author over a long period of time helps to standardize metadata throughout the library, in an ongoing long-term process of metadata quality control that never quite reaches the goal of everything being consistent. Review and edit metadata in relevant columns at various levels: per author overall, per each set of coauthors where Current Author is the first author listed in Authors, per each series by the author, and per each varied-author series to which Current Author contributed books. Then Polish each book that has changed cover or metadata.
  • Edit Updated. For all books where Current Author is the first or only author in Authors, edit custom column Updated with the current year and month or the inactive code. (See Metadata > Custom Columns > Updated.)


Reading
Spoiler:

Overview
  • Purpose. Read books on reading devices.
  • Topics. Set Up for Loading, Load Device, Read Books, Revise Metadata, Set Up Device.


Set Up for Loading
  • Set Up New Device. If loading to a new device that has not yet been set up with calibre, see "Set Up Device" at the end of this section. Before buying a new device, consider size, weight, resolution, glare, color versus black and white, capability to handle preferred format, and general suitability per needs, strategies, inclinations, habits, types of preferred content. Generally for me, smartphone is too small, large tablet too heavy, 6 or 7 inch screen okay. I read most fiction or popular nonfiction on e-ink device (PRS-T2). I read technical, image-laden, or complexly formatted content on small tablet (Nexus 7). If I read them at all, I would read large-display content such as glossy magazines or comic books on computer or large tablet (old iPad).
  • Set Up a Way to Find Books to Load. This is an initial one-time action, supplemented by ongoing tagging. I use a saved search "To Load" which finds books with tags for "to read", "catalog", or "news". At any time, usually while processing books or browsing the library, for any books I want to read soon, I enter the tag for "to read" in custom column Read. Alternatively, many users rely on the plugin Reading List to manage reading lists and sync specific lists to specific devices.
  • Fetch News. Optional. In the Fetch News feature, initially schedule automatic News downloads, or use one of the manual Download buttons at any time. Calibre automatically converts news downloads to preferred output format and enters the tag "News" in Tags. For news downloads, I do not assign any other tags or fix formatting problems. Occasionally I use the News feature manually: select a source, download now, glance through the resulting book in Viewer, and if something looks interesting, leave the book in the library so I can load it on device later from the results list of the saved search "To Load" which includes news downloads. In the Fetch News dialog, I set "delete downloaded news older than:" to 30 days. In Preferences > Behavior, I unchecked "do not automatically send downloaded news to ebook reader" and unchecked "delete news from library when it is automatically sent to reader."
  • Create Catalog in Book Form. Optional. Create a catalog in book form that lists all the books in the library, prior to loading devices. Calibre automatically assigns the tag "Catalog" in the Tags column. My saved search "To Load" automatically includes that. I do this occasionally when I want to load a recent catalog onto a device. I set the catalog to show different symbols for wishlist placeholders and exclusion placeholders. See calibre User Manual, Creating AZW3, EPUB, MOBI Catalogs.
  • Choose Device and Loading Method. Choose the device to load, and choose a loading method. See calibre User Manual, Frequently Asked Questions, Device Integration. Loading methods include:

    • Calibre Companion. Send (push) from calibre through wireless connection between calibre on computer and Calibre Companion on device. Calibre Companion is currently available for devices running Android 2.3 or higher. I like this method because it is simple, does not require a USB cable, allows loading books onto device in bulk from calibre, and also allows removing books in bulk from device through calibre.
    • USB Cable. Send (push) from calibre through USB cable to device. For newer Android devices that require Media Transfer Protocol (MTP), including Nexus 7 and Kindle Fire HD, USB connections are supported on Linux and Windows Vista or newer, and not supported on OS X. For Apple iOS devices, use the optional device interface plugin iOS Reader Applications. For Kindle e-ink devices, other e-ink devices, or any device that the operating system recognizes as a USB drive: loading through USB cable works on Windows, OS X, and Linux (as far as I know). I like this method because it allows bulk loading from calibre, and also allows removing books in bulk from device through calibre.
    • Email. Send (push) from calibre through email to device. For various smartphones or tablets, books can be emailed to a mail account on the device then opened in a reading application. For Kindle, in calibre Preferences > Sharing > Sharing Books by email, amend the email addresses listed on Amazon > Manage Your Kindle > Personal Documents, add the word "free" after the kindle name and @ sign: "free.kindle.com." Calibre automatically includes a delay between emailing each book of a bulk selection, per the tweak in Preferences > Tweaks > "The number of seconds to wait before sending emails." I avoid the email method because the device must be connected to remove books in bulk from device through calibre, and it is simpler and faster to load in bulk while the device is connected, without the forced smtp delay.
    • App on Device, from Server. Pull wirelessly by web browser or reader app on device, using a catalog on server. Requires first generating an online catalog with calibre content server or alternative catalog-on-server tool such as Calibre-PHP Content Server, calibre2opds, or Calibre OPDS PHP Server. A web browser downloads the selected format then usually pops up a dialog box for choosing which reading application to open the book; this works on any device with a web browser. Or a reading app downloads the selected format directly; this works for tablet and smartphone reading apps with capability to access an online OPDS catalog. I usually avoid this loading method because it does not allow loading in one bulk operation, and the device must be connected anyway to remove books in bulk from device through calibre. For the rare occasions I do use the content server, in Preferences > Sharing > Sharing Over the Net my non-default settings are server port 7070, username Prime, password: my_password. (See Links > Catalog on Server.)
    • Cloud Syncing Service. Using Dropbox as an example (substitute a different cloud syncing service as desired): Pull wirelessly from a calibre library stored in a Dropbox folder on the computer by selecting a book link in the Dropbox folder on device, then selecting a reading application from the Dropbox popup dialog box. This presumably works on all devices on which Dropbox is installed, with various reading applications that can read a book with that file extension. I avoid this method because it does not allow loading in one bulk operation, the device must be connected anyway to remove books in bulk from device through calibre, and I minimize my use of clouds in general.

Load Device (For Each Device to be Loaded)
  • Connect Device. Ensure no other device is connected to the computer. Then connect wirelessly to load an Android device such as Nexus 7, or connect with USB cable to load other devices such as Sony PRS-T2, Kindle Touch, Kindle Keyboard, Apple iPad, or Apple iPhone.

    • Android. Ensure the wireless connection is started in calibre, in the Connect/Share menu, "Start or Stop wireless device connection." Turn on device. Open Calibre Companion on device and click the Connect button to start the wireless connection to calibre.
    • Other Device. Turn on device, and connect USB cable to computer and device. If iOS device, open relevant application per [Device Plugin] iOS Reader Applications.

  • Revise Metadata for Previously Read Books. For any previously read books still on the device, do the steps in the "Revise Metadata" subsection below if they were not done shortly after reading the books.
  • Remove from Device Previously Read Books. In calibre Library View select and remove from device the previously read books.
  • Find Books to Load. In calibre Library View, choose the Saved Search "To Load" to bring up a list of books to load to devices.
  • Select Books to Load. Select books in the "To Load" list to load for this device, according to the type of reading material or type of formatting best suited for this device.
  • Send Books to Device. In calibre Library View, send selected books to device.
  • Eject and Disconnect. In calibre, eject device. If connected by USB cable, disconnect it. If connected wirelessly through Calibre Companion, Calibre Companion automatically disconnects the wireless connection.


Read Books
  • Read Book(s). Read book(s) on device(s).


Revise Metadata, For Each Book Read
  • Edit Read. Soon after reading each book, in calibre Library View enter the content rating in Read and remove the tag for To Read. (See Metadata > Custom Columns > Read.)
  • Edit Format Quality Rating. If any formatting problems were discovered while reading, fix any that are more than mildly annoying right away, or downgrade the format quality rating in FormatQ. (See Metadata > Custom Columns > FormatQ.)
  • Edit Other Metadata. Edit any other relevant metadata, such as Keywords, to reflect newly gained knowledge about the book. (See Metadata section for details.)
  • Remove Throwaway Format. If AZW3, MOBI, or other format was generated from EPUB to read on Kindle, in the calibre library retain the EPUB master and remove the AZW3 or MOBI format. Some calibre users prefer to keep such extra formats, rather than remove them. I remove them. I will not reread a book again for at least two years. By then I will probably have newer and better reading devices. If or when necessary, with calibre I will convert-on-demand a fresh device-specific format from the EPUB.


Set Up Device (Initial One-Time Tasks)
  • Review General Device Settings. Optional.

    • Sending Books to Devices. These settings apply to devices in general. Metadata management: Automatic management, which syncs metadata in calibre library and metadata on device at every connect. A save template constructs the folders and filename for storing books on any device; this generic save template can be overridden for a specific device by customizing the save template in the Device Interface plugin for that device, which I have not needed to do. I limit author_sort to the first author_sort name, and shorten series. Title is automatically truncated when necessary, so there is no need to shorten title in the template. My save template for Sending Books to Devices is:
      {author_sort:sublist(0,1,&)}{series:shorten(3,-,16)| - | }{series_index:0>2s}{title:| - |}
    • Collections, Bookmarks, Annotations, Syncing. I manage books from within calibre and I load devices temporarily a few books at a time, so I do not need or use collections to organize books on devices. (See Metadata > Calibre Columns > Collections.) Also I do not use bookmarks or annotations. The only syncing I want is calibre's syncing of metadata per "Automatic Management" on any device, and Calibre Companion's syncing of metadata on connected Android device. Setup for collections, bookmarks, or annotations is usually different for different types of devices, involving different plugins, custom columns, or settings, which I avoid simply because I have no need to use those features.

  • Customize Relevant Device Interface Plugin. To customize a Device Interface plugin, in Preferences > Plugins > Device Interface Plugins select the plugin and click Customize Plugin button, or when the device is connected choose Configure this Device under the Location Manager icon. Generally all my device settings are default, except any settings for collections. For each device interface, I use the same save template as in Sending Books to Devices; and if the option is available "use sub directories" is checked. These are the Device Interface plugins I use now or recently used:

  • Create Metadata Plugboard(s) for Device. When a format is sent to a device or folder, the templates in the metadata plugboard for that device and format put metadata into the relevant destination fields inside the format. The metadata displayed on the device is usually read from the fields in the format. My metadata plugboards are:

    • Android Devices. Works with Calibre Companion. Format: any format. Device: SMART_DEVICE_APP.

      • Destination field authors:
        {author_sort:sublist(0,1,&)}
      • Destination field title:
        {series:|| }{series_index:0>2s}{title:| - |}

    • Apple iOS Devices. Works with optional device interface plugin: iOS Reader Applications. Format: any format. Device: APPLE.

      • Destination field authors:
        {author_sort:sublist(0,1,&)}
      • Destination field title:
        {series:|| }{series_index:0>2s}{title:| - |}

    • Kindle. Format: any format. Device: KINDLE2.

      • Destination field authors:
        {author_sort:sublist(0,1,&)}
      • Destination field title:
        {series:shorten(3,-,8)|| }{series_index:0>2s}{title:| - |}

    • Sony, Plugboard 1. Format: device_db. Device: PRST1. See (Sony)…How do I set it up..., thread Post #15, by jackie_w.

      • Destination field authors:
        {author_sort:sublist(0,1,&)}{series:shorten(3,-,16)| - | }{series_index:0>2s}{#genres:shorten(3,,0)| - |}

    • Sony, Plugboard 2. Format: any format. Device: PRST1.

      • Destination field authors:
        {author_sort:sublist(0,1,&)}{series:shorten(3,-,16)| - | }{series_index:0>2s}{#genres:shorten(3,,0)| - |}


Maintaining
Spoiler:

Overview
  • Purpose. Improve performance and minimize maintenance issues.
  • Topics. Maintain Libraries, Tune Performance, Update Software.

Maintain Libraries, For Each Library
  • Beware Making Direct Changes Inside Library Folders (Ongoing). In the operating system avoid making direct changes to the calibre Library folders, subfolders, and files. Make any desired changes through the calibre Graphical User Interface or the calibre Command Line Interface. Make exceptions to this rule only with beforehand knowledge of the potential dangers. For further discussion, see Want to change the folder structure of the Calibre library?, sticky by Manichean & Worldwalker
  • Backup Library and Configuration Folders (Frequently). Frequent backups are vital insurance against disastrous loss of the large investment of time, money, and work in ebooks. I created a workflow application (on OS X) that does automatic weekly backups of my calibre library folder(s) and configuration folder to a separate storage device. That is in addition to Time Machine conducting automatic backups of my system — including the calibre library and configuration folders — hourly, daily, weekly, and monthly to a third storage device. (The calibre configuration folder can be found by clicking the button Open calibre configuration directory in Preferences > Miscellaneous.)
  • Quarterly.

    • Remove Duplicates. Use the plugin Find Duplicates to search the entire library, in two passes: Title/Author with similar Title and similar Author; then Title/Author with similar Title and fuzzy Author. For each set of duplicates, ensure they are duplicates, if necessary correct metadata for whichever book has the better format quality rating, then remove duplicate(s) with worse format quality rating.
    • Clean Up Metadata. While in Find Duplicates plugin, use the command Find Metadata Variations, to clean up minor variations including typos. I do this for similar Authors, and fuzzy Series, Publisher, Tags. In the Tag Browser I similarly clean up variations in metadata in the custom columns Awards, Genres, Keywords, Types, and Source. Periodic clean up of metadata improves accuracy of searches.
    • Use Quality Check and Modify ePub. Use the plugin Quality Check to look for various problems in metadata and formats, and fix some problems, such as book sizes, without doing a conversion. Use the plugin Modify ePub to fix other problems identified by Quality Check, without doing a conversion.
    • Check Library. On the Library icon drop-down menu, select the Library Maintenance command Check Library. Check if the files in library match information in the database. If calibre reports any problems, look in the operating system in the calibre library folder at each problem to ensure each problem is okay to delete, but do not change or delete files there. To delete any problems, in the Check Library dialog box put a check in checkboxes for the problem files to mark them, click the Delete Marked button. Or click the Fix Marked button when it is available for any problems that calibre can fix automatically.

  • Update Awards (Annually). Once a year, review awards for best novel on Wikipedia or relevant award sites, and update the library for awards. (See Metadata > Custom Columns > Awards.)
  • Update Authors (Ongoing). Gradually during a long-term cycle, one by one, update each author for newly available books. (See Metadata > Custom Columns > Updated.)

Tune Performance
  • Set Antivirus Auto-Scan Exclusions (Once, and After Library Moves). Initially, set the security/antivirus application to auto-scan all storage devices and partitions, but to exclude calibre Library folders from scans. After moving a library, reset the exclusion for that library; I usually forget to do that until the slower performance makes me remember. When auto-scan is on, books are scanned for malware such as viruses at download time so there is no need to scan them again whenever calibre accesses book files, which can cause significant slow-downs in performance because calibre accesses a lot of files during the normal course of business.
  • Adjust Performance Settings (Once). Job priority (Preferences > Behavior) controls the speed and resource use of calibre worker processes; I keep this at the highest setting, Normal. Maximum simultaneous conversion/news download jobs (Preferences > Advanced > Miscellaneous) limits the number of CPU intensive background tasks; I set this to 6. Also I checked the option to Limit the maximum simultaneous jobs to available CPU cores (Preferences > Advanced > Miscellaneous). I chose these performance settings based on testing CPU intensive jobs such as bulk conversion on a computer with an Intel i7 quad-core processor, while leaving some capacity for the processes of other open applications.
  • Manage Composite Columns and Templates (Once and Ongoing). In Preferences > Add your own columns, remove unnecessary composite columns (custom columns built from other columns with templates). For remaining composite columns uncheck the checkbox Show in tag browser. If displaying formats in a composite column in the booklist, use the template {:'approximate_formats()'} rather than the template {formats}. Translate any complex template programming to General Program Mode, and ensure the tweak compile_gpm_templates is set to true.
  • Manage Column Coloring and Icons (Once). Limit or remove rules for column coloring and column icons in large libraries. For me so far the usefulness of column coloring outweighs the slight delay in library startup time.
  • Limit Searched Metadata (Once and Ongoing). In general, use column lookup names in all search criteria in the search bar, advanced search dialog, saved searches, and virtual libraries. In Preferences > Searching, turn on Limit the searched metadata, and enter just a few fields for non-prefixed searches, ensuring Comments is not one of them, to speed up search results when making a mistake typing the column lookup name or forgetting to use it. I limit non-prefixed searches to Authors and #aka.
  • Note, Defaults. To see what the default settings are for any Preferences pane except Plugins, click the button "Restore Defaults". Click the Cancel button to return to prior settings, or the Apply button to keep the default settings.
  • See. See for further information: Calibre Speedup (ways to fine-tune calibre performance), thread by Adoby.

Update Software
  • Update calibre. Updates of calibre are usually released every Friday. I like to install new versions right away, though it is not necessary so frequently.
  • Update Plugins. Updates of optional plugins that are available through calibre can be installed automatically with a few clicks from within calibre. For other optional plugins, periodically check the relevant plugin site and install any updates manually.
  • Update Other Software. To minimize the frequency of maintenance or performance issues, for computer and devices keep operating systems and other software such as editors, reader applications, and security software up to date. (I sometimes wait up to a year before doing major operating system upgrades, depending on how buggy they are at first.) I keep firmware up to date for all devices. So far I have had no need to jailbreak any devices, but if I did I would watch for and update software and firmware for each jailbroken device.


Library
Spoiler:

Overview
  • Purpose. A calibre library supports needs, strategies, workflow, and metadata scheme. A library is a structure that houses metadata, points to the book formats, enables searching and viewing metadata or contents to aid book retrieval or research, has sorting and ordering mechanisms, has need of librarian and janitor to keep it stocked and tidy, and so on.
  • Topics. Multiple Libraries versus Multipurpose Library, Library Structure, Tag Browser, Searching, Sorting, Column Structure, Order of Columns.

Multiple Libraries versus Multipurpose Library
  • Multiple Libraries. There are pros and cons either way to using one general library for various purposes, or multiple libraries each for a more specific purpose. Ideally each library would have a structure supporting a distinct and specific metadata scheme, defining the metadata elements for each column and associated conventions for use, all in a way that makes sense given the specific needs of the user. So the question is whether one library structure with corresponding metadata scheme is sufficient for all needs or not.

    • Multiple Libraries — Pro. Different libraries would better accommodate different library structures for different metadata schema with different metadata elements and conventions designed to achieve various other specific purposes. Cramming conflicting metadata schema together into one library increases complexity, resulting in a library that would probably be less useful and more time-consuming than several libraries, each specifically structured in conjunction with a distinct metadata scheme. Separate libraries could be useful in various scenarios, such as for privacy, security, or media type: adult library versus child library, shared library versus unshared library, any number of specific libraries each tailored for a specific publication type or media type with specialized custom columns. Another factor is that splitting a huge library into separate libraries reduces calibre startup time for each smaller library and reduces search time within them.
    • Multiple Libraries — Con. Each additional library equates to additional work in setup and maintenance. Many calibre commands apply only to selected books within the currently open library. Searching metadata in one large library is simpler and faster than searching sequentially in multiple libraries. Copying/pasting metadata or comparing books within one library is faster and simpler than across multiple libraries. Maintenance is faster in one large library as opposed to multiple libraries. Content server serves only the library from which it was started.

  • Multipurpose Library. I use one multipurpose library, named Prime. My needs are simple. I need just one permanent library, library structure, and metadata scheme tailored to my novel-centric reading habits. Prime also accommodates other publication or media types merely by entry of the metadata element for relevant type. I add books a few at a time to Prime, and process them there immediately. I store all processed books in Prime. I use virtual libraries within Prime. I test metadata changes and minor structural changes in Prime. Sometimes I test major structural changes in a temporary Test library created with Prime's structure, then delete the Test library after incorporating desired changes into Prime.

Library Structure
  • Library-Specific Structure. Calibre applies library-specific settings when starting up a calibre library. Each calibre library has its own library structure. When creating a new empty library with the option checked to copy structure from current library, all library-specific settings are transferred to the new library. When creating a new empty library with the option not checked to copy structure, calibre transfers only the default predefined columns and default predefined non-column fields as well as other default settings for the various aspects of library structure. Some aspects of library structure are:

    • Calibre Fields. Default fields, predefined, built in to calibre. Generally in calibre the term "column" is used loosely to mean "field" or "column". I use the terms more precisely here, in the sense that a column is one way of displaying data stored in a field. A field in a calibre library database stores metadata that can be displayed as a column, or not displayed as a column. A column in the calibre library booklist shows metadata stored in that field. For a selected book, fields editable by the user are presented in the Edit Metadata Individually dialog, or are shown or hidden in Book Details. (See Library > Column Structure > Calibre Columns and Other Fields.)
    • Custom Columns. User-defined columns that can be edited in Edit Metadata dialog, or hidden or shown in booklist or Book Details. When adding a custom column, the user enters the column lookup name and a column heading without the hash symbol prefix "#". Subsequently, in search criteria, the column lookup name includes the hash symbol prefix "#". (See Library > Column Structure > Custom Columns.)
    • Book Details. The Book Details pane is controlled by settings In Preferences > Look and Feel > Book Details. In library Prime, the fields shown on Book Details pane are: Authors, Series, Title, Identifiers, Date, #Genres, Formats, #FormatQ. I do not use Roman numerals for series, or show cover size. I show the cover in Book Details. I replaced the "default author link template" for Wikipedia with the Fantastic Fiction author link template copied from Search the Internet plugin:
      http://www.fantasticfiction.co.uk/search/?searchfor=author&keywords={author}
    • Column Alignment. Alignment of metadata in the column left, center, or right is set per right-click menu command under column heading. My text, comma separated text, and date columns are left aligned. My number columns are center aligned for more pleasing visual spacing between adjacent columns.
    • Column Coloring. Column coloring rules color text in designated columns according to specific metadata values in a specified column, as set in Preferences > Look and Feel > Column Coloring. I use text colors to emphasize the status of the format. See Metadata > Custom Columns > FormatQ.
    • Column Icons. Column icon rules assign icons to display in a designated column according to specific metadata values in a specified column. I do not use any column icons.
    • Column Size. The width of each column.
    • Column Visibility. Columns can be shown or hidden, per checkboxes in Preferences > Add Your Own Columns, or per right-click menu command under column heading. In library Prime I hide the columns Date, Modified, Languages, Rating, and Tags. I hide Date because I adequately see it per book in Edit Metadata Individually and Book Details. Modified, Tags, and Rating I do not use at all because I prefer my own custom columns for similar purposes. I do not use the Languages field. Occasionally I search a hidden column without showing the column. Note: I maximize horizontal space for seeing columns in the booklist by hiding the Tag Browser except temporarily while using it, and by using the configuration setting "user interface layout" Narrow (Preferences > Look and Feel) which positions Book Details below the booklist.
    • Other Library Structure. Additional aspects of library structure are discussed below under their own headings. There are a lot of library-specific settings; I did not list all of them. For details about a specific library's structure, in a text editor open the file "metadata_db_prefs_backup.json" (each library folder has one).

Tag Browser
  • Categories. Library structure includes settings for Tag Browser categories. I hide the Tag Browser categories Identifiers, Languages, News, Rating, and Tags. I do not have any additional custom categories defined, aside from the categories of custom columns that are shown in the Tag Browser — Awards, FormatQ, Genres, Keywords, Types, Read, Source. Saved searches I manage from the search buttons to the right of the saved search menu box in the search bar, rather than from the Searches category in the Tag Browser. In Preferences > Look and Feel > Tag Browser, "tag browser category partitioning method" is disabled, and "show average ratings in tag browser" is unchecked.
  • Hierarchical Categories. Library structure includes settings for whether or not a category is hierarchical. I use hierarchical metadata for Series. But I do not need to see Series in a hierarchical breakdown in the Tag Browser per Preferences > Look and Feel > Tag Browser > Categories with hierarchical items, which I keep set to the default "none". I use semicolon space "; " as subgroup separator in Series because it looks better than the period "." separator.
  • Tag Browser Searches. For me using the Quick Search box in the search bar is faster and more flexible than using the Tag Browser for searches. Taking the time to learn the search syntax was well worth it. I rarely do searches from the Tag Browser.
  • Managing Metadata Elements. I use the Tag Browser to manage metadata elements when there are a lot of them to rename or delete, usually in conjunction with restructuring columns or metadata elements. The plugin Find Duplicates has a very nice feature for finding and editing metadata variations in Authors, Series, Publisher, and Tags. Using the Tag Browser I also periodically clean up the minor variations for tags in the categories not addressed in the Find Duplicates plugin, for variations that really mean the same thing. Clean, standardized, non-varying metadata elements are easier to remember when entering searches, and enable more accurate search results. They also look better in Booklist and Book Details.

Searching
  • Quick Search. For searches I use the search bar above the booklist, rather than the Tag Browser or the Advanced Search dialog. I prefix most search terms with column lookup name — or with a grouped search term which uses column lookup name(s) — to get results faster than without lookup names. I limit non-prefixed searches to only authors and #aka; this allows typing into the quick search box just author first name, last name, or whole name without column lookup name or quotes, getting results as quickly as if the lookup names were used. See calibre User Manual, Search Interface. Example quick searches:
Code:
laurell k
series:blake
series:"anita blake"
title:"guilty pleasures"
#source:amazon
hamilton
  • Grouped Search Terms. Grouped search terms are an aspect of library structure set in Preferences > Searching. They are convenient for combining more than one column into one search term, or as shortcut aliases for lookup names of single columns. Enter the term name and column(s) for a new grouped search term then Save it and Apply, even if the desired column is not represented in the drop-down list, then restart calibre before trying to use the new term. Below are the grouped search terms I use in library Prime, each with an example "contains" search.
Code:
Term   Column Lookup Names(s)     Example Search   Explanation
----   ------------------------   --------------   ------------------------
a      authors, #aka              a:gaiman         Gaiman in either column
c      comments                   c:assassin
f      formats                    f:pdf
s      series                     s:"rogue angel"  
t      title                      t:mythago

aw     #awards                    aw:h             Hugo award winner
fq     #formatq                   fq:f1            wishlist
g      #genres                    g:fn             fantasy
k      #keywords                  k:mil            military
pp     #pages                     pp:<150          potential shorts
r      #read                      r:TR             books To Read
so     #source                    so:mr            books from MobileRead     
ty     #types                     ty:om            omnibuses
up     #updated                   up:1310          updated author Oct 2013

gk     #genres, #keywords         gk:fn            fantasy in either column

zz     title, series, comments,   zz:fantasy       fantasy in any of these
        #genres, #keywords        or zz:fn         five columns
  • Virtual Libraries. Virtual Libraries are an aspect of library structure. Virtual libraries use search criteria to partition sets of books within one library. In library Prime I have virtual libraries for Fiction and Nonfiction, based on saved searches with the same names. I also use temporary virtual libraries for Updated. It is convenient to switch quickly between the tab for "all books" and any virtual library, automatically switching the selection to the top of the virtual library booklist or the relevant place in "all books" booklist. See calibre User Manual, Virtual Libraries.

  • Saved Searches. Saved searches are useful as the basis of virtual libraries, or for quick reuse of frequent searches. Saved searches are an aspect of library structure. See calibre User Manual, Saving Searches. Some of the saved searches in library Prime are:
Code:
Search Name       Search Criteria                          Explanation
---------------   -------------------------------------    -----------------
To Load           #read:TR or tags:Catalog or tags:News    TR is "To Read"
Format Problems   not (#formatq:f4 or #formatq:f5)         f4 good, f5 great
NewBooks 10days   date:>=10daysago                         within timeframe
Authors > 1       authors:#>1                              coauthors exist
calibre-auto      tags:news or tags:catalog                automatic tags
Fiction           not search:"=NonFiction"
Nonfiction        #genres:nf or tags:news or tags:catalog

Sorting
  • Sort History. Calibre saves column sort order and associated column sort direction in a sort history for each library. This is another aspect of library structure.
  • Sort-Related Tweaks. Tweaks are not library-specific structure, but are configuration settings applied in any calibre library. Sort-related tweaks need to be considered because they may affect sort order of book records, or the number or order of sorted columns. In Preferences > Tweaks, search for "sort" to find them. For example, I set sorting of titles and series to 'strictly-alphabetic' for both library display and templates, primarily because that is how sorting is done at the bibliographic site I use most frequently, International Speculative Fiction Database (ISFDB). And I set dates to be sorted using visible fields.
  • Sort Order and Keyboard Shortcuts. Keyboard shortcuts are another type of configuration applying to any library. A Keyboard Shortcut set with the plugin View Manager quickly resets sort order to a desired pattern, and moves a selected newly added book from the top of the booklist resulting in the book being sorted into its sort position, centered in the booklist display. I use View Manager keyboard shortcuts frequently. A small price to pay for that convenience is having to manually sync the two views in View Manager with calibre's column order for a newly added column or a change in column order. The keyboard shortcuts I set in View Manager are:

    • Preferred Sort Order. The keyboard shortcut Command 1 resets to the view "Main TSA - Cmd 1" which is my static column order and preferred sort order: Title within Series within Authors, all ascending.
    • Alternate Sort Order. The keyboard shortcut Command 2 resets to the view "Main TS - Cmd 2" which is my static column order and alternate sort order for working with varied-author series: Title within Series, both ascending.


Column Structure
  • Calibre Columns and Other Fields in Library View. Some of these default calibre columns and other fields in Library View of my library Prime also have corresponding configuration or library-specific settings that are not default. See Metadata section for description of each column or other field.
Code:
Calibre Columns                 S  Shown             Visibility
and Other Fields                H  Hidden            ----------
in Library View                 -  not applicable    Tag browser   E
                                *  See description   Book details  d  Groupd
--------------------------------------------------   Column        i  Search
                                         Separator   Edit dialog   t   Term/
Name          Lookup         Type        or Format   T  B  C  E    ?   Alias
-----------   ------------   ---------   ---------   ----------   ---   ----
Authors       authors        text list   &    name   S  S  S  S   yes      a
Author Sort   author_sort    text list   &    name   -  H  -  S   yes
Comments      comments       text long               -  S  -  S   yes      c
Cover         cover          y/n >file        >jpg   -  S  -  S   yes
Date          date           date       yyyy-MM-dd   -  S  H  S   yes
Formats       formats        text list   ,    >ext   S  S  -  S   act      f
id            id             integer                 -  -  -  -    no
Identifiers   identifiers    text list   , label:x   H  H  -  S   yes
Languages     languages      text list   ,           H  H  H  S   yes
Marked        marked         yes/no      pin RowNo   -  -  -  -   act
Modified      last_modified  date       yyyy-MM-dd   -  H  H  -   act
On Device     ondevice       yes/no      checkmark   -  -  S  -   act
Path          path           text                    -  H  -  -   act
Published     pubdate        date          yyyy-MM   -  H  S  S   yes
Publisher     publisher      text                    S  H  S  S   yes
Rating        rating         integer         stars   H  H  H  S   yes
Series        series         text                    S  S  S  S   yes      s
Series Index  series_index   floatg pt     {0:.2f}   -  -  *  S   yes
Size          size           floatg pt     {0:.1f}   -  -  S  S   act
Tags          tags           text list   ,           H  H  H  S   yes
Title         title          text                    -  S  S  S   yes      t
Title Sort    title_sort     text                    S  H  -  S   yes
uuid          uuid           hex num                 -  H  -  -    no
  • Calibre Columns in Device View. Some of these default calibre columns in Device View of my library Prime also have corresponding configuration or library-specific settings that are not default. Collections is a special case. See Metadata section for description of each column.
Code:
Calibre Columns                 S  Shown             Visibility
in Device View                  H  Hidden            ----------
                                -  not applicable    Tag browser   E
                                *  see description   Book details  d  Groupd
--------------------------------------------------   Column        i  Search
                                         Separator   Edit dialog   t   Term/
Name          Lookup         Type        or Format   T  B  C  E    ?   Alias
-----------   ------------   ---------   ---------   ----------   ---   ----
Authors       authors        text list   &           -  S  S  -   act      -
Collections   collections    text list   ,           -  -  S  -    *       -
Date          timestamp      date       yyyy-MM-dd   -  S  S  -   act      -
In Library    inlibrary      yes/no      checkmark   -  -  S  -   act      -
Size          size           floatg pt     {0:.1f}   -  -  S  -   act      -
Title         title          text                    -  S  S  -   act      -
  • Custom Columns. These are the custom columns in library Prime. See Metadata section for description of each column.
Code:
Custom Columns                  S  Shown             Visibility
                                H  Hidden            ----------
                                -  not applicable    Tag browser   E
                                *  see description   Book details  d  Groupd
--------------------------------------------------   Column        i  Search
                                         Separator   Edit dialog   t   Term/
Name          Lookup         Type        or Format   T  B  C  E    ?   Alias
-----------   ------------   ---------   ---------   ----------   ---   ----
aka           #aka           text list   &           S  H  S  S   yes      a
Awards        #awards        text list   ,           S  H  S  S   yes     aw
FormatQ       #formatq       text list   ,           S  S  S  S   yes     fq
Formats2      #formats2      composite           *  no  H  S  -    no      -
Genres        #genres        text list   ,           S  S  S  S   yes      g
Keywords      #keywords      text list   ,           S  S  S  S   yes      k
Pages         #pages         integer         {0:,}   H  H  S  S   yes     pp
Types         #types         text list   ,           S  H  S  S   yes     ty
Read          #read          text list   ,           S  H  S  S   yes      r
Source        #source        text                    S  H  S  S   yes     so
Updated       #updated       integer                 -  H  S  S   yes     up

Order of Columns
  • Presented Order. Library structure includes the presented order of columns and other fields in dialog boxes and book details, and the presented order of columns in the library view and device view booklists.
  • Workflow Order. One practical way to order columns in the library view booklist is by workflow steps for processing new books.
  • Workflow Clusters. In workflow-step order for processing new books, the columns cluster by general workflow task: Add books, Edit metadata, Convert/Edit formats, and Update metadata.
  • Level Groupings. It is also useful to group columns by the level at which the metadata elements are applied to books in bulk. I apply metadata at these levels:

    • Format-level. I keep only one format per book record because I want to avoid confusion arising from multiple formats and various format-level tags existing in one book-level database record. A one format per book rule allows assigning format-level metadata at book-level with confidence and without having to prefix each format-level tag with format type. I briefly may have an additional format temporarily for a specific purpose, but I always know which format is the master digital format, I always remove the temporary format as soon as possible, and I know the format-level tags apply to the master digital format, except for one tag indicating paper book owned.
    • Book-level. One authors/title combination can represent many separate publications, each a different publisher, language, edition, binding, media type, etc. Calibre uses a mix of title-level and publication-level. For example, duplicate matching occurs at authors/title level, but many of the Metadata Download fields are publication-level. So I combine publication-level metadata with title-level metadata into one level and call that book-level. When necessary for different publications with identical authors/title, I append publication-level metadata to the title to make authors/title unique, which equates to book-level unique.
    • Series-level. I incorporate series hierarchy with series name in just one Series column, and if relevant, also append a tag in bulk to series name indicating that the book is a member in a varied-author series. I also often apply genres at series-level, though I categorize genre as book-level. When entering metadata for a series by identical set of coauthors, I ensure the order of the coauthors in Authors is the same for every book within the series.
    • Author-level. In author-level custom columns I apply metadata pertaining to the first or only author listed in Authors, across all of that author's books.

  • Metadata Cliques. I prefer working with columns in one static all-purpose order. Switching between different column orders with View Manager plugin to achieve specific purposes seems dissonant to me. After some trial and error, eventually I embedded workflow-step order within metadata-level group within workflow-cluster then adjusted for practicality and look/feel. As a result, the columns congregated in stable cliques that are useful in a static-order all-purpose view of metadata. In my library Prime, the metadata cliques are:

    • Core. Authors, Author Sort, Series, Series Index, Title, Title Sort, Date (date), Modified, id, uuid, Path, Marked.
    • Device. OnDevice, inLibrary [device], Authors [device], Title [device], Date (timestamp) [device], Collections [device], Size [device].
    • Fetch. Cover, Comments, Identifiers, Published, Publisher, Languages, Rating, Tags.
    • Book Custom. #Genres, #Keywords, #Types, #Awards, #Read.
    • Format. Formats, #Formats2, #FormatQ, #Pages, Size, #Source.
    • Author Custom. #aka, #Updated.

  • Static Column Order in Library View. In Library View booklist the static order of my columns is: OnDevice; Authors, Series, Title, Date (hidden), Modified (hidden); Published, Publisher, Languages (hidden), Rating (hidden), Tags (hidden); #Genres, #Keywords, #Types, #Awards, #Read; #Formats2 (showing Formats), #FormatQ, #Pages, Size, #Source; #aka, #Updated. The position of #Source is a special case, with its clique but outside of workflow order. (See the Metadata section for details.)
  • Static Column Order in Device View. In Device View booklist the static order of the limited number of columns is somewhat congruent with the order in Library View: inLibrary [device], Authors [device], Title [device], Date (timestamp) [device], Collections [device], Size [device]. (See the Metadata section for details.)
  • Static Custom Column Order in Edit Metadata Dialog. I set the order of custom columns presented in Edit Metadata dialog to be congruent with the Library View booklist order, per Preferences > Advanced > Tweaks > Order of custom columns in Edit Metadata.


Metadata
Spoiler:

Overview
  • Purpose. Define general metadata terms and guidelines. Discuss the specific metadata scheme I apply in library Prime.
  • Topics. Metadata Terms and Guidelines, Calibre Columns and other Fields in Library View, Calibre Columns in Device View, Custom Columns.


Metadata Terms and Guidelines (For Any Library)
  • Metadata Terms. Metadata is data about data. In a scenario of managing books, the books are the data, metadata describes the books, and a metadata scheme organizes and describes the metadata in a way that facilitates managing and using books. A metadata scheme includes definitions for each column or other field, definitions for each metadata element, and conventions for using them. For related terms and concepts, see Links > Metadata Terms.
  • Amount of Metadata. Limit the number of columns, types of tracked metadata, and metadata elements to the fewest necessary. The more there are, the more time necessary for processing each book and maintaining each library. Do not use a calibre column or custom column unless it meets a specific need that otherwise cannot be met as easily.
  • Distinct Attribute. With the exception of Tags, each default calibre column or other field targets a different distinct attribute of books. Ensure each added custom column does that too, and its column name identifies the targeted attribute. Ensure that there is a good reason to keep local metadata for each attribute, and that this metadata somehow facilitates the main purpose of the library.
  • Metadata Elements Represent Attribute. Ensure the metadata elements in each column or other field specifically represent the distinct attribute (except possibly the column Tags depending on its purpose in the library). For example, in a column targeting the attribute "publication type" each metadata element represents a specific publication type (collection, omnibus, anthology, and so on). In a column targeting the attribute "award" each metadata element represents a specific award (Hugo for novel, Nebula for novel, and so on).
  • Unique Metadata Element Name. Ensure the name of each metadata element is unique within its column or non-column field.
  • Abbreviating Metadata Element Names. If using abbreviations for metadata element names, ensure each abbreviation is easy to remember and provides a clue to its meaning. Easily memorized abbreviations, sounding indicative of the full word or phrase when spoken aloud, are almost as human-readable, and often more immediately human-readable than the full word or phrase. Particularly for "comma separated text" columns, on a small computer screen displaying a library booklist with limited column widths, it is not always possible to see at a glance unabbreviated metadata off to the right of the visible portion without doing an extra step of scrolling within the cell, or hovering the cursor to see the tooltip. Abbreviations are convenient for seeing at a glance more metadata in a smaller horizontal space. On the other hand, the more people using the library, the more potential confusion is caused by abbreviations.


Calibre Columns and other Fields in Library View (Definitions for Library Prime)
  • List of Default Calibre Columns and other Fields in Library View. Authors, Author Sort, Comments, Cover, Date (date), Formats, id, Identifiers, Languages, Marked, Modified, OnDevice, Path, Published, Publisher, Rating, Series, Series Index, Size, Tags, Title, Title Sort, uuid.
  • Authors (authors). Calibre column, comma separated text, names, editable. For author name or a list of coauthor names. Rule: must contain at least one author name. I apply the following conventions for Authors:

    • Coauthors. For books with coauthors, ensure all of them are entered and in correct or desired order. The order of coauthors and the choice of which coauthor is first affects sorting the booklist by Authors in library, sorting on devices, and which author name is sent through metadata plugboards or save templates that use only the first author in Authors or Author Sort. For a series with books all written by the same set of coauthors, in Authors enter the identical order of coauthors for each book in that series. Calibre separates the names in the list with space ampersand space.
    • FirstName LastName versus LastName FirstName. I use the convention FirstName LastName. This is helpful with some internet sites. For example the International Speculative Fiction Database (ISFDB) site wants the FirstName LastName convention pasted into its Search box. One way to swap First and Last name is to use the plugin Quality Check, with search scope set for selected books, Fix menu, Swap author, while being mindful of the Tweak setting for Author sort name algorithm and the potential need to revise Author Sort in Manage Authors. See Author Sort below.
    • Author Initials. I use a period (dot) after each initial in author name, primarily because the search box on ISFDB requires them.
    • Author Last Name Prefixes. If there are prefixes in an Author last name, correct any related Author Sort problems. See Author Sort.
    • Author Last Name Suffixes. If a last name has a suffix such as Jr, Sr, or III, in Authors delete the suffix along with any preceding comma or following period, then after downloading metadata enter the full name including suffix at the top of Comments. I delete the suffix in Authors because a comma before the suffix may confuse the Author Sort algorithm depending on how that is set, and when using an author template or directly copying/pasting to the search box of an internet site, the lack of the suffix also avoids some trouble with certain sites wanting or not wanting a comma before the suffix, the suffix itself, or the period. Removing the suffix from the name in Authors is a simple solution that does not work in all cases but avoids the need to implement more complex solutions for each site-specific author template, which still might not solve all cases for each site. See templates for various sites in the plugin Search the Internet.

  • Author Sort (author_sort). Calibre field, text, automatically filled, editable. Shown in Edit Metadata Individually. Calibre sets Author Sort automatically according to the Tweak Author sort name algorithm. I use author_sort_copy_method = 'comma' in conjunction with the FirstName LastName convention in Authors, and ensure there is no comma in Authors to interfere with the author sort. Rule: one to one correspondence with Authors. See calibre User Manual, FAQ, How does calibre manage author names and sorting?

    • LastName Prefixes. Potential sort problems can occur with prefixes that belong in the LastName such as: de, de la, del, le, van, St (regardless of upper or lower case). I want Author Sort to be set as the author likes it set, where possible, such as de la Cruz, Melissa or van Vogt, A. E. rather than Cruz, Melissa de la or Vogt, A. E. van. Consult a bibliographic site when unsure how to sort an author name (with the understanding that the site might be wrong).
    • Editing Author Sort. To fix Author Sort problems, select one book by any author, in Edit Metadata click the head and shoulders icon to the left of Author(s) field to get to Manage Authors, and manually correct Author Sort for any author with incorrect Author Sort.

  • Comments (comments). Calibre field, long text, editable in Edit Metadata Individually, shown in Book Details. For a short synopsis or review of the book, obtained through Metadata Download, or by copying/pasting from a bibliographic site. Links or additional notes can be added. For most books I prefer limiting Comments to one non-spoiler synopsis, with no reviews. Sometimes I add a link or a brief note at the top of Comments. Rule: a book can have zero to many synopses, reviews, notes, or links.
  • Cover. Calibre field, yes/no, in conjunction with an image file. Calibre automatically stores the image file as cover.jpg along with the format in the book folder in the calibre library folder. The image file is editable and is displayed in Book Details, Edit Metadata Individually, Cover Grid, and Cover Browser. Obtained through Metadata Download, copy/pasted from a site (bibliographic, author, publisher), or by clicking the book icon in Edit Metadata Individually to set from the selected format. Calibre rule: zero to one cover per book. My rule: there must be a cover for each book.

    • Alternate Cover. When the correct cover for a specific publication is unavailable, use any available cover from any other publication of the book. If none are available use the standard calibre cover.
    • Editing Cover. Trim borders manually or automatically in Edit Metadata Individually.
    • Cover in Format. Depending on the type of format, calibre usually inserts the cover into the exported or conversion output format if necessary when calibre exports or converts a book. Insert the cover into the format in the library using Polish Books (EPUB or AZW3) or the plugin Modify ePub (EPUB). I generally insert cover and metadata into EPUB format with Polish Books near the end of the Processing workflow.

  • Date (date). Calibre column (in Library View), date, automatically filled, editable. Date the book was added to library. I format it as yyyy-MM-dd per the Tweak control how dates are displayed. I keep this column hidden, but show Date in Book Details. Calibre rule: must be one date in Date. My rule: edit only for good reason, while preserving the meaning of "date book was added."
  • Formats (formats). Calibre field, comma separated text, automatically filled, not editable, in conjunction with the format file. Calibre automatically stores the format in the book folder in the calibre library folder. Calibre displays the format names with filetype icons in the formats box in Edit Metadata Individually, where user can add or remove formats. Calibre rule: a book can have zero to many formats. My rule: after processing, a book can have zero to one format in Formats. For me this rule avoids confusion.

    • Original Format Tweaks. The tweaks in "save original file when converting/polishing from same format to same format" affect whether an original format is saved or not with same-to-same Convert or Polish. To allow multiple conversions from the same original, I set save original format to True (default). Because polishing is not as potentially destructive as converting, I set save original format when polishing to False (not default).
    • Master Format. During processing, I fix any formatting problems, then remove all the formats for each book except the one format I want to keep as master. In most cases I keep only the EPUB; in rare cases I keep only the PDF.

  • id (id). Calibre field, integer, automatically filled, not editable. The identification number of a book, unique within a library. Automatically assigned in increments of 1 when book is added or copied to a library. In the library folder calibre also appends "id" onto title folder. In the save template for Saving Books to Disk, I append "id" onto the title folder to ensure there will be no author/title duplicates going to the file system. Rule: one library-unique id per book.
  • Identifiers (identifiers). Calibre field, comma separated text, automatically filled by Metadata Download, editable in Edit Metadata Individually. For ISBN10 or ISBN13 and other identifiers such as retailer-id for that book. Calibre constructs identifiers into links to the respective page for that book on the relevant site, clickable in the Book Details pane. Before Metadata Download, extracting a good ISBN from the format with the plugin Extract ISBN can help get metadata specific to the edition with that ISBN. Otherwise, delete identifiers first before doing another Metadata Download to try for better metadata. Generally I do not edit Identifiers, do not care if there is an ISBN or not, and seldom use the links. Rule: there can be zero to many identifiers per book.
  • Languages (languages). Calibre column, comma separated text, editable. For the language of the book. I fill this column through metadata download, and hide the column because I do not use it. Calibre rule: zero to many languages per book.
  • Marked (marked). Calibre field, yes/no, not directly editable. Shown with a pushpin next to the row number for each marked book. Set or cleared through the Mark Books feature, or through several optional plugins such as Find Duplicates and Quality Check. I do not use this field except indirectly with those plugins. Marks set by Find Duplicates are the only marks that persist through a calibre restart; otherwise marks are for temporary use. Rule: zero or one mark per book.
  • Modified (last_modified). Calibre column, date, automatically filled, not editable. Shows the last time changes were made to metadata or cover, or formats were added/removed. I format Modified as yyyy-MM-dd per the Tweak control how dates are displayed. I sort dates by visible fields, per the tweak Choose whether dates are sorted by visible fields. I sort dates by visible fields, per the tweak Choose whether dates are sorted by visible fields. I hide this column. Rule: none.
  • On Device (ondevice). Calibre column (in Library View), text, automatically filled, not editable. Column appears when device is connected. Shows with checkmark which books in the library are also on device, and shows the location on device (main or memory card). Changes by adding or removing books from device. Rule: none.
  • Path (path). Calibre field, text, automatically filled, not editable. The path to the book in the book folder in the library folder. Open the book folder with keyboard shortcut O. Alternatively, if Path is shown in Book Details, open the book folder with click to open. Path changes automatically with edits to Titles or Authors, and with move of library followed by pointing calibre to the new location. Rule: each book automatically has one unique Path.
  • Published (pubdate). Calibre column, date, editable. For date of publication. Obtained through Metadata Download, the publication date may need to be corrected per the publication data on a bibliographic site. I use date of first publication for fiction or popular nonfiction, or the date of a specific revised edition when that is important such as for technical books. I format the column as yyyy-MM per the tweak control how dates are displayed. I sort dates by visible fields, per the tweak Choose whether dates are sorted by visible fields. Calibre rule: zero to one publication date. My rule: must be one publication date for each book except exclusion placeholder.
  • Publisher (publisher). Calibre column, text, editable. For name of publisher of the book. Obtained through Metadata Download. Calibre rule: zero to one publisher per book. My rule: must be one publisher for each book except exclusion placeholder.
  • Rating (rating). Calibre column, integer, shown as stars, editable. The scale goes from zero to five stars, five the highest rating. I do not use this column, and keep it hidden. Calibre rule: must be one rating per book.
  • Series (series). Calibre column, text column for keeping series-like information, editable. The column displays series name (text), followed by bracketed series index (floating point number), both editable. Calibre rule: zero to one series name and index per book. My rule: must be one series name and index per book. The conventions I use in Series are:

    • Standardized Series Name. Enter or correct Series name and index according to the series name and series index used by a bibliographic site with consistent series metadata. This is important particularly for series with books by varied authors, for which the series name and indexing should be consistent across various authors.
    • Individual Books. For individual books that are not in a series, I enter (1) in Series, and 1 for Series Index. This displays as "(1) [1]" in the Series column, sorts above other series names in the booklist, and displays as "Book 1 of (1)" in Book Details. Rule: there must be at least one series name for each book, even if it is an individual title not actually in a series.
    • Hierarchical Series. When necessary, I use this convention for large series with multiple levels of subseries: unique Broadest Series Name Abbreviation; subseries name; most specific Sub-subseries name.... I prefer the appearance of semicolon space over the standard calibre convention of period no_space preceding each child subgroup. (See calibre User Manual, Managing subgroups of books.) For me the period preceding a subgroup is not necessary because I do not need to see a Tag Browser breakdown of series by subgroup. I apply the series index for the most specific subseries exactly as given at the ISFDB or Fantastic Fiction site, with another higher level of indexing accomplished when necessary by appending to a higher-level series name a number that sorts subseries in the desired order. I search for the Broadest Series Abbreviation or a relatively unique word or phrase contained in Series. Generally I avoid large series with multiple levels of subseries.
    • Unique Series Name. Except for individual books with series name (1) and index 1, ensure series name is unique, by appending author last name or initials in parentheses.
    • Varied Author Series. When a series contains books with different author(s) between at least two of the books, for all books in that series append to series name a space and dollar sign " $". This provides a clue to see the entire series with sort by title within series, rather than sort by title within series within author. Generally I avoid varied-author series with books by more than three different authors or sets of coauthors.

  • Series Index (series_index). Calibre field, floating point number, formatted for 2 decimal places {0:.2f}, editable. Displayed in conjunction with series name in Series column, and separately in Edit Metadata Individually. Rule: if there is a series name, there must be a series index number, ranging from 0 to higher (no negative numbers). I use these conventions for series index:

    • Publication or Internal Order. Many series have books where the publication order of the books is the same as the internal order of events for the characters. If those orders are different, I may change the series names and series index from publication order to internal order, if it does not have too many hierarchical levels, I have all of the books to date, and the recommended internal chronology is published on an official site or in one of the books. For example, for Valdemar series by Mercedes Lackey, I changed all the small series names to Valdemar and indexed the books with relevant Valdemar calendar year; and for the Recluse series by L. E. Modesitt Jr, I indexed using the relevant Recluse calendar year.
    • Auto-Increment. I leave at default the tweaks for auto-increment series_index: series index auto-increment = 'next', and use series auto-increment tweak when importing = False.
    • Indexing per Publication Type. I apply different indexing conventions depending on type of publication. For details, see Metadata > Custom Columns > Types.

  • Size (size). Calibre column (in Library View), floating point number, formatted for one decimal place {0:.1f}, automatically filled, not editable. The size in megabytes of the largest format in the book folder, automatically revised when formats are added or removed using calibre commands. If any editing is done outside of calibre to that format while it resides in the calibre library folder, the value shown in Size column may not show the changed size until after running the command Fix > Check and repair book sizes with the plugin Quality Check. Size is not an accurate indicator of book length.
  • Tags (tags). Calibre column, comma separated text, editable. For tags. I hide this column. I do not download tags in Metadata Download, because assigning my own tags in specific custom columns saves time by not having to correct or delete downloaded tags, while allowing better consistency and organization. Calibre rule: a book can have zero to many tags in the Tags column. My rule: a book must have zero tags in the Tags column, except for at most one calibre-automatic tag. In my usage, the only metadata elements in Tags are:

    • News. News download, automatically tagged by calibre. This is technically a publication type, but since calibre already tagged it here, I do not take the time to duplicate the tag in custom column Types. When necessary I search for tags:news.
    • Catalog. Catalog of books in book format, automatically tagged by calibre. This is technically a publication type, but since calibre already tagged it here, I do not take the time to duplicate the tag in Types. When necessary I search for tags:catalog.

  • Title (title). Calibre column, text, editable. Calibre uses Title as one component of a book format's filename when storing it in the operating system's file system. Calibre truncates the title in filename and changes non-compliant characters to underscore as necessary to stay within system limitations, while leaving Title unchanged in the database as seen in the booklist or Book Details. For information about system limitations for path and filename, see Links > Filename Limitations. Calibre rule: each book must have one title. My rule: the combination of Authors plus Title must be unique at book-level within the library. I use these conventions for Title:

    • Subtitle. I delete subtitle, unless the bibliographic site uses the subtitle as part of the title, in which case I change the preceding (colon)(space), to (dash)(space). Note: calibre automatically changes colon to underscore in the filename.
    • Unique Authors/Title. After Metadata Download, if necessary I append brief publication metadata to the title in parentheses to make the authors/title combination unique, in cases where alternate editions, versions, or titles are available now or will be in future, for example: (3rd Ed), (Anniv Ed), or (vt The Variant Title). I avoid a lengthy title by entering any lengthy information in Comments. Other users may employ custom columns for Edition or Title2, or use separate empty book records for alternate titles.

  • Title Sort (title_sort). Calibre field, text, automatically filled, editable. Shown in Edit Metadata Individually. I prefer strictly alphabetic sort order per tweaks control sorting of titles and series in the library display, and control formatting of title and series when used in templates, because International Speculative Fiction Database (ISFDB) is sorted that way. Rule: one to one correspondence with Title.
  • uuid (uuid). Calibre field, hex number, automatically filled, not editable. Can be seen in Book Details. uuid is a hex identifier, practically unique in the universe. Calibre uses uuid internally. Rule: one uuid per book.


Calibre Columns in Device View (Definitions for Library Prime)
  • List of Default Calibre Columns in Device View. Authors [device], Collections [device], Date (timestamp) [device], InLibrary [device], Size [device], Title [device]. Note that these columns display calculated metadata determined by internal calibre matching and decision making when device is connected.
  • Authors (authors) [device view]. Calibre column, comma separated text, names, filled automatically, not editable. Shows authors for books on device. Rule: one or more authors per book.
  • Collections (collections) [device view]. Calibre column, comma separated text, potentially automatically filled, potentially editable. Set up with metadata management options in Preferences > Sending books to devices, in combination with any relevant tweaks, perhaps with the aid of a specific device interface plugin, depending on specific device. Collections are editable in Device View for some combinations of those options for some devices. I do not use collections on devices at all because: first, I load devices only a few books at a time so do not need collections to organize them there; second, collections cannot be used in a standard way across all my devices because they are device-dependent, requiring in some cases various complicated setups including a dedicated custom column, and collections support from calibre is not available for all devices. Rule: zero to many collections per book.
  • Date (timestamp) [device view]. Calibre column, date, automatically filled, not editable. The date the book was sent to device from calibre or otherwise arrived on the device. I format it as yyyy-MM-dd per the Tweak control how dates are displayed. Rule: one date per book.
  • In Library (inlibrary) [device view]. Calibre column, yes/no, automatically filled, not editable. Shows with checkmark which books on the device are also in the library. In debug mode, also shows method used in matching. Rule: zero or one checkmark per book.
  • Size (size) [device view]. Calibre column, floating point number, formatted for one decimal place {0:.1f}, automatically filled, not editable. The size in megabytes of the format on the device. Rule: one size per book.
  • Title (title) [device view]. Calibre column, text, filled automatically, not editable. Shows title of book on device. Rule: one title per book.


Custom Columns (Definitions for Library Prime)
  • List of Custom Columns in Library Prime. aka, Awards, Formats2, FormatQ, Genres, Keywords, Pages, Read, Source, Types, Updated.
  • aka (#aka). Custom column, comma separated text, editable, contains names. For alternate name(s) of the first or only author in Authors, separated by space-ampersand-space, using the same conventions as in Authors. The acronym aka means "also known as". Rule: a book can have zero to many aka names.
  • Awards (#awards). Custom column, comma separated text, editable. For winning award in the novel category. I do not track awards for categories other than novel. Once a year, I review the awards for best novel on Wikipedia or the relevant award sites, for awards given the previous year (and prior years if necessary). If any of the winning novels or novel runners-up are not yet in the library, I buy and process them, then enter the award name for all owned books that won the award for novel. Updating the library for awards is an annual Maintenance task. Rule: a book can have zero to many awards. The metadata elements in Awards are:

    • b, Bram Stoker award for best novel. Awarded 1987-present. March-June.
    • c, Campbell Award for best novel. Awarded 1973-present. June.
    • h, Hugo Award for best novel. Awarded 1953-present, except 1954 and 1957. September.
    • if, International Fantasy Award for best novel. Awarded 1951-1955 and 1957 only. Defunct.
    • n, Nebula Award for best novel. Awarded 1966-present. May.
    • pz, Pulitzer Prize for fiction (previously for novel). Awarded 1918-present. April.
    • wf, World Fantasy Award for best novel. Awarded 1975-present. October.

  • FormatQ (#formatq). Custom column, comma separated text, editable. For Format Quality rating and any other format-quality or format-status tags. Based on assessment of format after adding book or fixing format problems, the format quality rating enables format quality control, provides a basis for comparison between duplicates, identifies books with no formats or annoying formats that can be replaced in the future, and ultimately applies only to one master digital book format. The rating scale is based on types of empty book (with no format to read) for the bottom ratings, then moving up the scale through worst to least amount of negative emotion caused by formatting issues when reading a format. Rule: a book can have one to many format-related metadata elements, and one of them must be format quality rating. The metadata elements are:

    • f0, Empty book (no format), exclusion placeholder, do not acquire format ever. Useful to prevent wasting time reinvestigating in future. I set Column Coloring to color text tan in all columns of each exclusion placeholder to visually distinguish the record in booklist. I set Create Catalog to indicate exclusion with "x". For each new exclusion placeholder, enter "f0" in FormatQ and make the Title unique by constructing it as follows: "_xLevel; reason_excluded; level_element", where Level is T, S, or A for Title, Series, or Author; reason_excluded is a very brief reason; and level_element is the title, series, or author. For title exclusion enter Authors and Series as usual. For series exclusion enter Authors and Series as usual but 0 for Series Index. For Author exclusion enter only the excluded author in Authors, (0) in Series, 0 in Series Index. Save excluded books to disk into an Exclusions folder then remove them from the library. Example Title for each level of exclusion:

      • Title-Level. _xT; bad writing; the book's title
      • Series-Level. _xS; poor & boring; name of series
      • Author-Level. _xA; ro (romance); name of first author in Authors

    • f1, Empty book (no format), wishlist placeholder, acquire format in future. When adding new wishlist placeholders, enter Authors, Series, and Title as usual then process the empty books as if they were real books for everything except format-related tasks. When a format is finally acquired, drop the format into this wishlist record then process it like any other newly added book, revising previously entered metadata where relevant. I set Column Coloring to make text magenta in all columns of each wishlist placeholder to visually distinguish the record in booklist. I set Create Catalog to indicate it with "w".
    • f2, Not readable, or is readable with strong annoyance, anger, rage, or despair due to formatting problems impossible to fix, or not worth the time to fix. Immediately remove format, and enter "f1" indicating wishlist placeholder. Never assign "f2".
    • f3, Readable with mild annoyance. Some mildly annoying formatting problems remain because they are not possible to fix quickly at current knowledge/skill level. Or the format is a pre-release advance or review copy. The rating "f3" means the format is acceptable for now but replace it with better in future. I set Column Coloring to make text mediumvioletred in all columns of an "f3" book record to visually distinguish the record in booklist.
    • f4, Readable with no annoyance. Assign after assessment and any necessary fixing of all annoying problems. Most formats in my library are "f4".
    • f5, Readable with awe. Excellence of formatting is a moving target, due to gradually increasing knowledge of formatting and skill fixing formatting problems. Problems announce themselves loudly, as compared to quiet and unobtrusive excellence of formatting without perceived problems, possibly not consciously noticed at all. So for practicality I assign "f4" instead of "f5".
    • pb, Own the paper book (hardback, trade paperback, or mass-market paperback). This metadata element is independent of format quality rating or other format-related elements. It is also the only element in FormatQ that does not pertain to an ebook format.
    • rc, Review copy, advance reader copy. I want to replace this format with the proofed final-release version when available, so I also assign a format quality rating of f3 even if there are no annoying formatting problems.

  • Formats2 (#formats2). Custom column; built from other columns, behaves like tags; template {:'approximate_formats()'}, sort/search column by text, unchecked show in tag browser, unchecked show as HTML in book details. Automatically filled, not editable. For seeing formats at a glance in the booklist. Named Formats2 to avoid potential confusion with the calibre field Formats. During library startup, the template {:'approximate_formats()'} populates the field from format metadata in the database, which is fast but would be inaccurate if any formats were added or deleted directly in the library folder in an operation unknown to calibre (which I avoid doing). In comparison, the template {formats} looks at the format files in each book folder of the library in the filesystem, which is potentially more accurate but requires more time during library startup.
  • Genres (#genres). Custom column, comma separated text, editable. For high level genre classifications. I do not download tags. Instead I determine genres by context with other books in a series; by assessing key words and phrases in Title, Series, and Comments in the book record, by sometimes assessing genre tag or synopsis on the title page of a bibliographic site, or if all else fails, by reading a few pages of the book. Rule: a book must have at least one, and no more than two genres; preferably just one. This genre classification is solely for my own convenience. The metadata elements are:

    • adv, Adventure. Action in exotic places, usually not contemporary.
    • fn, Fantasy. All subgenres except urban fantasy.
    • hor, Horror. Focused on instigating horror or terror.
    • main, Mainstream. Not marketed as a specific genre.
    • mys, Mystery. A sleuth investigates to solve a crime; other crime fiction is thriller.
    • nf, Nonfiction. Based on real people, events, and facts.
    • par, Paranormal romance. Supernatural characters in historical or contemporary setting, and emphasis on romantic relationship.
    • ro, Romance. All subgenres of romance except paranormal.
    • sf, Science fiction. All subgenres.
    • thr, Thriller. Suspense, contemporary action, excluding horror.
    • uf, Urban fantasy. Usually contemporary, urban, with supernatural characters, and emphasis on other issues than romantic relationship.
    • ws, Western. Action in the old wild west of North America, usually sometime between 1850 and 1890.

  • Keywords (#keywords). Custom column, comma separated text, editable. Specifically for descriptive keywords and lower level genres. In the past I called these tags "secondary genres" but now "keywords" seems a more appropriate term. Many of them can apply to fiction or nonfiction, and many can apply to more than one fiction genre or nonfiction subject. Determine them by examining words in Title, Series name, and Comments. Revise them as necessary after reading the book. Rule: zero to many keywords per book. My metadata elements in Keywords are:

    • alt, Alternate history.
    • ngl, Angel.
    • asn, Assassin, hitperson.
    • bio, Biography, biographical.
    • cls, Classic.
    • cmp, Computer.
    • cri, Crime.
    • dmn, Demon.
    • dgn, Dragon.
    • ero, Erotic.
    • fae, Fae, faery, fairy, elf, sidhe.
    • gds, Gods.
    • gst, Ghost.
    • hx, History, historical.
    • hmr, Humor.
    • juv, Juvenile, child, marketed for anyone younger than young adult.
    • law, Law, legal, lawyer.
    • mgc, Magic, witch, wizard, sorcerer, magic user of any persuasion.
    • med, Medicine, medical.
    • mil, Military, soldiers, legion, army, navy, marine corps, air force, space corps, military conflict, war.
    • mys, Mystery, sleuth investigates to solve a crime.
    • orc, Orc, goblin, troll.
    • phz, Physics.
    • poe, Poetry.
    • psi, Psychic, psychic powers, healing touch, telepathy, telekinesis.
    • psy, Psychology, psychological.
    • ref, Reference.
    • sf, Science fiction.
    • shf, Shifter, werewolf, any were-animal, polymorph, skin-walker.
    • spy, Spy.
    • stm, Steampunk, clockpunk.
    • tns, Tennis.
    • vmp, Vampire, vampyre, wampyr, dhampir, human-shaped or close to human-shaped being who consumes human blood or human life energy for nourishment.
    • wri, Writing.
    • yng, Young adult.
    • zmb, Zombie.

  • Pages (#pages). Custom column, integer, formatted for comma {0:,}, editable. For the number of pages in the book, computed by the plugin Count Pages, using algorithm Paragraphs (APNX accurate). I do not use the plugin's word count or readability capabilities. I do not care whether the page count on device is the same or not. Page count is a good indicator of book length, while file-size in the Size (MB) column is not. For example, page count helps determine whether a book is a novel or a separately published short with the same author and title; the short may have a high resolution cover making the short's file-size greater than the novel file-size. Rule: each book record with a format must have a page count.

  • Read (#read). Custom column, comma separated text, editable. For content rating entered after reading each book, and for "to read" tag to identify books to read or reread soon. The presence of a content rating means I read the book. No content rating means either I have not read the book, or I read it too long ago to enter a meaningful content rating. For books read more than once, the content rating represents the last time the book was read. Rule: a book can have zero to one "to read" element and zero to one content rating, with maximum two metadata elements. The metadata elements for Read are:

    • TR, To Read. (This is included in the saved search "To Load".)
    • r1, Content rating: bad. The writing is bad.
    • r2, Content rating: poor. The writing is poor.
    • r3, Content rating: okay. The writing is okay. Some boring parts.
    • r4, Content rating: good. The writing is good. Interesting. Not boring. Read it all.
    • r5, Content rating: excellent. The writing is excellent. Fascinating, gripping. Read it all.

  • Source (#source). Custom column, text, editable. For the name of download site, retailer, or other source of the format. In workflow Source is entered immediately after Add Book, rather than later when processing the other format-level fields. Rule: each book with a format must have one source. Some (of many) metadata elements in Source are:

    • Amazon, Amazon US site.
    • author site name, Various author sites.
    • Baen.CD, Baen CD at the site Fifth Imperium. 23 free CDs each with multiple books.
    • Baen.Eb, Baen Ebooks at baenebooks.com.
    • Baen.FL, Baen Free Library at the Baen Ebooks site.
    • B&N, Barnes and Noble.
    • BookView, Book View Cafe.
    • manufacturer name, Various manufacturer sites, for user manuals.
    • MR, MobileRead. Free public domain books with good formatting.
    • O'Reilly, O'Reilly Media. Technical books.
    • ProjGute, Project Gutenberg US site. Free public domain books.

  • Types (#types). Custom column, comma separated text, editable. For publication types. All publication types except novel are rare in my library, because I prefer reading whole text-based novels, rarely add or keep the other types, and make exceptions for only a few authors or publications. Rule: zero to many per book. The metadata elements in Types are:

    • an, Anthology. An anthology contains shorts by various authors. For anthologies with credited editor(s), in Authors I do not include the authors of content but use the names of the editor(s), without appending (ed) to the names. For anthologies without any credited editor I list the authors of internal content, for example for teaser anthologies of paranormal romance novellas each with different authors. For Series Index, treat anthologies as if they were novels, or index them as 301, 302, 303, depending on whether the timeframes of the contained shorts easily fit into novel indexing or not.
    • col, Collection. A collection is a group of shorts all by one author, or all by one set of co-authors. For Series Index, treat collections as if they were novels, or index them as 201, 202, 203, depending on whether the timeframes of the contained shorts easily fit into novel indexing or not.
    • cmc, Comic book. Image-based comic book or graphic novel. For Series Index, number graphic novels or comics like novels, or use issue numbers.
    • mgz, Magazine. Periodical, usually a combination of image-based and text-based content, often a combination of fiction and nonfiction. For Publisher use magazine name. For Series Index use issue numbers. For Published use issue date.
    • nvl, Novel. Fictional events and characters in a long prose-based narrative, with length over 40,000 words (the standard to qualify as a novel for the Nebula award novel category). Absence of a Types tag indicates the type is novel, or the type is tagged elsewhere (news or catalog in Tags, nonfiction in Genres). Most of my books are novels, so for convenience I enter the novel tag only when I have duplicate authors/title books each with a different type, for example a short story and a novel, in which case I also append to Title the relevant type tag in parentheses, (shr) or (nvl), to make each Authors/Title unique at book-level. For Series Index, use whole numbers, beginning with 1 for the first novel in the main series; for prequels use 0 with decimals: 0.01, 0.02, 0.03.
    • nvz, Novelization. A novel based on a previously existing video game or movie.
    • om, Omnibus. A compilation containing at least one novel, usually two or three, often accompanied by one or more shorts. The contents may be by one or several different authors, in the latter case usually in the same series. I replace an omnibus with the separate novels, or when those are not available I split the omnibus into separate novels with the plugin ePubSplit or an editor, preserving the copyright information for each novel. Separating the novels simplifies series numbering and author updating. For Series Index of omnibuses, use numbers higher than the novel numbers such as 123, 456, 780; or alternatively 101, 102, 103 for a series with more than 9 omnibus-contained novels.
    • shr, Short. I use the term "short" for short story, novella, novelette, chapterbook, or essay. This is purely for my convenience, because I prefer novels and rarely want any of the "short" types. In Series Index, use decimal number indicating placement between relevant novels in the series, for example 3.51 and 3.52 for two shorts belonging between novels 3 and 4.
    • spl, Split. This book was split from a larger book with multiple components such as an omnibus. In Source, enter the source of the original book that was split.
    • srl, Serial. A serial is published in parts usually in consecutive issues of a magazine. The combination of parts is usually novel-length. I do not collect them unless it was not separately published as a novel and I can gather all the parts, in which case I use plugin ePubMerge to merge the parts into one book. Enter the magazine name in Publisher, and the publication year and month in Published. In Series Index, indicate separate parts of a serial with decimals after the relevant novel number, for example three separate serial parts of novel 6 would be 6.01, 6.02, 6.03.
    • Catalog, Catalog in book format, listing 2 or more books in a library. Calibre automatically assigns this metadata element elsewhere, so I do not enter it in Types. See Tags.
    • News, News download. Calibre automatically assigns this metadata element elsewhere, so I do not enter it in Types. See Tags.
    • nf, Nonfiction. For convenience I treat nonfiction as a high-level genre, so I do not enter it in Types. See Genres. Note: International Speculative Fiction Database treats nonfiction as a publication type. (See Help:Screen:NewPub, ISFDB wiki, definitions for publication-level metadata.)

  • Updated (#updated). Custom column, integer, editable. Year and month in yyMM format of when the first or only author in Authors was last processed as Current Author for newly available books, or a code indicating author is inactive but books are up to date. Entered in bulk for all books by the author. This field is useful for determining when to do the next update for an author. I usually update a few authors in the library every day in an ongoing long-term update cycle. This column is frequently searched and edited, so the data type is integer rather than date for easier editing, and integer rather than text to allow use of < and > in searches. Rule: each book must have one 4 digit integer in Updated.

    • yyMM, Year and month the first or only author in Authors was last updated.
    • 9999, The first or only author in Authors has been inactive (not published anything) in the past 10 years, and I already have all the previously published books by this author that I want. This helps avoid wasting time periodically checking an inactive and up to date author for newly available books. If a book by this author is published later, probably I will notice it while reviewing lists of forthcoming or newly released books on bibliographic sites.


Links
Spoiler:

Overview
  • Purpose. The Links section provides links to ebook-related information and tools.
  • Topics. Calibre (various subtopics), Bibliographic Sites, Catalog on Server, Conversion, CSS, Devices, DRM, eBooks, Filename Limitations, Filename Renamers, Format Editing, Formats (subtopics per format), HTML/XHTML, Icons, Images, Metadata Terms, Regular Expressions, Searching, Strategy, Templates, Workflow, Version History.
  • Links Key. The links are color-coded.

    • Green. Information on MobileRead Forums.
    • Purple. Information on other websites.
    • Blue. Software tools that may be useful with ebooks. Tools recommended by experienced people at MobileRead Forums are tagged recommended. Any I have not tried myself are tagged noted.

Calibre: Download
Calibre: Getting Started
Calibre: MobileRead Forums
Calibre: User Manual
Bibliographic Sites
Catalog on Server
Conversion
CSS (Cascading Style Sheets)
Devices (Reading Devices)
DRM (Digital Rights Management)
eBooks
Filename Limitations
Filename Renamers
Format Editing
Formats: EPUB (Electronic Publication)
Formats: Kindle* (AZW3, MOBI, . . .)
Formats: PDF (Portable Document Format)
Formats: Word Processor — TXT, RTF, DOC, DOCX, ODT (Text, Rich Text Format, Document, Document XML-based, Open Document Text)
HTML/XHTML (HyperText Markup Language / eXtensible HyperText Markup Language)
Icons
Images
Metadata Terms
Regular Expressions (regexp, regex):
Searching
Strategy
Templates
Workflow
Version History

Last edited by unboggling; 06-23-2014 at 04:24 PM. Reason: Ongoing revision.
unboggling is offline   Reply With Quote
Old 07-19-2012, 04:15 PM   #2
unboggling
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
Thread Rules

Please post elsewhere any questions about how calibre works, or any requests for help with a problem, by starting a new thread in the main calibre forum or relevant calibre subforum.

Please post in this thread feedback about the first post, recommended changes to it, or comments about different ways to manage ebooks with calibre.

Last edited by unboggling; 02-15-2014 at 09:55 AM. Reason: clarify.
unboggling is offline   Reply With Quote
Old 07-19-2012, 04:45 PM   #3
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 6,521
Karma: 42666590
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
@Unboggling; any thought into putting this into an alternate format, maybe (oh horrors) an ePub?
PeterT is offline   Reply With Quote
Old 07-19-2012, 05:00 PM   #4
unboggling
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
Quote:
Originally Posted by PeterT View Post
@Unboggling; any thought into putting this into an alternate format, maybe (oh horrors) an ePub?
I'd thought about it. Generating an ePub would be relatively easy to do. But it seems like it's more accessible online here than in an ePub that might be added to someone's library then forgotten. At least if I added it to my library, I'd probably forget it was there. It's easier for me to deal with frequent revisions by just pasting new version over older version at Post #1 of this thread.

Last edited by unboggling; 07-20-2012 at 01:05 PM. Reason: Minor clarification
unboggling is offline   Reply With Quote
Old 07-21-2012, 09:12 PM   #5
unboggling
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
Quote:
Originally Posted by PeterT View Post
...maybe (oh horrors) an ePub?
Giving a book version further thought:

Translating the BB code formatting of the version at Post #1 into HTML is relatively painless, per suggestion of theducks to copy the HTML of the post page and paste it from there into an editor, but then spoilers need to be stripped and formatting redone to be more suitable for a book. I need to revise this document frequently with changes to fix unwitting errors and keep it congruent with my actual settings, customizations, and workflow. A book version doesn't seem practical.

Last edited by unboggling; 07-22-2012 at 12:25 AM.
unboggling is offline   Reply With Quote
Old 08-13-2012, 05:35 PM   #6
unboggling
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
Direction of "How I" Project

The content of Post #1 now seems relatively stable and problem-free, though it will still need revisions for changes in calibre or in my workflow. Please let me know if you find any problems I missed, so I can fix them. Meanwhile I've been thinking about the future direction of this project.

1. Searching the forums works best for small posts, when each post is limited to one specific topic. This large document has lots of topics, so it pops up often when using forum-search for keywords, which may annoy people searching for solutions to specific problems. That brings up the post but doesn't show where keywords are in the document unless the spoilers are expanded to allow looking for the highlighted keywords. Alternatively, if all spoilers are expanded first, a browser search of the document for keywords works well. Without expanding spoilers first, a browser search for keywords when viewing the page source also works but shows any keyword hits from HTML tags. All the search methods are awkward for one reason or another.

2. If I transform this project into a website:
  • + People could link to it from a post or sticky here, avoiding the search problem here.
  • + Site would have internal search feature.
  • + HTML and CSS would be more flexible than BBcode for formatting.
  • + After each revision of content, it could be transformed into an ebook with significantly less work than from the BBcode-formatted document here.
  • - Implementation would require learning more HTML and CSS along with the initial work of setting up the site. Good for me in the long run for eventual use with ebooks, but still time-consuming.
3. I'm not sure how useful this project is to other people. The content is different from and broader than the usual calibre forum content of "how do I do this new-feature-to-me?" or "how do I solve this small specific problem?" Last year, as a new user I looked for content like this, overall maps of how other people used calibre, including strategy and workflow along with examples. Couldn't find anything as broad as I wanted, just disorganized bits and pieces throughout these forums. So started the KISS thread, and that eventually grew into this document. I still wonder if others find it useful, if there's a common need for this kind of content.

What do you think? Feedback will help me decide what to do. Please comment, either by private message or in this thread.

Last edited by unboggling; 08-17-2012 at 12:36 PM. Reason: details about searching
unboggling is offline   Reply With Quote
Old 08-30-2012, 03:34 PM   #7
brainvision
Enthusiast
brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.brainvision ought to be getting tired of karma fortunes by now.
 
brainvision's Avatar
 
Posts: 47
Karma: 603120
Join Date: Aug 2012
Location: Monte Los Angeles (Califoggia)
Device: Android Smartphone
I already gave you karma points - as you know - 'cause I found this thread very very useful: I think that it contains a lot of information that everybody needs! Once starting using Calibre more in depth, than maybe will not follow al of your suggestion, but sure they are excellent example to show us what Calibre can do and how we should think at this program.

I prefer to read the post little by little, a paragraph at once,to be sure to really understand it and with the intention to try by myself your approach..
I don't know what you should do to give more emphasis to this post, but I'm pretty sure you should do it! And, more, if you can and if you want, you should try to make it always better and complete!!
Examples are always the best way to learn when we are talking about software and practical things to do..

thank you very very much, once again!
I really appreciate this "little" but precious work!
see you!!
brainvision is offline   Reply With Quote
Old 08-30-2012, 04:23 PM   #8
unboggling
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
You're welcome. Glad to hear you find it useful. Thanks for the glowing endorsement. Made my day.

Edit:
And thank you again for the karma last week. I like karma. And I like feedback even more. I plan to continue revising and adding to it, at least for awhile. I'm still thinking about the "give more emphasis to this post" thing, as well as future direction of the project.

Last edited by unboggling; 08-31-2012 at 02:27 AM. Reason: added comments.
unboggling is offline   Reply With Quote
Old 09-01-2012, 11:07 PM   #9
travger
Evangelist
travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.
 
travger's Avatar
 
Posts: 469
Karma: 270594
Join Date: Aug 2010
Device: palm tx, Windows XP, Windows7
OK, I have some thoughts:

'Assessing Formats' - Bulk Convert to Preferred Format.
Seems a bit pointless to me. Pros - makes nice uniform library.
Cons - 1. Book that I fix today and feel happy about coloring headings purple may seem rather ghastly few years hence.
2. Any conversion may lose or add something (I'm presently reading a mobi book where there's no blank lines for the scene breakes, I doubt that they will appear after conversion. This particular book has such a twisted css that I must uglify very nice ePub in order to replace some code especially for the conversion to mobi. In this case I'll keep both nice formats).
3. When I don't read the book immediately, leaving original format be saves lot of time - just quick quality check is all.
4. By the time I get around to reading it, several things may have happened: I may have found better quality/preferred format; I may have gotten new reading device; my html skills will definitely increase.
5. Maybe next book in the series has different but nicer formatting, then I can change first one more easily, having an example.

Fixing Formats - Convert to Editable Format
IMHO converting anything html to word and back is not good practice. Editing for example ePub in Sigil is much more under control, it won't accept mistakes, and you can see right away the change (and maybe undo it).

Editing Metadata - Title
I prefer shorter title. Probably it's subconcious answer to Windows path length limit. A friend gave me his zipped library, and many books there were unextractable because the title was too long.

Maintenance -
Maybe warning about not to mess around directly in the Calibre folders should be capitalized?
travger is offline   Reply With Quote
Old 09-02-2012, 05:02 AM   #10
unboggling
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
Quote:
Originally Posted by travger View Post
OK, I have some thoughts:

'Assessing Formats' - Bulk Convert to Preferred Format.
Seems a bit pointless to me. Pros - makes nice uniform library.
Cons - 1. Book that I fix today and feel happy about coloring headings purple may seem rather ghastly few years hence.
2. Any conversion may lose or add something (I'm presently reading a mobi book where there's no blank lines for the scene breakes, I doubt that they will appear after conversion. This particular book has such a twisted css that I must uglify very nice ePub in order to replace some code especially for the conversion to mobi. In this case I'll keep both nice formats).
3. When I don't read the book immediately, leaving original format be saves lot of time - just quick quality check is all.
4. By the time I get around to reading it, several things may have happened: I may have found better quality/preferred format; I may have gotten new reading device; my html skills will definitely increase.
5. Maybe next book in the series has different but nicer formatting, then I can change first one more easily, having an example.

Fixing Formats - Convert to Editable Format
IMHO converting anything html to word and back is not good practice. Editing for example ePub in Sigil is much more under control, it won't accept mistakes, and you can see right away the change (and maybe undo it).

Editing Metadata - Title
I prefer shorter title. Probably it's subconcious answer to Windows path length limit. A friend gave me his zipped library, and many books there were unextractable because the title was too long.

Maintenance -
Maybe warning about not to mess around directly in the Calibre folders should be capitalized?
@travger. Thank you for the feedback. It helps me. A lot.

Assessing Formats - Bulk Convert.
I will clarify what options I use in conversion. The issues of bulk convert and deleting original format both need careful reconsideration. I'll at least add some discussion of pros/cons for each of those issues, both of which I've seen discussed here more than a few times. I vaguely recall we also talked about some of this last year, and revisiting these issues now from the perspective of a year's additional experience will be good.

Fixing Formats - Convert to Editable Format
I agree, that needs some clarification and rethinking too. I've been learning HTML, CSS, and Sigil gradually and am probably to the point where I should ditch fixing in Word, OpenOffice, or LibreOffice in favor of preferably fixing in Sigil in my real-life workflow. In the workflow document I was thinking of changing the simple fix example from Word back to OpenOffice, but first I need time to figure out and test how to anchor graphics to paragraphs in OpenOffice. I wanted to use a simple example. And I wish I'd started learning ins and outs of Sigil sooner. I'll rewrite this section into at least 2 fix subsections, one for Sigil fix, one for OO fix.

Editing Metadata - Title & filename lengths
Yeah, Windows filenames does throw a wrinkle into long titles. And I suppose also long series names, when those are included in saves/sends. It's been years since I used Windows and I simply forgot…. Now you pointed it out and I read this Naming Files, Paths, and Namespaces, (Windows Dev Center) I realize I should change how I deal with title addendums and long series names such as with hierarchical series subgroups, since we all might as well try to be interoperable as possible with filenames, for principle's sake if nothing else. For Title, off the top of my head: only append very short items such as " - 3 Ed" or " - ARC" to make title more unique for editions. And put any omnibus elements into their own empty book wishlist record, rather than appended to title. And put long title addendums in custom column Note. For Series, I need to think about it some more, and test some templates for truncating and abbreviating - - and I can probably use or adapt one of chaley's examples from calibre user manual section on Template Language. This issue is probably why he used that particular example in the first place.

Maintenance - Calibre Library Folders
Done. Made entire paragraph bold - at least until I think of some other way to emphasize it significantly but more subtly (if that's even possible….)

It may take a few days to a week before any of these other changes (and subsequent dependent changes) go into effect in the document. My general rule is that any changes in the document get tested and worked into my real-life workflow first. Though to my regret and later rework, on some occasions I screwed up by including something I hadn't tested sufficiently. Such as un-KISS series numbering for Complex Series (in Edit Metadata section), that I'd tested with insufficient number of cases when overly tired. Which I just somewhat fixed, and probably still needs more work along with the other series conventions toward being simpler. (The more specific I get, the more I get in trouble….)

travger, thanks again for the feedback.

@all, you're welcome to chime in with discussion, comment, feedback.



Edit: So what is a good compromise on title or series name length, taking into account the whole path with filename needs to fit into what, 260 char? - Would you say 40 char for title? Seems too short…. Oops, let's see. Would also need path including title + series + series_index. So series name saved or sent by template needs to be a very short abbreviation derived from longer string in Series column.

Last edited by unboggling; 09-02-2012 at 07:28 AM. Reason: minor corrections & last question
unboggling is offline   Reply With Quote
Old 09-02-2012, 10:38 AM   #11
travger
Evangelist
travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.travger ought to be getting tired of karma fortunes by now.
 
travger's Avatar
 
Posts: 469
Karma: 270594
Join Date: Aug 2010
Device: palm tx, Windows XP, Windows7
Always happy to be in the position to help; you're welcome.

You better be prepared that in Windows path starts with C:\ ...

I think it may be rather personal for what purpose anybody uses save/send and accordingly the needs for filename may be different. For example if screen is small and reader app is showing filenames, there's not much point having every filename like: Gordon Dahlquist - Miss Temple, Doctor Svenson, and ... disappearing at the edge of the screen.
That said, now I must confess that I use send/save very rarely - for me it's drag'n drop from Calibre folder to SD card.

I keep Book panel open, so Comments field is perfect for me to enter long title for the book. For example - Simple Smith: How Even a Village Fool Can Have a Fulfilling and Happy Life I enter into Comments with h5 or h6 tags, and Simple Smith into book title field.

For series I have 'Cycle' field as subseries, but if I needed to save to disk, I'd use just main series name and numeration. Cycle, Awards, Artist etc. info goes into file itself.

OMG, you speak so easily about templates and still hesitate about html ! (In Sigil you just have to click 'Insert picture'. Though I suspect you need something more complicated.)

One more point - when adding retail ebooks (or even most ePubs/mobis) it's rather safe to read metadata from the file itself. Pro side being also, that name of the downloaded file will not matter. I must add some info into custom columns manually anyway, so I'm eyeballing all the other fields too.
travger is offline   Reply With Quote
Old 09-02-2012, 11:19 AM   #12
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,254
Karma: 5495470
Join Date: Aug 2009
Location: The (original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by travger View Post
You better be prepared that in Windows path starts with C:\ ...
Only absolutely true if your system has a Single HD (with a single Windows accessible) partition)
Modern Systems can have multiple R/W drives (you should not have to worry about path length with Read Only media ), some removable. some networked. On a few occasions, I have had more than half of the available drive letters in use (data collection and processing. Hoo-Ray for the old Netware 4.x. No problems.)
theducks is online now   Reply With Quote
Old 09-03-2012, 04:24 PM   #13
unboggling
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
Revision notes:

Approach:
Added brief intro to clarify what it is and isn't.

Adding Books - Choose Method for Adding Books:
Added sentence that reading from file contents usually works well for retail-quality books from good sources.

Editing Metadata, Title or Series - and path/filename length issue:
Shortened the appends to Title. Mentioned an alternative for people wanting shorter titles might be putting that metadata into comments or custom column(s). Added separate paragraph (after Authors) about string length in Series and Title. Decided not to go into specifics about calibre automatically truncating when necessary to deal with various file system path/filename limitations, or the tech specs of those limitations (that would have been off-topic; and I don't know all the details). But added a few links under Filename Limitations in Links section. And added links for template info and examples in separate paragraph Set Templates in Reading Books section, and added links in Sending Books in Preferences section.

Other issues:
Still thinking about the issues with bulk convert, delete original format, and preferred fix with Sigil. Any changes there may be awhile since I want to get comfortable with any changes in real-life workflow before revising the document.

But I'm tending toward never deleting original format. In most cases that would give every book 2 formats, Preferred and Original. In the case of Preferred to Preferred, it would be Preferred and Original_Preferred, requiring Tweak save_original_format=true. For users learning how to fix books it's probably more important to keep the original formats, than for experienced book fixers who know what they're doing. I'm still in the learning how category, particularly with Sigil.

And I finally learned how to anchor a selected graphic to a paragraph in OpenOffice, preceding add of ODT into calibre. With menu command Format/Anchor/To Paragraph. All I had to do was look.

I'm also tending toward always bulk converting after adding books, for all books. For me using EPUB as preferred format, that makes sense for the assessment in calibre viewer. With further assessment of original format if necessary in other reader app for that format.

Last edited by unboggling; 09-04-2012 at 05:45 AM. Reason: revision of revision notes :)
unboggling is offline   Reply With Quote
Old 09-04-2012, 12:33 AM   #14
mraxehandle
Junior Member
mraxehandle began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Feb 2007
Device: Palm T3
I still don't understand why so many people use forums as a kind of wiki (I think of XDA in particular.) It seems strange to me, given how many great wiki-type tools are out there.Does MobileRead or Calibre have a wiki space that can be used?
mraxehandle is offline   Reply With Quote
Old 09-04-2012, 02:10 AM   #15
unboggling
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
Quote:
Originally Posted by mraxehandle View Post
I still don't understand why so many people use forums as a kind of wiki (I think of XDA in particular.) It seems strange to me, given how many great wiki-type tools are out there.Does MobileRead or Calibre have a wiki space that can be used?
Welcome to MobileRead Forums.

MobileRead Wiki Main Page, links to ebooks, libraries, stores, and information on devices, readers, tools, and formats.

There is a calibre article on MobileRead wiki. AFAIK nothing prevents there being other calibre wiki pages with different focus or content.

These are some reasons this document is here:
  1. It is still under development, being added to and refined, with rapid revisions.
  2. Feedback spurs that along.
  3. I want to keep control of the edits. AFAIK, any registered user can edit a wiki article. I might be wrong because I never contributed to a wiki before, and don't know specifics of that process.
  4. The focus, as currently written and structured, is on how "I" manage ebooks with calibre, my needs, strategy, workflow, configs, and settings and how they interrelate. Not necessarily on how other people manage ebooks with calibre, which is usually different for each person. (General discussion of what other users do is mostly for context or contrast. Any discussion of methods to choose from is usually for context or contrast, and/or workflow contingencies.)
  5. I suspect most of the people reading this in a calibre forum are more calibre-focused than a broader group reading it as a wiki article.

If I'm wrong about anything in that reasoning, anybody, please let me know.

Last edited by unboggling; 09-04-2012 at 06:36 AM. Reason: clarify
unboggling is offline   Reply With Quote
Reply

Tags
calibre workflow, ebook management strategy, ebook management workflow

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
PRS-T1 Manage Collections in Calibre (Help!) FatCat0 Sony Reader 19 08-11-2012 12:00 PM
How to find & manage ebooks from various apps? rapidlanguage Library Management 3 01-06-2012 08:13 AM
Development Using Calibre to manage eDGe library mrspaceman enTourage Archive 76 05-12-2011 12:38 PM
Neo How to manage ebooks? ivanm BeBook 11 08-19-2010 11:01 AM
How do you manage your read queue with ebooks? DuncanWatson General Discussions 7 05-14-2010 01:30 PM


All times are GMT -4. The time now is 02:35 PM.


MobileRead.com is a privately owned, operated and funded community.