KISS for New calibre Users

[unboggling]

My basic approach to strategy and workflow is fundamentally back to what I started with in KISS Post #1 plus some refinements, tweaks, and expansion of detail and discussion throughout. (Edit: re strategy back to start, exception: going slow is a major change.)

I'll be cleaning up any minor formatting and link glitches and whatever other small problems I find for a few hours if necessary.

I considered all feedback, input, and discussion in this thread from start to present, and incorporated what I could into my own work habits.

The rule I'm using for inclusion of content changes is: if I haven't tested it myself and worked with it for awhile, it doesn't go in except where I note it as a recommendation from people at MobileRead. If you think I missed including something important, please let me know - I went through 5 pages of scrawled poor handwriting to add/revise content based on responses in this thread, and I probably missed things.

As usual, any feedback, input, and discussion from whoever cares to participate will be helpful to me for the next revision.

[kacir]

Originally Posted by kiwidude

So what I ended up doing was almost starting again by having multiple libraries - the initial one that is the "partially clean mess", and my "clean one" that books get migrated into that is intended to contain only books that are cleaned up. Note that if I was installing Calibre for the first time today I would not have two libraries for this purpose - it is only the fact that I had already invested so many months of effort into partially cleaning the initial one that I have kept it.

I took a bit different approach.
I have created a new user column, called "Clear".
I go through my "Kitchen sink and everything else mess imported from God-Knows-Where" Calibre Library and I am going Author by author, and correcting metadata, gathering covers, descriptions, series info (Series are extremely important metadata), merging duplicates, examining what version to ditch during merge (*). Then I mark the book[s] "Clear".
I use your plugins heavily in the process.

(*) I never delete formats I find "not useful" at the moment. Experience taught me that sooner or later you are going to get a device that supports different bunch of formats, and disk space is *SO* cheap nowadays (well, not if you collect HD movies, but 500kB *.lit file it costs very little to store, Just-In-Case)

[unboggling]

@kacir, so you use just one library for everything?, do a pass (requiring months maybe?) to work by author from A to Z, meanwhile adding whatever you download, then start over with A-Z iteratively forever? Or do you eventually move the "clear" ones to a Clear Library?

I keep all downloaded formats, even when they are many or bad, in a Raw Books folder on a 2TB hard drive and try to use only the best of each lot inside calibre. For example, after trashing everything and restarting I downloaded all the free Baen CDs, put them in expanded folders into Raw Books/Pending/Baen CDs/CDnameFolder and started adding only the best EPUB or in a few cases a MOBI if no EPUB, one CD at a time which has usually mostly the same author. So far I've done 4 CDs. Using just one library. I also keep the original compressed downloads of important things in a different folder that also has subfolders for things like application installers, including old versions of those. Because disk space is cheap and just seems to keep getting cheaper.

[kiwidude]

@kacir - trying to clean up the "kitchen sink" an author at a time in the same library was also my initial approach. I can't recall all the things I experimented in terms of a tag or a custom column to make a book as processed but I also used a user category to mark an author as processed amongst them.

However the issue I found was twofold. Primarily that the Calibre performance was pretty dire - taking many seconds or sometimes minutes to perform simple operations. A lot of work has been done on Calibre performance since that time to improve things, but it made for intolerably slow performance at the time. Also back then I had not yet written a number of the plugins like Find Duplicates to assist.

The second reason was more the sense of lack of progress, every time I opened the library I was faced with the bulk of it being unprocessed, and the books I had done kind of got lost in the morass. Putting just the cleaned ones into a new library gave me a sense of achievement when I opened that up to know that all the books contained within were ready for reading. And also importantly the books I most wanted to be reading, rather than including many others that I had just collected for the sake of collecting. Of course you can use a search restriction to similar effect, but with the performance issues I had I chose a clean start instead.

As for the formats thing, I don't disagree that space is cheap but the more files you have you will pay a penalty somewhere. Whether it is at the time you do a search in Calibre, the time you do a search via the file system or doing a backup. As for the formats themselves, well it depends on whether you own a legacy reader (I don't) or not. Formats like lit, lrf, etc are dead as a dodo. Txt, rtf, doc are a poor man's ePub. HTML is a hop skip to an ePub. PRC, AZW - just use MOBI (which in itself is easily converted to/from ePub). These are my opinions based on the devices I own. Of course there will likely be some other format come out one day to obsolete ePub - but I am hedging my bets that given its dominance a product like Calibre will ensure you can convert from ePub to it. And quite apart from any of that - keeping say an ePub and a LIT format of a book will always leave a question in your mind as to which version is the "better" one if they were not converted from the same source. Which you can only resolve by inspecting. Better imho to do that job once, make your decision (converting the LIT to overwrite the ePub if it wins) and move on.

But we all have our own ways, that is part of the fun

[unboggling]

Originally Posted by kiwidude

As for the formats thing … keeping say an ePub and a LIT format of a book will always leave a question in your mind as to which version is the "better" one if they were not converted from the same source. Which you can only resolve by inspecting. Better imho to do that job once, make your decision (converting the LIT to overwrite the ePub if it wins) and move on.

@kiwidude, you're right. I've experienced that question in my mind. So I need to decide whether the chance of keeping any of the others is worth it. The Baen CD HTML v's seem tailored for full size browser use (without having inspected them much and without knowing much HTML yet), so they along with the other dead-dodo formats probably aren't useful except for other reasons such as learning some HTML or messing with a LIT to learn how it's put together. Which aren't essential reasons to keep them, vs the confusion they generate later.

I think I want to start deleting everything except the format I put in calibre and sometimes cleanup. But that raises the question of when to delete those other formats. If I don't clean it right away, I might want to refer to one of those other formats in the future when I do clean the one in calibre or even use it instead if I find a big problem while cleaning. So then there's the mechanical question of how do I automatically remember things need to be deleted if I don't delete them right away at Add time? It would have to be at Clean time. But Clean time doesn't apply to all books.

btw, a couple hours ago I put reference and links to your workflow and metadata bigposts into 0.60 after posting, using an ((Edit date)) marker. I'd integrated some of your ideas already and am still thinking about other stuff in those posts that's not integrated into my workflow yet.

[kiwidude]

@Unboggling - re timing of deletion. If I understand you properly your question concerns the situation where you dont clean the book straight away. A situation I have for much of my library given the earlier previous attempt to put it in my old library. What I do is a combination of things. Like Kacir mentioned, I have a #done custom column on each book to indicate I have cleaned/processed it. I also have an "Unprocessed" user category that I will add an author to if at the time I add a book I don't clean it right away (or they have so many books I just want to get them in and clean when I have more time). Strictly speaking the latter is kind of redundant since I could do a search for #done:false, but I like to see a count and list of the authors that are outstanding in my library, and a simple click on an author in that list in the tag browser gives me the books just for that author.

In terms of timing, as soon as I have processed (cleaned) a book any other formats in Calibre for it are deleted. As to when I delete from my source folders, I certainly do that when I have processed all books for that author, but sometimes for a specific book as I go. It depends on how I am processing the author, how many books they have and how many different versions of the book I have. Sometimes I end up doing a windows explorer search for a specific book rather than just the author, so I can more easily see and find my best format. In that situation it is just as easy to delete the files in the search results window when I have processed the book. That can help when you have an author who has short stories as well (which don't tend to be listed on Fantastic Fiction) and you are trying to figure out what books for that author you have not yet put in Calibre.

Note that a valid alternative that some people do instead of deleting is to move the books that they added into Calibre from their raw source folders into some other kind of "processed" folder. Just so they have them available should they do something bad with their conversions or find they picked the wrong format in thinking it was their best one. However I would certainly *not* attempt to store those in a Calibre library in any way - just use a filesystem search to find a book from there in the extremely rare circumstance you need to do so.

Re the 0.6 version - I'll be honest, I haven't read it in detail . I saw spoilers within spoilers, and lots of text, and as I am not the target user for it anyway I moved on. I will say what I saw kind of read more as a "diary" than a quick summary?

If I was a user looking to get a quick and simple recommended setup, I don't really care about reading a whole paragraph or two of the "why" too much as a new user (if at all for many users I would suggest) - I just want to know the "what" to do and "when" to do it. The problem with explaining the why is that a why can be very verbose - after all, look at my own post here taking three paragraphs to explain some opinions on deletion.

I don't have the answers for you, as I saw someone else post writing documentation is freaking hard, and you have to do a fine balance between just enough to get them going without turning it into a whole user manual in its own right. Like I said, I am not your target user though, and others may find what you have done to be exactly what they were after so don't let me discourage you too much!

[travger]

Originally Posted by unboggling

I think I want to start deleting everything except the format I put in calibre and sometimes cleanup. But that raises the question of when to delete those other formats. If I don't clean it right away, I might want to refer to one of those other formats in the future when I do clean the one in calibre or even use it instead if I find a big problem while cleaning. So then there's the mechanical question of how do I automatically remember things need to be deleted if I don't delete them right away at Add time? It would have to be at Clean time. But Clean time doesn't apply to all books.

My solution (not very good, but working) to that is to keep everything in 'books' folder. When I decide to read something, I inspect available formats, import best one to Mobi Creator (or Calibre) and convert to mobi. While reading, I mark the typos and irritations, then check other files for the same mistakes. Often there are several formats with exact same text, but happens also that dubious places are more natural in another file.
Very rarely I may try new conversion from another format, usually it's just massage the html > create prc to check if last improvement took.

Also, during reading I can find tendencies like for example </i><i> instead of space. If such things look like continuing throughout the whole book, I search and replace them, make new prc and continue reading that.

After reading I keep original pdf/doc/html I converted from, opf+edited html+picture(s) and final prc. If converted from epub, I may keep one nice-looking pdf also. Everything else goes to the trash.

Sometimes during that process I add it to the Calibre. Original format usually before I start to read, with metadata search (have to be sure about series and reading order), final prc when I finish reading it.

I can't really imagine more naked format than html. If I can get my claws into that, I can tweak anything in the book (or learn to tweak). And I believe it will convert less painlessly into some future format (more likely, noname readers will accept several formats). Now I try to download only epub or mobi.

I do not have good experience with lrf/lit conversion. And if I can't get acceptable html now, then I doubt there will be better results converting into some future super-duper format. Of course could have been lousy lit/lrf.

@kiwidude - I must disagree, I remember being very interested in 'why'. Especially new user needs to know why somebody does things, in my opinion it makes a lot easier to decide 'no, not for me, don't need to know how'.

[kiwidude]

Originally Posted by travger

@kiwidude - I must disagree, I remember being very interested in 'why'. Especially new user needs to know why somebody does things, in my opinion it makes a lot easier to decide 'no, not for me, don't need to know how'.

My understanding of what unboggling was trying to put together was a simple guide for new users to help point them in the right direction in a few areas. My first impression when I saw the latest post and I expanded some of the spoilers was "crikey, that is a lot of text" and that makes it look very complicated. Which imho is in contrast to the concept of trying to make this a simple recommendation.

If I take the case of my parents as an example Calibre user, they really don't care about the intimate details of a "why" for most things - they would just want to know a "what" that is built from the experiences of other users to save them the hours of figuring it out for themselves.

However I completely understand (as I am the same) that there are users like yourself who want to know the "why" for a recommendation so they can choose to use it or not, particularly when there are alternatives. I think it is valid to have the information available, I just suggest it is pretty verbose burying the "what" in amongst the "why".

Perhaps an alternative approach would be to have the "Why" within the spoilers - so people can see at a glance the recommendations, and if they are curious they can drill into the detail.

[unboggling]

I'm a lousy editor. And worse self-editor. It was even worse before I cut a lot of conversational background/why chat out, reducing size considerably. But I did intentionally leave in some why's. Without them, it would be here's how to set up calibre for new users step by step and follow my instructions and don't ask why each thing is important. Which I wanted to avoid - I'm really not trying to provide a manual. Just a map of what I'm doing with some examples. Examples in a structure of some reasoning help me learn better than examples not in some kind of bigger structure that makes sense.

Re nested spoilers, not to sound defensive, but I thought a flat 3 - 6 pages of text per high-level topic would have been potentially more intimidating to new user than another few buttons to press. (I'm aware of the python "philosophy" flat is better than nested.) I like pressing buttons, it's fun.

But I probably should go find a professional editor (edit: a human one I mean) - there are some on MR - and ask for a slash job. Any volunteers?

[kiwidude]

@unboggling - as I said above - it is just my opinion and what you have done may be exactly what others are wanting. For myself I wanted to just skim through the recommendations to see "where you were at" in certain areas, but instead found myself with lots of reading to do. Which I personally decided I didn't want to do, so I stopped. I totally agree that if the post was three pages long I would have skipped it as well. Can't please everyone can you

[JSWolf]

Originally Posted by theducks

As someone who comes from a Analog Electronics background,
Every conversion Loses Detail and Inserts Artifacts.

Once you have your ePub, don't convert it to ePub. That will change it and in most cases, it will change it for the worse, not better. Calibre doesn't know how to just add a cover, just change metadata or change just the margins. It know how to deconstruct what you have and rebuild it to make your changes. Basically, if you have your ePub, open it up, make the fixes and then rebuild it or use Sigil.

[unboggling]

Originally Posted by travger

I can't really imagine more naked format than html. If I can get my claws into that, I can tweak anything in the book (or learn to tweak). And I believe it will convert less painlessly into some future format (more likely, noname readers will accept several formats). Now I try to download only epub or mobi.

@travger, re the other things in your post, I'm really trying to use calibre as the light or matter-that-I-can-touch-and-rely-on in my eBook universe with everything not calibre more like dark matter/dark energy that I have to use sometimes but would like to avoid if possible. (With exception allowed for a good HTML editor or two.) I guess it's a pretty big difference between starting places or approaches we have. And if yours works for you I'm glad. Though some of your methods don't seem to fit in my universe of trying to use calibre as the core of everything particularly since I don't know enough HTML yet.

About HTML and learning HtML being the primary keys to the lock of most clean-up and conversion problems, I fully agree. Maybe after I learn enough HTML, my core library will hold onlyHTML formats. Edit them, generate formats for devices on the fly, read, immediately delete any other format. It'd be even nicer if I could edit that HTML directly on reading device as I read. Raw HTML is probably the truest "lossless" format currently available in the state of current technology, in my admittedly new and limited understanding of the eBook world.

@kiwidude, I like the idea of "why" inside spoilers. That's a great idea, and I'll work on implementing something like that for next version. Have a feeling that won't be easy for me, leaving the do-this's and do-that's naked with the why's hiding under the bed button. I'm the kind of person who likes to know why's first too.

[unboggling]

Originally Posted by JSWolf

Once you have your ePub, don't convert it to ePub. That will change it and in most cases, it will change it for the worse, not better. Calibre doesn't know how to just add a cover, just change metadata or change just the margins. It know how to deconstruct what you have and rebuild it to make your changes. Basically, if you have your ePub, open it up, make the fixes and then rebuild it or use Sigil.

OK. But I gotta learn HTML first

[cjallan]

Originally Posted by unboggling

@cjallan,
When I answered your first post here I was tired and shouldn't have responded because what I said wasn't relevant to your concern re first paragraphs following chapter breaks flush, and the others indented getting messed up by conversion. Sorry. (I should've gone to sleep, without answering, till the next day.)

Now that I think about it more clearly, it sounds like a structure detection problem related to headers (h1, h2, h3 etc) at chapter breaks. There may be a direct way using calibre to handle it, in the structure detection tab of conversion dialog box, to include treatment of the following para's indent as a special case as part of the chapter detection during Conversion. I've never messed with that personally and aren't a conversion expert, and I don't work with MOBIs much either. So I don't know.

If you're okay with the way it looks now, there may be no need, but if you want to pursue it, you could ask more about the issue in the calibre conversion subforum.

Yes, I do want to pursue it, but due to a heavy work load, must come back a bit later.

Thanks so much for your help... that got me over a big problem, but I want to learn all I can about Calibre conversions.

[theducks]

Originally Posted by unboggling

OK. But I gotta learn HTML first

Only the stuff that goes in between the Body tags