KISS for New calibre Users

[kiwidude]

Cool . You have a very specific workflow of a staging library and presumably not ever touching the book when it is in your final library. However for many people books in their "final library" will never be in a "don't ever touch it again" state in terms of metadata. It may be they find they have catalogued a book incorrectly, have a typo in the title, change their author naming scheme, get fresh metadata based on a new metadata source plugin being available, decide to add a better cover etc etc. In all these cases the metadata is not going to be updated in their book format stored within the library without further actions. So if they are like you in being "bothered" by it then they are going to have to keep adding extra steps to their workflow to be able to sleep at night

You have agreed to de-emphasize it which is the main point - let Calibre do what it is good at in terms of managing the metadata itself in its database/opf file, and only "worry" about metadata in the book at the point you export from Calibre to a device or software that depends upon it, where plugboards can help get this the way you want it.

[DoctorOhh]

Originally Posted by unboggling

Because, if those books are Copied to Library rather than Save/Adding, that metadata doesn't get set when going from the fix-it library to the storage library.

What metadata isn't being transferred? I move books all of the time and as best I can tell all of the metadata is copied with the book.

Originally Posted by unboggling

So in a sense they are in an incomplete state. And will stay that way forever, because once in storage library, any convert, save, or send will only update those internal fields on a copy, not the original sitting there in incomplete state forever. I understand the way people use calibre, that doesn't matter. But it bothers me.

In my opinion a proper library backup is much better then any storage library.

Have you investigated using one library and library restrictions to treat different books as if they were in different libraries.

[travger]

Originally Posted by dwanthny

Have you investigated using one library and library restrictions to treat different books as if they were in different libraries.

Please can you clarify? What restrictions exactly? Examples? How can I treat for example all short stories as if in another library?

[unboggling]

Originally Posted by kiwidude

Cool . You have a very specific workflow of a staging library and presumably not ever touching the book when it is in your final library. However for many people books in their "final library" will never be in a "don't ever touch it again" state in terms of metadata. It may be they find they have catalogued a book incorrectly, have a typo in the title, change their author naming scheme, get fresh metadata based on a new metadata source plugin being available, decide to add a better cover etc etc. In all these cases the metadata is not going to be updated in their book format stored within the library without further actions. So if they are like you in being "bothered" by it then they are going to have to keep adding extra steps to their workflow to be able to sleep at night

You have agreed to de-emphasize it which is the main point - let Calibre do what it is good at in terms of managing the metadata itself in its database/opf file, and only "worry" about metadata in the book at the point you export from Calibre to a device or software that depends upon it, where plugboards can help get this the way you want it.

Well, I don't use the separate Process Library and Store Library anymore, just doing it all in one. And I'm aware that after initial add and metadata edit, also any later changed metadata doesn't update in internal fields on the copies until further action. So personally, just for me, not as a recommendation but just my peace of mind, what I've decided to do is once a month or so: Save everything out, along with covers and OPFs. Create a new empty primary library. Add them all into new library. Check to make sure nothing is missing. Rename old primary library to "Trash", rename new primary library to the old primary library name (I use "Core"), delete the old library Trash.

Now everything is set and I'm happy. It'll all get reset once a month. And the renamings shouldn't mess with automatic backups if they're temporarily off during the save/add process, or if it's done completely between scheduled backup times. All it means for auto-backup is that it will backup the entire library from scratch the first time after each metadata-reset operation.

[unboggling]

I know very little regex too, and the one dwanthny uses looks good even to me. Thanks. I tested it a little and it's now added to my Prefs, Add Books, add by filename menu and selected for when needed. I'll test it on various things some more then if all that is ok will change to that in the next revision. The nuances of the differences between it and kacir's and calibre's menu regexs are over my head. I know there must be some differences in what they do, but don't know enough regex to decipher it all:

mine (and on calibre's default menu at install?):
(?P<author>[^_-]+) -?\s*(?P<series>[^_0-9-]*)(?P<series_index>[0-9]*)\s*-\s*(?P<title>[^_].+) ?

dwanthny's:
^((?P<author>([^\-_0-9]+)(?=\s*-\s*)(?!\s*-\s*[0-9.]+)|\b))(\s*-\s*)?((?P<series>[^0-9\-]+)(\s*-\s*)?(?P<series_index>[0-9.]+)\s*-\s*)?(?P<title>[^\-_0-9]+)

kacir's:
^(?P<author>((?!\s-\s).)+)\s-\s(?:(?:\[\s*)?(?P<series>.+)\s(?P<series_index>[\d\.]+)(?:\s*\])?\s-\s)?(?P<title>[^(]+)(?:\(.*\))?
OR
(?P<author>[^-]+)(( - | *-- *)[[(]?(?P<series>[^-]+)[[( ]+(?P<series_index>[0-9.]+)?[])]?)?( - | *-- *)(?P<title>.+)

[unboggling]

Originally Posted by dwanthny

What metadata isn't being transferred? I move books all of the time and as best I can tell all of the metadata is copied with the book.

I was talking about internal fields in the format itself.

Originally Posted by dwanthny

In my opinion a proper library backup is much better then any storage library.

Have you investigated using one library and library restrictions to treat different books as if they were in different libraries.

Yup, that's what I do now, as it's much easier. Which meant I had to revamp my whole tag system, which is now mostly completed and allows easy informal searching by just typing one or several tags in the search box, using no formal field prefixes and just simple operators AND or OR. I'm amazed at how easy and precise that is once the tags aren't common words.

Edit: And also don't bother with saved searches or formal restrictions unless it's a very complicated operation, which is rare.

[travger]

For me main library seems to be quite much 'don't touch it again'. I even drag the files directly from Calibre innards to SD-card.

I usually prefer not to change cover in the file, sometimes metadata search finds nicer cover and I use that for Calibre but not in the file. I can't very well imagine what there is to change.
Well, if it is so new that series was not considered... but I can change html before I read it again (if). If not, who cares as long as info in Calibre is correct.

[unboggling]

Originally Posted by travger

Please can you clarify? What restrictions exactly? Examples? How can I treat for example all short stories as if in another library?

What I do is use a symbol as prefix to a tag abbreviation in a combination simple enough to remember and unique enough not to produce matches with other words or abbreviations in the metadata landscape.

So if I type the tag !sf (that's exclamation point Science Fiction) alone in the search box, the book list restricts itself to just books tagged with !sf. It doesn't need
this prefix
fieldname:"=
or this suffix
"

A search restricts the booklist from everything to a subset that the search matches if Preferences/Search is set like so:
Unchecked, Search as you type.
Unchecked, Highlight search results instead of restricting the book list to the results.

So just by typing
!sf
I get all my science fiction.

Or just by typing
!sf or !fn
I get all my science fiction and fantasy.

Or just by typing
!sf and !fn
I get all books that have both tags.

But this informal method is predicated on having relatively unique tags. For example, the tag ref doesn't work because I'll get all records that contain any word starting with ref, rather than just my reference books. But what does work is one of these:
_ref, !ref, %ref, @ref, $ref

and with Authors, it's easy to just type in the complete author name if one part of it isn't unique, such as mercedes lackey rather than lackey or mercedes alone.

Edit: Oops, should've used short stories example:
%shr

Edit2: Oh, in case I didn't make it clear, it doesn't matter what column those tags are in, or what field that doesn't get displayed in columns such as Comments long text field seen in Edit Metadata. If the tag exists anywhere and the search matches a record, the record is listed. Edit3: excepting fields not included in Preferences/Search, if checked yes to restricting to certain listed fields.

[unboggling]

Originally Posted by travger

For me main library seems to be quite much 'don't touch it again'. I even drag the files directly from Calibre innards to SD-card.

I usually prefer not to change cover in the file, sometimes metadata search finds nicer cover and I use that for Calibre but not in the file. I can't very well imagine what there is to change.
Well, if it is so new that series was not considered... but I can change html before I read it again (if). If not, who cares as long as info in Calibre is correct.

I usually leave the cover alone in the file too, unless it has a hugely annoying problem.

I'll probably change a lot of things when I'm more knowledgable and comfortable with HTML. Lately I'm getting so much more comfortable with BBcodes for these revisions that I can read the raw text including code tags and not get too distracted by the tags. So I wonder if I'll just read books in raw HTML someday?

[travger]

Thanks, it shows me I should use more tags. Now I'm using genre field for 'Fantasy' and 'Science Fiction' and that's almost all the genres I have. To change one tag (Novel->%novl) is easy.

All this depends on having tags.

[unboggling]

Originally Posted by travger

Thanks, it shows me I should use more tags. Now I'm using genre field for 'Fantasy' and 'Science Fiction' and that's almost all the genres I have. To change one tag (Novel->%novl) is easy.

All this depends on having tags.

Yup. Tags are now important to me. When I first started with calibre, using tags was new to me. I didn't start really trying various methods with search box until after maybe 4 months using calibre, relying before that on the tag browser exclusively. I didn't start relying on this informal search method until months after that, which is mostly this month. It takes awhile to get comfortable with a new way of doing something, and I tended to just keep doing whatever I was doing without trying anything different, until some blunder or crisis made me think about other ways of doing things and motivated me a little.

[kiwidude]

I would suggest "use with caution" the "special character tag" approach. A couple of reasons. First you might find that in a future version of search your tag characters conflict with some special search syntax, so then you have to change them and "unlearn" what you may have adapted yourself too.

But far more importantly also are you 100% sure that you are the only person who will ever browse your library in future? No intention of a family member or friend sitting down to browse it or shared with them over the web? Because you are going to find it incredibly painful to explain to them that "[!@$sf" is your foo genre or whatever.

My point being - again I don't think it should be a "recommended" approach in your guide. For sure you can mention it as something that you like, but in my opinion it is not something that new users should be steered towards.

Now if you think I'm totally anti using symbols as tags, you would be wrong. As I have mentioned previously I did use to use a number of them, but right now I am down to just two:
a minus sign - for a wishlist item that I put on my empty books.
an asterisk * for a book that I need a better quality format for.

Sorry unboggling, I just can't grok the prefixing of % signs, brackets and all other sorts of things to genres or whatever. I think you are out on a limb with that one - I understand that you like it for yourself but I would hate to have to try to explain it to someone else

[unboggling]

Originally Posted by kiwidude

I would suggest "use with caution" the "special character tag" approach. A couple of reasons. First you might find that in a future version of search your tag characters conflict with some special search syntax, so then you have to change them and "unlearn" what you may have adapted yourself too.

But far more importantly also are you 100% sure that you are the only person who will ever browse your library in future? No intention of a family member or friend sitting down to browse it or shared with them over the web? Because you are going to find it incredibly painful to explain to them that "[!@$sf" is your foo genre or whatever....

Sorry unboggling, I just can't grok the prefixing of % signs, brackets and all other sorts of things to genres or whatever. I think you are out on a limb with that one - I understand that you like it for yourself but I would hate to have to try to explain it to someone else

No problem, kiwidude.

I deleted most of the abbreviation and tag discussion, it didn't show up in the revision posted last Friday except maybe a passing mention. Next revision I'll make sure that's explicitly cleared out of all nooks and crannies except maybe to mention how I do it with an explicit "not recommending this" notation.

Most people have family and friends they share with. I happen to live alone and don't share books with friends presently but if either ever change, it'd be easy to rename abbreviations back to human-speakable/comprehend-at-glance-able. Re symbol prefixes, yeah, that's happened to me already, learning something doesn't work anymore in my new method or a new calibre revision. Those are easy to change too. Hopefully by the time most symbols can't be used, I'll be more proficient at formal searches and restrictions on the fly, or move to using some other method.

Nothing I said in the KISS/Workflow posts after the first few was meant in general as advice or recommendation, except one thing: try to keep it simple. Everything else is supposed to be just examples of how one user does things. I don't know how to make that more explicit than it already is stated in the first section, Orient. When in the workflow I say, "Do this" or "Do that" I'm writing out the steps I use, or want to use, for myself. Those become examples, which any other user can try out, use for awhile, or ignore as they please. I thought of changing the title to something like: "Examples of One User's Workflow, Not Necessarily Recommended" but that just didn't work for me as a title.

[kiwidude]

I haven't read your latest revision in detail (or any in detail for that matter!) - I just randomly opened a couple of spoilers and sometimes spot discussion points. I remember seeing a post from you that you were revisiting your tag approach and I thought "cool, maybe he is going to be more conventional in his recommendation". Then I saw your post above and it seems you just traded some symbols in your tags. So I wondered if it was going to feature in your guidelines like it once had when I first saw it. Your explanation above is good on your intentions with that.

Much kudos to you for your continued efforts on the iterations.

[travger]

Example:

I want to find novellas.
One of the books has "author of award-winning novella" in comments (though I try to avoid this, but...).
Now this book is also included in the search result.

Having something unusual instead of 'novella' avoids situations like this. Nvella or *nlla is my own creation.
Also, I am 98% sure of never having to explain. Nobody sits behind my computer without me hanging around with watchful eyes. If I share over the net - recipient is free to delete/change all the tags.