[Old Thread] Why doesn’t calibre let me store books in my own directory structure?

[darknessangel]

Before anything, I purposedly copy-pasted that very question from the Calibre FAQ. Perhaps this is a lone cry from a crazy guy in the middle of the forest but I think it's an important one.

"Why doesn’t calibre let me store books in my own directory structure?"

While in the FAQ is stated that the Calibre "search/tagging based interface is superior to folders" I dare to disagree. In reality I think it's a huge misconception!

This argument may hold some water for normal books (sci-fi, etc.) which you just read once. But it fails horribly, when (like me) you have a library of 20k books on diverse subjects, mostly scientific, you NEED a folder structure like in a normal library! Because :

#1 you rarely look for an author, you look for Science ->Chemistry->inorganic chemistry-> intermetallic phases, not for "J. H. Westbrook" who I don't know and I don't really don't want to know.

#2 When you use reference books... you read them more than once!

#3 It becomes really helpful to have a structure when most of your metadata is wrong or missing.

#4 For sake of OCD people and for the eventuality that you want to give part of your library to someone... it is good to keep libraries separated from each other! I don't want to literally hunt for every author for 30 books on "catalysis" if I want to give it to a colleague!

#5 I don't think it's difficult or impossible to implement. Calibre does the same as iTunes but iTunes can keep the folder structure.

Ok, perhaps I'm overreacting a little, perhaps I'm just absolutely horrified at the prospect of my Library being thrown in a heap after I carefully organized it... but I think this option would make a lot of people very, very happy.

Unless there is some consensus that Flash is evil and kills babies if it runs in computers... oops... sorry, wrong prejudice! But that's how I regard how this option is being handled!

GJMS

[itimpi]

The problem is that the Calibre internal directory structure cannot be changed without a major rewrite. Assumptions about the current structure permeate much of the code.

That means that if you cannot live with the Calibre internal structure then you will not be able to use Calibre for library management.

You can still use the Calibre conversion facilities if you use the command line utilities as there you provide the paths for both the input and output files.

[kovidgoyal]

1) What's preventing you from making such a query using tags?

A hierarchical query for a > b > c is logically and functionally equivalent to a AND b AND c

2) So what?

3) If you have a structure but no metadata in files, then your metadata is in the structure. One of my goals in writing calibre is to discourage the use of filesystems to store metadata. I have explained why several times before and I'm not going to again.

4) That's what Save to Disk is for

5) No it's not. It was a design choice. Just look for old threads in this forum if ou care to read my rationale for this design choice. And calibre does a lot more than iTunes.

[ficbot]

This question has been answered in this forum numerous times. Just make your own folders. Use Calibre to edit and convert, then 'save to disk' and put it into your own folder. Personally, I was taken aback at first by Calibre organizing all my stuff, but I really like it now and found the tags very easy to use. I also like that you can apply more than one tag to the same book, so I can search (for example) for books which are a mystery AND unread AND purchased from X store.

[Starson17]

Originally Posted by ficbot

I was taken aback at first by Calibre organizing all my stuff, but I really like it now and found the tags very easy to use.

I just looked at my remembered searches:

Yesterday I wanted all my books that were not one of my wife's books, that I had not read and that did not have a text format.

Next I wanted all my naval fiction books that I hadn't read. I saved that search (thanks Chaley!)

I decided I wanted to see the naval fiction search books from above that didn't have text format or rtf, so I called up the first search and restricted it further with the second and rtf format.

Then I wanted all my books except the news books that didn't have a cover.

My wife did a search to see her books. There were too many, so she restricted by removing those she'd read and sorted by date to find the ones that she'd added in the last few months.

Then I wanted my computer technical books on html.

On two of these searches I wanted a subset of books found saved out with a name that had the series number first and saved inside a directory named by the series. One of my readers cuts off the end of the name, so I sometimes want the first part of the name to be the series number. I modified the save template.

You can't do all these searches if you just rely on the directory and filename. The flexibility of tags, the power of the searching system and the ability to rename on the fly are what I like about calibre.

Having said that, I do understand why you want a different directory and naming structure. You are used to going directly to the directory to get your books, and you'd like to keep on doing that. The problem is that different people want different structures and different filenames. Calibre isn't designed for direct access to your books via its book storage directories and it can't be changed to accommodate all the different ways people want their books organized even if the developer's wanted to make that change. The current design is too deeply embedded in the code.

[Worldwalker]

Originally Posted by darknessangel

This argument may hold some water for normal books (sci-fi, etc.) which you just read once.

Nice start, taking a dig at the SF readers and calling our books "sci-fi" -- and I seem to get the implication you think we're doing really well to read at all.

But it fails horribly, when (like me) you have a library of 20k books on diverse subjects, mostly scientific, you NEED a folder structure like in a normal library!

Calibre is not the point of failure in this case.

#1 you rarely look for an author, you look for Science ->Chemistry->inorganic chemistry-> intermetallic phases, not for "J. H. Westbrook" who I don't know and I don't really don't want to know.

tag:"=intermetallic phases"

Or, at its most detailed, in case you have books that are chemistry but not science, inorganic chemistry but not chemistry, etc., etc., you could go all the way to :

tag:"=science" and tag:"=chemistry" and tag:"=inorganic chemistry" and tag:"=intermetallic phases"

You don't have to worry about where they're stored, or the file names, or anything else -- just every book that has something about the subject. That also means that if you have a book that covers several different topics in inorganic chemistry, you could give it multiple tags, rather than limiting it to a single classification like your directory tree would.

#2 When you use reference books... you read them more than once!

Amazingly, not all calibre users are empty-headed, frivolous "sci-fi" readers. We store a lot of things in calibre. Technical references of all types are a big one. And we read a lot of books more than once.

#3 It becomes really helpful to have a structure when most of your metadata is wrong or missing.

Aside from the obvious need to fix your metadata, the simple solution is to tag your books as you import them. When you import books on subject X, you bulk edit them to add "subject X" to their tags. Or "subject X", "sub-topic Y", and "detail Z" if appropriate.

#4 For sake of OCD people and for the eventuality that you want to give part of your library to someone... it is good to keep libraries separated from each other! I don't want to literally hunt for every author for 30 books on "catalysis" if I want to give it to a colleague!

Select tag "catalysis", export to disc, hand it to him.

#5 I don't think it's difficult or impossible to implement. Calibre does the same as iTunes but iTunes can keep the folder structure.

What is this new trend of insulting the developers as some sort of incentive to make them rebuild their software the way you want it?

You -- and several other people -- have said things that imply a challenge to Kovid and others to prove they're not reaaaaally stupid, and the acceptable proof being a willingness to throw away the whole design philosophy of calibre and rewrite it into something that challenger wants. I mean, seriously: if someone starts off by implying you're incompetent, stupid, ignorant, or just unconcerned about what you're doing, are you going to bend over backwards to help them out? Kovid is too nice a guy to defend himself from insinuations like this, but I'm neither a calibre developer nor a nice guy. So knock it off.

Ok, perhaps I'm overreacting a little, perhaps I'm just absolutely horrified at the prospect of my Library being thrown in a heap after I carefully organized it... but I think this option would make a lot of people very, very happy.

Your library is right where you left it. If you use your current folder names as tags when you import it into calibre, it'll be organized in calibre just as it is in the filesystem. AND you'll have the option of organizing and searching it in many different ways as well.

I do happen to be one of those contemptable "sci-fi" readers. And I have a lot of SF books in calibre (sorry, I can't keep calling it "sci-fi", even in quotes, because there's only so long I can endure insulting myself). I can find them by author or title, sure. But I can also bring up all my alternate-history SF. All my military SF. All my SF that is both military and alternate-history. Or that isn't. Etc., etc. If I want to find all my alternate-history SF short stories that are not military-themed and weren't written by H. Beam Piper, I can do that. Try that with your directory structure.

I would really love to see you trying to use a database. I can just imagine your post on the Microsoft forums:

Access sucks because I can't read my data with Notepad! How can I find what I want if I can't do a text search for it in Notepad? You need to get rid of all that binary data, and all that relational stuff, and just make Access a front end onto a big flat file! Other people can use flat files, I'm sure if you tried really hard you could too!

For that matter, I can see you complaining to Henry Ford about your Model T:

This car is no good. There's nowhere on it to hitch my horse to. And it's so heavy that even if I hooked my horse to it somehow, he wouldn't be able to pull it more than a couple of miles. You need to make it a lot lighter (maybe get rid of that heavy iron thing in the front) and put shafts on it so I can hitch up my horse.

Calibre will do exactly what you want, and do it better, if you use the tools that it has for doing that.

[DoctorOhh]

Originally Posted by darknessangel

While in the FAQ is stated that the Calibre "search/tagging based interface is superior to folders" I dare to disagree. In reality I think it's a huge misconception!

The misconceptions and limits you infer in your post are your limits and misconceptions.

Originally Posted by darknessangel

This argument may hold some water for normal books (sci-fi, etc.) which you just read once. But it fails horribly,

It only fails horribly because in your ignorance you are limiting your thinking.

Originally Posted by darknessangel

when (like me) you have a library of 20k books on diverse subjects, mostly scientific, you NEED a folder structure like in a normal library!

The "normal library" is in fact moving toward digital storage and ebook access. The "normal library" has decided that tags verses folders are the way to format their collections for easiest access. The "normal library" is progressing forward, expanding and changing with the medium they are working in.

Originally Posted by darknessangel

#1 you rarely look for an author, you look for Science ->Chemistry->inorganic chemistry-> intermetallic phases, not for "J. H. Westbrook" who I don't know and I don't really don't want to know.

Your point is what? That every book in this folder would have to be tagged with Chemistry, inorganic chemistry, and intermetallic phases to mimic your folder structure? OK, so do it. Oh the horrors the book would also be tagged by Author and Title just in case you read a paper by "J. H. Westbrook" and wanted to know what else he wrote it would be easy to find.

Originally Posted by darknessangel

#2 When you use reference books... you read them more than once!

Wasting space now aren't you, I know I am responding to this point.

Originally Posted by darknessangel

#3 It becomes really helpful to have a structure when most of your metadata is wrong or missing.

Your right fixing your metadata could be a bitch. I am sure though that there are smart people here who could help with guidance on the import so that each book at least would be tagged by each folder level from the start.

Originally Posted by darknessangel

#4 For sake of OCD people and for the eventuality that you want to give part of your library to someone... it is good to keep libraries separated from each other! I don't want to literally hunt for every author for 30 books on "catalysis" if I want to give it to a colleague!

Again you are limiting yourself by your ignorance of the flexibility of the tagging system. It would take almost no time to select/search on "catalysis" select the titles, then click save to disk and be done with it. It would be hard for me to imagine that every book covering "catalysis" would be in the same folder, seems to me your example is an argument for a tag structure.

Originally Posted by darknessangel

#5 I don't think it's difficult or impossible to implement. Calibre does the same as iTunes but iTunes can keep the folder structure.

This would also entail accepting the limits and insecurity of iTunes. Having books indexed all over the disk allows users to be their own worst enemy.

Originally Posted by darknessangel

Ok, perhaps I'm overreacting a little, perhaps I'm just absolutely horrified at the prospect of my Library being thrown in a heap after I carefully organized it.

First if you do this right your library is untouched. When you start Calibre do not point it to your library. Point it to an empty folder. This way all books imported into Calibre go into the new folder and your library/folder structure is untouched. The best of both worlds.

[Knipfty]

darknessangel,

What happens when a reference book can be categorized in more than one folder? Do you place it in two different folders?

With tags, you can you are not restricted in how it is categorized. Here are all the tags that I have on the current book I am reading:

Fiction, General, XReading, Fiction - Science Fiction, Science Fiction & Fantasy, Space Opera, Science Fiction, Adventure, Science Fiction - Adventure, Science Fiction - Space Opera, Star Trek fiction, Science Fiction - Star Trek, Science Fiction And Fantasy




I understand that you have a way of organizing your reference material. You are in essence storing your metadata in your directory structure. With Calibre, you get to store your metadata in the tags. And with tags, you get a lot more flexibility.

[iain_benson]

Originally Posted by darknessangel

This argument may hold some water for normal books (sci-fi, etc.) which you just read once.

Um, you may just read fiction books just once, but some of us like to re-read. A lot of books have much greater depth when you read them several times, picking up on foreshadowing that isn't obvious until you know what's going to happen for example, and often you need to refresh your memory of what happened, before reading the next book in a series when it comes out two years later. If I didn't intend re-reading I'd just get books from the library rather than buying them.

Originally Posted by Worldwalker

Nice start, taking a dig at the SF readers and calling our books "sci-fi" -- and I seem to get the implication you think we're doing really well to read at all.

Okay slightly off topic, I know, but I'm a little puzzled

What's the difference between referring to science fiction novels as "sci-fi" rather than SF?

Admittedly, I do get slightly annoyed by bookshops that have one section labelled "Science Fiction" containing all the fantasy stuff as well. Using "Science Fiction and Fantasy" is better, as splitting the two can become awkward when you get books that cross the divide such as the Pern novels, but whether you use SF, sci-fi or Science Fiction seems a little irrelevant.

Originally Posted by Worldwalker

Calibre will do exactly what you want, and do it better, if you use the tools that it has for doing that.

Agreed! I used to organise everything in my own folder structures (music and books etc), but then I discovered iTunes, and realised that it did so much more for me that the folder structures became completely irrelevant, so I just let it do it's own thing.
Similarly Calibre, with tags, series, author, publisher etc. fields, the ability to have many different methods of virtual organisiation is so much more powerful than just a directory structure. You could, for example, keep your own system of naming, and add in tags for the dewey decimal system.

[darknessangel]

Ok, thank you very much. While I also think the first post was indeed not in a nice tone I am fairly satisfied that it got solid answers including some developers!

First of all, I am not saying that Calibre is bad or evil, it's just that I'm not completely sold on the idea. And I will try to use it by importing my library part by part and edit the metadata slowly like that, my problem will be that right now my library's about 60 GB and and I don't have 60 GB free in another HDD.

And as someone mentioned, I also have some problematic books that are in the "in-between" several themes and would profit from the tag system.

Originally Posted by dwanthny

First if you do this right your library is untouched. When you start Calibre do not point it to your library. Point it to an empty folder. This way all books imported into Calibre go into the new folder and your library/folder structure is untouched. The best of both worlds.

I hope I can get it working like that.

And... someone caring to explain me the sublte nuances between Sci-fi, science fiction and SF? Does one sparkle more? o_O

I also read "fiction", but it's not those books I have a problem with, that's why I was mentioning it.

GJMS

[theducks]

Originally Posted by darknessangel

First of all, I am not saying that Calibre is bad or evil, it's just that I'm not completely sold on the idea. And I will try to use it by importing my library part by part and edit the metadata slowly like that, my problem will be that right now my library's about 60 GB and and I don't have 60 GB free in another HDD.

As you load your Library into Calibre, you could pause and burn the original to DVD if you don't want to get an external drive for backup/archival purposes.

And... someone caring to explain me the sublte nuances between Sci-fi, science fiction and SF? Does one sparkle more? o_O

GJMS

SciFi was equated with the cheesey flic's of the 50's. Poor plot, poor Science (never violate known science without explaining it away).
Voyage to the bottom of the Sea (TV) "Captain! The Reactor has gone Critical" Well, Doh! If it does not go critical, it does not generate any steam. Critical is the point of a sustained, controlled reaction.

As to shelf labels: I prefer including Fantasy on the label unless, somehow the store manges to separate "Hard SF" from the cross-over SF and that from (Epic/Sword n Sorcery) Fantasy. Modern Fantasy seems to cover so much territory that finding a neat slot to classify it is impossible.

[Worldwalker]

Theducks pretty much covered it: "sci-fi" is a pejorative term for science fiction, and many longtime readers/fans find it highly insulting. That's particularly true in the context it was used by the OP, as an example of "trash" books that some non-serious reader, some lesser being, might read once and throw away.

Trying to sort out SF from fantasy can be a strain. Okay, on one end of the spectrum you have hard SF, and on the other end you have pure fantasy, but it's the cases in the middle that keep the aspirin manufacturers in business. The rule of thumb "if it has spaceships, it's SF; if it has dragons, it's fantasy" breaks down and cries when it meets genetically-engineered dragons or magically-powered spaceships. There is fantasy that is as "hard" in the sense of developing a world consistent with a given set of conditions as the hardest SF, and SF where you just need to rename the spaceships to dragons and you wouldn't notice much of a difference. And what do you do with something like Wen Spencer's "Tinker"? (which looked to be the start of a great series before the main character went all Mary Sue) My personal rule of thumb for the intermediate cases is to consider the continuity with the present. If the world is supposedly the future or past of the real world, that loads some weight on the SF end of the scale; if it's in a world of its own, without placement in real-world chronology, that pushes down on the fantasy end. That's usually enough to make the difference. It's not perfect, but it helps. I have a calibre tag of "pulp SF" to cover the ray-gun fantasies of ERB, etc., which have the usual SF trappings but are so soft they need to be wrapped up like pasteurized processed cheese food product in order to be handled.

Regarding the drive space issue: Theducks is right about DVD, though an external HD would be another good option. They're cheaper than chemistry textbooks now. In either case, if you moved your originals to the DVD or external HD when importing them into calibre, they could then be stored safely somewhere to serve as a backup. Offsite backups are a Good Thing.

That would make it easy for you to do the work in stages, too. Import one folder of books into calibre, bulk-edit them to tag them by source folder, move the now-redundant folder to the backup medium. Repeat until all the books are imported/moved. Then you can go back through and edit again to catch the books that require tags beyond those defined by their original place in your directory structure, such as the ones mentioned that cover several different topics. Then you'll be able to find that one chapter on intermetallic states that's part of a book which is mostly about some other topic.

As to your first post, darknessangel, you got some solid answers despite the tone of that post, rather than because of it. This is a very important distinction.

[darknessangel]

*jawdrop @ the sci-fi explanation* Now that I read that... I guess that if I dump everything into fiction it would be difficult to find some sub-genre I like most. But for me Sci-fi is just an abreviation not an insult to anyone.

And I have two copies of my library, it's just that right now I'm travelling and left my large HDD at home and I got only the small with me... so... I cannot start with the bulk transfer.

GJMS

[rollercoaster]

I for one, would like to see a customizable folder structure.

but more then that I would like to distinguish tags I add from tags added via metadata lookup. simply because tags I add would be much more accurate to me and would not clutter as it does with online metadata lookup.

I think so because online tags are meant for make it easy for everyone by including all sorts of equivalent tags where as calibre is single user oriented and on a personal level that many tags are just a big pile of junk.

That would be a better way to organize then a custom folder structure. Am I the only one thinking this?

[iain_benson]

Originally Posted by rollercoaster

but more then that I would like to distinguish tags I add from tags added via metadata lookup. simply because tags I add would be much more accurate to me and would not clutter as it does with online metadata lookup.

You could prefix all the tags you create with a % (or some other of your choice) character so that yours are all grouped together.