[Old Thread] Why doesn’t calibre let me store books in my own directory structure?

[wilo108]

wow -- i wasn't expecting such a vociferous (or supercilious) response!

I can't address all of the points raised, but I'll try to speak to some of the reasonable ones.

I use a number of different systems and pieces of software to interact with my various collections, and while they all play nicely with each other, Calibre is the exception. I can see how Calibre will work if you're prepared to allow it to take sole and complete control over your libraries (the "black-box" approach), but that's not what I want. I am well aware of the advantages of tagging systems and metadata systems (being a developer of [open-source] reference management software myself), but there is no reason why such a system needs to be tied to any particular file-system representation. This is an unnecessary limitation which seems to be hard-coded so deep into Calibre that it is close to impossible to extricate it. It would be far preferable that Calibre be agnostic with respect to the file-system representation (e.g. via a modular and customizable f/s backend).

In answer to the more tangential comments, of course I keep multiple backups of my systems and data (off-site, automated), and the issue with maintaining multiple copies has nothing to do with the cost of storage (do people really think in these terms? -- comments like this almost come across disingenuous).

Being a (primarily) python developer I am quite capable of working with the command-line tools, but I have no need to. My post is to record my opinion that with a little bit more flexibility Calibre would be a very nice addition to my own setup, but the insistence on either taking over my file system or duplicating it is obstructive and entirely unnecessary. I'm reading a lot here about ways to work around the issues this creates, and clearly these work-arounds are acceptable for most, but (and I quote), "for me its not worth the bother."

[DiapDealer]

Originally Posted by wilo108

Being a (primarily) python developer I am quite capable of working with the command-line tools, but I have no need to. My post is to record my opinion that with a little bit more flexibility Calibre would be a very nice addition to my own setup, but the insistence on either taking over my file system or duplicating it is obstructive and entirely unnecessary. I'm reading a lot here about ways to work around the issues this creates, and clearly these work-arounds are acceptable for most, but (and I quote), "for me its not worth the bother."

And yet it was worth the bother to go to the trouble of joining a forum just to voice the opinion that it wasn't worth the bother?

[cvlowe]

Originally Posted by wilo108

In answer to the more tangential comments, of course I keep multiple backups of my systems and data (off-site, automated), and the issue with maintaining multiple copies has nothing to do with the cost of storage (do people really think in these terms? -- comments like this almost come across disingenuous).

Obviously you didn't even take the time to read the thread you necroed as a portion of it was devoted to the amount of space peoples libraries take as a reason for wanting to keep their old file and folder structure.

Actually, I want to add something here:

Comparing two similar programs that work for different types of files- Calibre vs XBMC

Calibre- I can't control the folder structure. Adding a new file is simple, just drag and drop it into the GUI, then edit the metadata add tags and never have to worry about it again. Finding a book is simple once I figure out what to search for.

XBMC- I have full control of the folder structure, but it has to be built in a certain specific way in order for the software to find the metadata- folder and file name has to be exact, and if it can't find the info, I have to google to find the info, rename the file and folder and try again and repeat until I get it right. Tags are possible but not easy to add/change/use. Say for example I want to watch a zombie movie, I can't easily just bring up a list of all my zombie movies and find one to watch.

After using both, I'd prefer to have a Calibre type system to organize my movies. I don't want to deal with files and folders- tagging is superior in every single way.

[eschwartz]

Originally Posted by wilo108

wow -- i wasn't expecting such a vociferous (or supercilious) response!

I can't address all of the points raised, but I'll try to speak to some of the reasonable ones.

I use a number of different systems and pieces of software to interact with my various collections, and while they all play nicely with each other, Calibre is the exception. I can see how Calibre will work if you're prepared to allow it to take sole and complete control over your libraries (the "black-box" approach), but that's not what I want. I am well aware of the advantages of tagging systems and metadata systems (being a developer of [open-source] reference management software myself), but there is no reason why such a system needs to be tied to any particular file-system representation. This is an unnecessary limitation which seems to be hard-coded so deep into Calibre that it is close to impossible to extricate it. It would be far preferable that Calibre be agnostic with respect to the file-system representation (e.g. via a modular and customizable f/s backend).

In answer to the more tangential comments, of course I keep multiple backups of my systems and data (off-site, automated), and the issue with maintaining multiple copies has nothing to do with the cost of storage (do people really think in these terms? -- comments like this almost come across disingenuous).

Being a (primarily) python developer I am quite capable of working with the command-line tools, but I have no need to. My post is to record my opinion that with a little bit more flexibility Calibre would be a very nice addition to my own setup, but the insistence on either taking over my file system or duplicating it is obstructive and entirely unnecessary. I'm reading a lot here about ways to work around the issues this creates, and clearly these work-arounds are acceptable for most, but (and I quote), "for me its not worth the bother."

I think I already answered that one (I tried to be thorough):

Originally Posted by eschwartz

Point of order -- Zem has pointed out a few good reasons why calibre doesn't work for him, basically boiling down to the inability to keep in sync with other software and the need for, as it turns out, multiple calibre libraries (I have a feeling that was added in since his time as a result of conversations like this). If you have any arrangements like that, feel free to mention them, your opinion will be entirely understandable in that case.

But -- it doesn't sound like that is why you object. It sounds like you want something like iTunes for ebooks which uses an inefficient database to track potentially-moving files all over the filesystem, because you want to keep using an IMO lousy flat directory-based structure with no flexibility whatsoever, but you want to use calibre to view it as a library, and think this free (!!!) software is required to change to suit your needs.

Also adding in frivolous support tickets when people lose their books and Kovid has to explain why. iTunes, which does exactly what you want, with music, has been the bane of several people I know, for example for importing music downloaded from the browser -- since they didn't realize it was stored in the temp folder (because they opened it instead of saving) it disappeared when they rebooted. I am sure Apple gets plenty of complaints about that sort of stuff too -- but they are a giant multibillion-dollar corporation and can afford to field those calls. calibre can't and doesn't care to.

Ironic that you refuse to use iTunes since on the contrary it is exactly what you seem to like...

[...]

Finally, why are you registering to this site for the sole purpose of complaining about software that you don't intend to use, for which you didn't have to pay anything, and doing so in an insulting and demanding manner?

Perhaps you should've mentioned that you are an edge case. And I still don't get why you object to iTunes...

Not sure why you think it isn't worth the bother, but that is entirely up to you. However, please do not come to us with your unnecessary opinion that it is an unnecessary limitation. (Obstructive, yes -- it is obstructing you. By design.) You are both wrong and rude.

(If you wish to take over the full workload and commit long-term to doing so, perhaps Kovid would be willing to let you maintain and answer support questions/bug reports about whatever new database structure you choose. Why is he required to make things difficult for himself to pander to your exclusive needs?)

Your needs are fine and just, and totally understandable. They are also incompatible with calibre's deliberate premeditated design.

You may work around that by duplicating everything via Save-to-disk, and considering calibre to be the canonical source of information. The only cost of duplication there is monetary/size. (And not everyone here can afford to drop money on extra hard drives, so perhaps they were merely speaking about what they know. You certainly did not indicate that that is not where your concern lies.)
If you cannot treat calibre as a canonical source of information, then there would be problems regardless, the problem being that those books are in one workflow and not the other.

You do not have to, but the option is there.

For a software developer, you show a surprising reluctance to reduce yourself to command-line tools, or deal with bother. Have you thought about writing simple scripts to incorporate the bits of calibre you want into your workflows? Should be easy, since you say you mainly work with calibre's development language. Or is that too much bother too?

Like DiapDealer says, as opposed to the I suppose non-existent (in comparison?) bother of going to the trouble of joining a forum for the sole purpose of insulting a software that you don't intend to use, because it is too much bother to use it.

[eschwartz]

Originally Posted by cvlowe

Obviously you didn't even take the time to read the thread you necroed as a portion of it was devoted to the amount of space peoples libraries take as a reason for wanting to keep their old file and folder structure.

If you post in a thread about a specific type of complaint, it is assumed you share the same or similar complaint. I did mention something about necroposting, didn't I? If you have a different complaint, surely a more suitable topic would be to start a new thread, about a different issue related to calibre's filesystem.

Then, you could've even explained why it was an issue for you, and saved everyone a lot o confusion. I for example could've cut right to the point, instead of spending most of my post theorizing that you were Yet Another Flat Tags Lover.

[DoctorOhh]

Originally Posted by wilo108

I'm reading a lot here about ways to work around the issues this creates, and clearly these work-arounds are acceptable for most, but (and I quote), "for me its not worth the bother."

Fair enough, have a great day.

[fidvo]

A tip for those having a hard time transitioning over from a hierarchical organization system:

Create a custom field that stores whatever hierarchy you want. Set it up as a hierarchical field, and use periods as your folder separators. Create a Save-to-Disk template that converts these periods into slashes so that you can export your library into exactly the folder structure you want if you ever decide to stop using Calibre. (Yes, this works, despite what the documentation says)

Now you can use the tag browser just like the Folders pane of Windows Explorer. Voila! Calibre is now Windows Explorer on steroids.

Before you start bringing up all the reasons why this still doesn't do what you want, play around with Calibre for a month. If you still don't like it, uninstall it. No harm done. But I think you'll find the pros far outweigh the cons.

You're welcome.

[Katsunami]

This old topic again

The one thing to do is: trust the program to do its job, and you'll be very happy using tags to find what you need. Just keep out of the internal library, unless you have a very good reason to be there. One thing mentioned is having to share the books with other people and programs continuously; in that case Calibre would not be a good choice as it, indeed, black-boxes the directory.

I have al lot more music than I have books (around 800 albums, some of which are 10-CD box sets; one even is a 40-CD classical set), and I love to find things using tags. However, as I use a program that lets me define the directory structure it uses when importing (Foobar, which basically is the 'calibre for music' IMHO, although it only exists on Windows), I have enough OCD to actually define the folders as seen in the screenshot:

\Music\FLAC\%artist%\%year% - %album% [%genre%]\Disc %number%-%total%\%track% - %title%.%extension%

(All my music is FLAC, but I also have an MP3 folder, which houses game soundtracks provided by GOG.com. Some of the newer games actually come with FLAC soundtracks. YAY!)

Yes, I know Howard Shore is not the artist, but I don't care. Over here, Howard Shore (composer), Capercaillie (band), Karen Matheson (singer), Jimmy Smith (composer/lead artist/sidekick) and the London Orchestra are all artists. Also I only ever have one genre for music. As with my e-books, I always use the very first publication date for an album, not the publication date of the actual edition. Thus I have CD's that were published in 1957.

[Kirtai]

Originally Posted by Katsunami

\Music\FLAC\%artist%\%year% - %album% [%genre%]\Disc %number%-%total%\%track% - %title%.%extension%

I do that sort of thing with the apps that use automatic renaming. It's actually annoying since I spend a lot of time on it and then never use it.
That, after all, is what metadata and seach tools are for, though I do find myself wishing for better ones sometimes.

(All my music is FLAC, but I also have an MP3 folder, which houses game soundtracks provided by GOG.com. Some of the newer games actually come with FLAC soundtracks. YAY!)

You should check GOGs older games too; they've added FLAC soundtracks to a lot of them without notification.

[BetterRed]

Originally Posted by DiapDealer

And yet it was worth the bother to go to the trouble of joining a forum just to voice the opinion that it wasn't worth the bother?

For me, that says it all, I can't be bothered saying anything further

BR

[Katsunami]

Originally Posted by Kirtai

You should check GOGs older games too; they've added FLAC soundtracks to a lot of them without notification.

Yes, I've noticed. I now have a high-DPI screen, and their installer was borked in 125% mode. They have silently updated many older games with their 2.0 installer, and I've seen that they also replaced mp3 with FLAC for some games.

I've recently started an update round for all of my 30-ish GOG games

[bethohio3]

Originally Posted by arspr

Another happy user with Calibre directory structure.

In fact I don't remember which one I used before Calibre... (Yes there was a before Calibre era, till one friend told me "don't you know about this GREAT piece of software?") But what the hell!, my books are neatly stored and shorted by author in HDD and if I rename or whatever them in the GUI the files are also neatly updated.

Agreed!

I'm actually surprised that people care how the files are organized. I worried with a little bit when it was clear that books with multiple authors weren't stored the way I thought of them. Then I realized it didn't matter at all--there's never any reason for me to access the books through the file system, so it really doesn't matter to me how they're organized as long as calibre can retrieve them and I can send them to my e-readers.

The only file based "work" done on my calibre library is by the backup software that backs it up regularly. I don't even look at it.

It's taking me a long time to clean up the metadata for my existing books, but once I finish it, everything will be accessible easily and quickly through searches.

If I discover at some point that I wish I had more metadata (different tags, a new hierarchy) then I can add it--and the files don't change at all. I think that's awesome!

Gotta go

[Katsunami]

Originally Posted by bethohio3

I'm actually surprised that people care how the files are organized.

If one wants to be able to use the files outside of Calibre and not store them twice to avoid keeping two locations in sync, I can imagine that this entire issue of Calibre creating its own library can be problematic. This is exactly the case why I never allowed iTunes to organize my files during the time I used an iPod; because I wanted to access the files using other programs, and using the filesystem.

With my books, I don't have this same problem. I use calibre exclusively, because there is no viable alternative, and I don't search for books in the filesystem itself.

It's taking me a long time to clean up the metadata for my existing books

Same here. My suggestion would be to polish the metadata into the books. If your library file ever gets corrupted, you can just move your books to another location, make a new library, and re-import the books. The metadata will be re-read.

If I discover at some point that I wish I had more metadata (different tags, a new hierarchy) then I can add it--and the files don't change at all. I think that's awesome!

As said above, I poslish the metadata into the books, as it only changes the OPF-file. I'm fine with that. I could get any calibre-specific metadata out of the books using Modify EPUB, should I ever wish to transition to a different program; something which seems unlikely at this point in time. As it stands, with calibre's current feature list and tremendous development pace, it's impossible to catch up to it when starting from scratch. I think that any non-Kovid developed successor to calibre will be a fork.

[bethohio3]

Originally Posted by Katsunami

Same here. My suggestion would be to polish the metadata into the books. If your library file ever gets corrupted, you can just move your books to another location, make a new library, and re-import the books. The metadata will be re-read.

Is "Polish the metadata into the books" the same as "Embed metadata"? (That's what I did)

Thanks!

[Katsunami]

Probably, yes, AFAIK.

The function is called Polish Book (which does NOT mean it'll translate the book into Polish ); it has an option to update / embed metadata into the book's internal OPF file. If you you only check that option, nothing else will be changed.