Calibre - An excellent program, but no thanks

[macr0t0r]

I'm amazed this thread is so long. I didn't realize so many people still insist on having their own specific directory structure.

To my knowledge, the directory structure was created mainly because the original flat single-file database was deemed "fragile." If that file got corrupted, you lost everything. The folder structure is not only more robust, but it allows you to browse and access your ebooks without Calibre running. I was quite happy when he made the switch.

However, I thought the whole point of Calibre was that you could get all your books in there and then just browse and search with the Calibre interface. It's a lot easier than mucking with the folders. To expect kovidgoyal to provide a completely customizable folder structure without breaking Calibre's management of those folders when you edit the Meta-data in the GUI seems...well...a bit much to ask of a single developer surviving on donations.

I look at Calibre the same way I look at iTunes. Some people cried foul when iTunes re-arranged things, but that's because they were determined to access their files the "old way." I think it was Google who said, "Search, not Sort!"

- Jim

[rcuadro]

Originally Posted by RobbieClarken

I'd like to be able to synchronise my ebook reader's SD card with the calibre folder so I can easily add and remove books in calibre without having to manually copy and delete them from the SD card one by one. However calibre's directory structure would be unusable on the DR1000.

I asked about a sync function a while back and I think it basically came down to having to extract the device serial number (or something like that) so Calibre could tell one reader from another and keep them in sync.

[dvs0826]

Originally Posted by kovidgoyal

calibre does not compress files, it does not store all books in one file, it is perfectly possible to do differential backups with calibre's library (I do it all the time). You can change where calibre stores its books, in fact you can have multiple libraries at different locations. You can run calibre entirely off a USB key if you want to.

The OP really should ask about things before making assumptions and then complaining.

And I defy anyone to give me a single use case where having calibre support their pet folder structure is better.

Here's a use case. You have been collecting e-books for a long time and you have a library of about 20GB of ebooks. You have 15GB of free disk space.

Honestly, making a copy is silly. The books are already there. What are you gaining by using your own custom format? In the FAQ it says it's more efficient than any scheme one could come up with on their own. But I doubt it's noticeably more efficient than storing links to the paths in an sqlite database.

Another use case: You have an ebook collection at home with a proprietary naming scheme, you use calibre and you want to bring your library to work so you can read books from work. But you don't have privileges on your work computer to install programs, all you can do is copy files. What do you do? Searching / sorting is impossible with a flat directory layout like what Calibre uses. If it didn't make a copy and instead used a more reasonable method like referencing the existing files you could just copy the directory tree, drop them down on the new machine at work and you'd be good to go. Instead you have to maintain 2 parallel databases, one in calibre and one in your proprietary format.

[rcuadro]

Originally Posted by dvs0826

Another use case: You have an ebook collection at home with a proprietary naming scheme, you use calibre and you want to bring your library to work so you can read books from work. But you don't have privileges on your work computer to install programs, all you can do is copy files. What do you do? Searching / sorting is impossible with a flat directory layout like what Calibre uses. If it didn't make a copy and instead used a more reasonable method like referencing the existing files you could just copy the directory tree, drop them down on the new machine at work and you'd be good to go. Instead you have to maintain 2 parallel databases, one in calibre and one in your proprietary format.

What I do is just install Calibre on a flash drive (you can also use an external drive)... You can just copy the calibre directory into the flash drive after making changes and it will be good to go when you get to work. That is what I do.

[dvs0826]

Originally Posted by rcuadro

What I do is just install Calibre on a flash drive (you can also use an external drive)... You can just copy the calibre directory into the flash drive after making changes and it will be good to go when you get to work. That is what I do.

Right, but my point was that not being permitted to install Calibre at work means that using this method forces you to end up with a flat directory structure at work. Now suppose you want to find some article from Volume 27, Issue 3 of a certain journal. How do you do it? If the database referenced your proprietary folder layout it's trivial. When you download the article you save it wherever you want, copy the layout to the flash drive, and that's it.

Copying the files is bad for the same reason code duplication is bad. And I know the answer is going to be "delete the original copy", but bottom line is it's not possible in some cases. And while you could respond with "well then calibre isn't for you", I would respond to that by letting you know that that's a total cop out, isn't it better to make the software work for as many people as possible than for as few people as possible? The responses in this thread are disheartening, all people are doing is attacking people who, even though they might be approaching things the wrong way or having a poor tone or attitude, ultimately just want to increase the user base by making the software work for them in addition to all the people it currently works for.

[Kostas]

I really can't understand all these complains about a free product...
Noone forces us to use it.

[dvs0826]

Originally Posted by Kostas

I really can't understand all these complains about a free product...
Noone forces us to use it.

I never acted like anyone was forcing anyone else to use it. Your logic is actually a little hard to understand though. Are you saying that nobody should ever suggest enhancements to free products? That's a little strange? What I never understood is people who love something so much that they don't want to see it get better.

[mypolar]

I'm grateful to the developer of Calibre since this software helps me to manage and organize my books as well as change them to EPUB format when necessary. Sure, I have encountered a few issues that I was able to work through- but I’ve had issues with a lot of software (Vista ). On the whole I really like this software.

Sony's library software is really .... have no words for how much I distain it. I only use Sony's software for buying or searching for Google books then just import them into Calibre for metadata and tagging etc.

[Kostas]

Originally Posted by dvs0826

I never acted like anyone was forcing anyone else to use it. Your logic is actually a little hard to understand though. Are you saying that nobody should ever suggest enhancements to free products? That's a little strange? What I never understood is people who love something so much that they don't want to see it get better.

1) My post was not addressed to you personally.
2) Actually, there are a lot of features in Calibre that I don't like at all. But still it is free and it does perfectly well things that no other free or professional tool can do. That's why when I discovered Calibre 2 months ago I personnaly considered that my first obligation was to say a "thank you".
3) AFAIK, enhancement suggestions of Calibre are welcomed and very often implemented. That's the way that Calibre has evolved. One month ago, Calibre supported conversions to only 2 formats (epub and lrf). Now it offers conversion to 11 formats!!
4) I still cannot understand the insistence of complains about Calibre directory structure that the author of this free program is unwilling or considers worthless/useless to modify. He might be right or wrong, but he's free to do what he wants with his program and we're free not to use it.
Regards

[dvs0826]

Originally Posted by Kostas

2) Actually, there are a lot of features in Calibre that I don't like at all. But still it is free and it does perfectly well things that no other free or professional tool can do. That's why when I discovered Calibre 2 months ago I personnaly considered that my first obligation was to say a "thank you".

You're right, it does a lot of awesome things. But since this wasn't a thread about the awesome things it does, I only focused on things that were relevant to this thread.

3) AFAIK, enhancement suggestions of Calibre are welcomed and very often implemented. That's the way that Calibre has evolved. One month ago, Calibre supported conversions to only 2 formats (epub and lrf). Now it offers conversion to 11 formats!!

The conversions are great. That's one of the awesome things mentioned above. But again I don't think it's really relevant to this thread, which is about the directory structure.

4) I still cannot understand the insistence of complains about Calibre directory structure that the author of this free program is unwilling or considers worthless/useless to modify. He might be right or wrong, but he's free to do what he wants with his program and we're free not to use it.

He's definitely free to do what he wants with his program. But people are also free to voice their opinions on how they disagree. On the previous page he asked for a use case on why keeping one's directory structure is better than converting to the calibre directory structure. I proposed two such use cases. I don't see how that qualifies as "insistency on complaining", I was merely answering the author's question.

It's obvious this is a highly desired feature. If there is an actual reason about why it is *bad* I'd love to hear it. My understanding is that it was because under the old format it was possible for the database to get corrupted and you'd lose everything, rather than just one book. My answer to that was to use an sqlite database. Everybody is happy under this scheme. I just don't see how "everyone is happy" is anything but superior to "only some people are happy".

[kovidgoyal]

@dvs0826

1) calibre's folder structure is not flat. Have you actually used calibre?

2) Storing links in a database is extremely non-robust. Already with even the current scheme I get endless bug reports from people that mess with the file names in the folders and the calibre loses track of the files. I can only imagine how many more such bug reports I'd have to deal with if I let people keep their calibre libraries distributed all over the place. And there's there's the problem of different file systems having different file name and case conventions on different OSes, in different phases off the moon.

3) As for your use case of not being able to install programs on an office computer. It is perfectly possible to keep calibre *and* its library on a flash drive and run it off that

4) Not having enough disk space. Are you serious? In this day and age of 1TB hard drives?

So no, your use cases do not even come close to outweighing the disadvantage I outlined in (2)

[dvs0826]

Originally Posted by kovidgoyal

@dvs0826

1) calibre's folder structure is not flat. Have you actually used calibre?

You're right, it's not flat. It turns out all of my "ebooks" are actually papers from academic journals, which pretty much always have different authors. So for my purposes it's flat. So I concede the statement that "it uses a flat directory structure" and instead just say that it uses a flat directory structure for me. If I'm still wrong then please advise.

2) Storing links in a database is extremely non-robust. Already with even the current scheme I get endless bug reports from people that mess with the file names in the folders and the calibre loses track of the files. I can only imagine how many more such bug reports I'd have to deal with if I let people keep their calibre libraries distributed all over the place. And there's there's the problem of different file systems having different file name and case conventions on different OSes, in different phases off the moon.

I don't think storing links in a database is bad as you make it sound. It's very easy to check, when loading a database, if a file exists. Or perhaps for performance scalability of larger databases defer the checking until the file is being opened. Display a message box "The file <path> could not be located. It is possible the file was moved via an external mechanism. Would you like to [delete] or [re-locate] this document?" Clicking delete removes all entries from the database, re-locate opens a file browser dialog and once a file is selected, it automatically receives all the previously stored metadata for free. Very simple.

As a side benefit of this approach, all of calibre's performance problems would be solved. It's better under 1.6.x, but it's still pretty slow. It took me 1 hour earlier to import a database of about 2,000 academic papers (about 9GB). It would have taken a few minutes tops if using the database method I proposed. calibre is also currently occupying 600MB of memory on my computer, no doubt due its database format. This would be solved as well, as it would only be necessary to maintain information about the items currently displayed on the screen, and perhaps an additional page or two for the purposes of caching and fast scrolling (I'm sure this is solvable as well under your current architecture, but you get it for free with a database).

Different filesystems having different case conventions is hardly a problem. All operating systems provide mechanisms to obtain the name of the file in the same case that the filename is stored in, so the database will simply store filenames in the correct case, and will not be modified directly by the user, only by calibre. If the user does modify the database by hand, that obviously cannot be supported. The only thing you will have to deal with is the slash character. Luckily / is an invalid filename character on all operating systems I'm aware of, so you use that for the separator.

4) Not having enough disk space. Are you serious? In this day and age of 1TB hard drives?

It's not that surprising. Rarely will you find an office pc with a 1TB hard drive. At my company I don't even get development boxes with more than 200-300GB, I'd be surprised if a non-technical office person got more than 100GB.

[kovidgoyal]

Umm calibre's "performance problems" have nothing to do with the database format. calibre currently uses an sqlite database to store metadata, as well as the relative path to the directory for each book, storing absolute paths would actually increase the memory footprint.
Incidentally the bottleneck when importing books is reading metadata from them, especially for PDF files.

As for the robustness issues, you're looking at it from the perspective of a single user. Sure *you* may be organized enough and careful enough not to lose track of files lying around in obscure corners of your filesystem. Most people aren't.

[rguion]

I am very surprised by this thread! I find Calibre to be a very useful program. The private ebook cache doesn't bother me--I've got tons of unused disk space on my external drive.

In fact, researching and finding Calibre's capabilities finally convinced me to buy the Sony Reader PRS-505. I've been able to transfer my collection of LIT/PDF/HTML books to the device pretty well. And the subscription download service is great--especially the New York Times.

[dvs0826]

Originally Posted by kovidgoyal

Umm calibre's "performance problems" have nothing to do with the database format. calibre currently uses an sqlite database to store metadata, as well as the relative path to the directory for each book, storing absolute paths would actually increase the memory footprint.
Incidentally the bottleneck when importing books is reading metadata from them, especially for PDF files.

Well, copying 10GB of data is a pretty slow operation. I would wager that's a much worse bottleneck than reading the PDF metadata, although I haven't tested it.

As for the robustness issues, you're looking at it from the perspective of a single user. Sure *you* may be organized enough and careful enough not to lose track of files lying around in obscure corners of your filesystem. Most people aren't.

As long as calibre is open, you can actually detect changes to any of its referenced files immediately by using o/s specific APIs to watch the filesystem. In any case I don't think this is a very big problem, as long as you can tell them what happened (i.e. the file must have been deleted manually) then I think it's very minor. Besides, since it's *already* using an sqlite database to store the paths, the absolute path thing could simply be an option specified at library creation time. You could put it in an "advanced" tab and put (Recommended) by the default option of allowing calibre to copy.


I'm just saying, there are solutions to all the problems you mention, as long as you're willing to think about the problem instead of just writing it off. It can even be made so that everyone who loves the current system would still be able to use the current system by default, it seems like a win-win. I don't get it.