Can we stop storing metadata in the filesystem please?

[Don Edwards]

The long explanation of why we aren't allowed to alter the directory structure of Calibre begins with pointing out the disadvantages of storing metadata in the filesystem.

But Calibre stores metadata - specifically the author and title - in the filesystem. In two places each. The author happens to be one of the specific things that the aforementioned long explanation cites as problematic.

If Calibre could refrain from doing this, it would also be more compatible with other software that has its own requirements for filesystem use. Such as Nook, which is useful for downloading files from Barnes & Noble but cannot handle subdirectories and gets confused if the name of one of its files - which includes a catalog number - is changed.

[eschwartz]

Metadata is not stored in the filesystem, it is used in the filesystem and stored in the database.

The only solid requirements Kovid had when designing, I am sure, is giving calibre absolute dictatorial control over the storage, allowing every book record to have a unique location (hence ids), and the kindness of leaving it easily accessible via the file browser.

None of that is going to change. Your free-for-all suggestion will not be used anyway, although theoretically another rigid structure could be used, to no one's satisfaction.

On the other hand this was quite the original objection, it quite made my day.

[BetterRed]

@Don Edwards - Calibre has to store the format files somewhere, given it integrates with external devices like ereaders and tablets the file system is the obvious (only) choice, rather than blobs in a database.

Calibre could use guids to name folders rather than author names and titles, or it could put everything into one folder using a common guid for each file for the same book.

But if it did, then when the book owner wanted to stop using calibre, the book owner would be left with the daunting task of making sense of a set of folders and files with guid names.

Would you rather be left with a bunch of folders that have names that are meaningful (even if stripped of diacritics and sometimes truncated) such as

MyLibrary\Daniel Defoe\Robinson Crusoe (86)\Robinson Crusoe - Daniel Defoe.epub

or something like

abd51a3b-b980-4c04-9680-dac5162026f7\6d592674-b5cd-43bb-9390-8a00c9b808b4\34e13190-0994-4445-abc8-9b37d840e37d.epub

Not being a B&N customer I wont address the specific issues relating to it.

BR

[eschwartz]

^^This, regarding the whole "Kovid is nice and lets us use the file browser in a pinch", also ensuring that calibre is not the only program that can ever read your data.

It would make his life easier regarding support/questions if he used a black-box format.

[Don Edwards]

Originally Posted by BetterRed

Would you rather be left with a bunch of folders that have names that are meaningful (even if stripped of diacritics and sometimes truncated) such as

MyLibrary\Daniel Defoe\Robinson Crusoe (86)\Robinson Crusoe - Daniel Defoe.epub

or something like

abd51a3b-b980-4c04-9680-dac5162026f7\6d592674-b5cd-43bb-9390-8a00c9b808b4\34e13190-0994-4445-abc8-9b37d840e37d.epub

I would rather be left with a single folder containing all the ebooks, with whatever filenames their source(s) might have stuck on them. This is what the Nook software requires for books from B&N (and it can't hardly find any other books), and the much-superior Mantano Reader software simply does not care where the books are stored or what the filenames are as long as they aren't moved or renamed. Both store their metadata in a database (not compatible, of course).

The best way to put it may be that Calibre is a library controller where I want it to be a library cataloguer.

Unfortunately, the firm word is that this is not going to happen. Which means that most of what I want Calibre for just cannot be done with Calibre, and there's no point in even letting the program see the 99.9% of my ebook library that I didn't write myself. I'd end up with two non-synchronizable copies of my library, which would be a real mess. And I'm still looking for a good e-reader program for Windows.

Oh well, that's life. The people who actually write the software get the final say on how it works. I've written quite a bit of software and understand that it MUST be that way - even when I disagree with their decision.

And I can still use Calibre for file-format conversions.

[cvlowe]

Originally Posted by Don Edwards

... the much-superior Mantano Reader software ...

Sounds to me like you don't need Calibre then, so why complain?

[itimpi]

Calibre can export the books it manages into whatever folder/file structure you want. Sounds as if that is what you should be doing to get the books into the structure you want.

[eschwartz]

Originally Posted by itimpi

Calibre can export the books it manages into whatever folder/file structure you want. Sounds as if that is what you should be doing to get the books into the structure you want.

As the FAQ says: http://manual.calibre-ebook.com/faq....tory-structure

If you are worried that someday calibre will cease to be developed, leaving all your books marooned in its folder structure, explore the powerful “Save to Disk” feature in calibre that lets you export all your files into a folder structure of arbitrary complexity based on their metadata.

[Don Edwards]

Originally Posted by cvlowe

Sounds to me like you don't need Calibre then, so why complain?

Mantano Reader doesn't run on Windows.

Originally Posted by itimpi

Calibre can export the books it manages into whatever folder/file structure you want. Sounds as if that is what you should be doing to get the books into the structure you want.

It can't put the original file names back. So it's incompatible with the Nook software. (To the extent that I've looked at the Kindle software, it's similarly lame, so I suspect the same applies there; but I haven't looked at this particular issue.)

And still, on a day-to-day basis I'd have to have two copies of my entire library on my Windows box: one for Calibre and another for my Android devices to sync with. With no easy way of automatically keeping them in sync.

But it's okay. Calibre does what the people who wrote it intend for it to do, which just happens to be a poor match for most of my needs. There's lots of software out there like that. I'll happily use it for where it does meet my needs.

[eschwartz]

It is for this exact reason that I have a custom column called "Original Filename". So I can always replace the original copy on my Kindle Touch cleanly, preserving annotations and pagenumbers.

I would be even happier if reading metadata from filename would fall back on reading from the file contents for metadata fields not named in the regex. That way, I could have the filename automatically imported. Even so, as part of my obsessive cleanup and normalizing routine, it doesn't take long. Small chunks at a time is the key to metadata control.

[jgaiser]

Originally Posted by Don Edwards

Mantano Reader doesn't run on Windows.

It can't put the original file names back. So it's incompatible with the Nook software. (To the extent that I've looked at the Kindle software, it's similarly lame, so I suspect the same applies there; but I haven't looked at this particular issue.)

And still, on a day-to-day basis I'd have to have two copies of my entire library on my Windows box: one for Calibre and another for my Android devices to sync with. With no easy way of automatically keeping them in sync.

But it's okay. Calibre does what the people who wrote it intend for it to do, which just happens to be a poor match for most of my needs. There's lots of software out there like that. I'll happily use it for where it does meet my needs.

I'm kind of confused with your needs. You seem to want something that runs on Windows (that you can view ebooks?) and something that can connect with your nook (?), but you don't want that unreadable conglomeration of letters and numbers that comes down from B&N to be changed, but you want something that you can manage your ebooks on. Does that sound about right?

Unless you are removing DRM, there is nothing beyond the unsupported Nook reader for windows and if you're removing DRM than there is no reason to not use calibre as it can very successfully transfer your ebooks to the nook. I've owned a couple of nooks over the years and have never had a problem transfer ebooks from calibre to nook. In fact I just tried it with nook touch and it did beautifully.

[cvlowe]

Originally Posted by Don Edwards

Mantano Reader doesn't run on Windows.

In my opinion, then, it's not "much superior"

Also, IMHO-

I've worked with software that "lets" me store files however I want. I used to think that was great. I've only been using Calibre a couple months (and probably only 10% of it's capabilities) and I couldn't be happier.

I dropped almost 1000 books into it and I've been working ever since trying to get the metadata correct. I'll admit I don't trust some of the automatic parts yet so I want to update my books one at a time. I've found so many duplicates and other issues and it's all so easy to correct and fix using the interface. Frankly I don't care about the file system structure- it's easy to navigate manually if I ever need to, that's all I need to know.

I haven't gotten the hang of exporting to my phone to read yet (my tablet got dropped and is broken and won't be replaced for a while). I do know from hanging around the forums it's possible once I decide to take the time to figure it out. From what I understand all I'll have to do is get the metadata correct then export the books directly to the device. It sounds to me you want to have a separate folder on your computer to manually copy- why do it the hard way? Just let the software handle it.

I have a question of my own- does anybody know of a program similar to XBMC I can use for my movies that handles metadata in a way more similar to Calibre (allowing tags and the like)? And SPECIFICALLY not "allowing" me to have to control the EXACT folder structure and filenames?

[BetterRed]

@Don Edwards - you don't have to retain the format files in the calibre folders.

You might put new books there for the purposes of creating the catalogue entry, updating the metadata, or doing a conversion etc. After which you could delete the format file from library. You could put a link into Comments or a Custom comments-like column to the book in your B&N folder structure or you could wrangle a Windows shortcut to the B&N book folder into calibre book folder.

The cataloguing features are driven from calibres database not the book folders. You cant get rid of the author and book folders because calibre uses the cover.jpg files, and it needs to have a metadata.opf file for the purpose of doing a database restore.

Some folks have two libraries one where they process new books to get the metadata up to scratch i.e. a 'preparation library' and an 'operational library'.

BR

[BookJunkieLI]

Originally Posted by Don Edwards

If Calibre could refrain from doing this, it would also be more compatible with other software that has its own requirements for filesystem use. Such as Nook, which is useful for downloading files from Barnes & Noble but cannot handle subdirectories and gets confused if the name of one of its files - which includes a catalog number - is changed.

Are you talking about the Nook App or a Nook ereader such as the STR or tablets?

Because if it's the app, what are you doing? Exporting to Calibre and then bringing it back into the app? Why? Are you using the Nook app to read books gotten from somewhere other than B&N? Again, why? Why not use a good all-around reader app like Moon+ or Aldiko? They don't care if the filename matches a catalog. And I'm fairly certain they can probably handle sub-directories.

If it's one of the ereaders then I know for a fact that they can handle sub-directories. I use sub-directories extensively on my STR. It's called the MyFiles section.

And if you really want to store all your ebooks in a single directory use the Save To Disk=>Save To Disk in a Single Directory option.

[jecilop]

@Don Edwards,
I would address one thing about libraries on the PC and Android and keeping in sync.
There are different ways of doing this easily. It only matters WHAT you really want.

Do you want your ENTIRE library on your Android device? I have a 12 gb library so I don't story it all. But if I did, rather than using Calibre to sync, you can store you library in a cloud drive folder that will always sync with changes. On Android, you can use apps like OneSync, DropSync, DriveSync, and some other variations to interact with your cloud storage in very specific ways (not just EVERYTHING). This can keep your entire library synced based on the changes you may make when editing on your PC. Technically, you can set the sync options from the app to be one way or two way or mirroring, etc....Also, you can set the library to sync into a particular folder on Android such as Books (the go-to folder for most e-reader apps). That folder doesn't have to match a folder name on the PC/Cloud location. There is no effort involved after this simple setup.

In my case, rather than store an entire library, I can either access the cloud drive and get what I want individually if not on the device through various apps including my reader app (Moon+ in my case), OR I can easily keep a large handful of books on hand on the device that store with the name I choose as well as location. This is another way Save to Disk is so powerful.

• I Save to Disk in my chosen format (epub) in a specific folder on my PC. This file names according the the template I determined (Author,Series,Index,Title,some custom info).
• That folder is synced to the cloud.
• That cloud is synced to my android devices (plural) into the Books folder.
• My e-reader auto imports or I can just open individually as my naming scheme makes it easy to find.
• My e-reader also syncs reading positions to any Android device (due to cloud storage access). I'll use those before I ever use the PC to read anymore (but I would use Calibre viewer to do so if I did).
• Also, when I finish with the file, I can delete it via my special "books in the bag" folder on the PC OR delete from my android device. It deletes everywhere due to syncing (not from the original Calibre storage, of course).
• As far as library management such as tagging that I've read it, I do have to open Calibre on my PC to add that.

I have no fuss, no mess to deal with in this setup. Cost: Android device(s), any free reading App or paid for ability to read it to me, cloud access I already use, inexpensive syncing app for phone, Rooting of phone or tablet for writing files where I want because Google screwed everyone over with KitKat 4.4 on that front :-)
Oh, and a reasonable donation to Kovid for the amazing FREE Calibre software I wish I'd found years ago.

I just pick the books I want, Save to Disk, and I'm done.

Should you decide you do want to keep the entire library on the Android device, I suggest USB copying. Then set up the apps sync to take care of any changes or miscellaneous stuff.

Pretty simple..copy your library folder under Books on Android (or whatever folder your reader reads from). That's it. The reader on Android should take care of imports from there. Just be sure to Polish the metadata into the book so you don't miss seeing that info you might update.

I'm curious, does Nook handle your own epubs (or other formats)? You can likely find a way to de-DRM so you can manage your book files as you want.