Changing the Calibre File Structure

Worldwalker · 05-30-2010, 09:49 PM

The simple version:

You can't. Not yours.

The detailed version:

You are probably accustomed to storing your metadata in your computer's filesystem. That is, you are using the names of the folders to keep track of information about your books. That's how we all started. Calibre is why we stopped.

For example, you might have a folder called /ebooks, and under that you have /ebooks/nonfiction, /ebooks/fiction, etc. One of those, in turn, might have /ebooks/fiction/fantasy, and under that /ebooks/fiction/fantasy/lackey. The name of the folder gives you the type, genre, and author. So far, so good ... if you want all your Mercedes Lackey fantasy novels, you can find them right there. And you've probably got a folder called /ebooks/fiction/fantasy/flint where Eric Flint's books are stored.

But where do you put Shadow of the Lion? Under /lackey, /flint, or /freer? How about a book that has both fantasy and SF short stories? Where does a SF mystery, a historical fantasy, or a near-future thriller go? And that's just fiction. Non-fiction gets even hairier. Filesystems are actually a pretty crappy way to store metadata.

That's where calibre comes in. Calibre stores metadata as metadata, as much of it as you want. You can tag Shadow of the Lion as fiction, fantasy, historical fiction, and to-be-read, if you like, and it will come up in the list when you select any or all of those. It can be listed simultaneously under Lackey, Flint, and Freer. You can also note that you got it from Baen. And none of this has anything to do with the filesystem.

Calibre organizes books. Calibre is not a file organization program. Calibre is a book organization program. It's more than just the card catalog to your library -- it's the librarian who goes back to the stacks and brings you the book you're looking for.

The reason calibre copies all those ebooks over into its own set of folders is so that it can have exclusive access to them, and so it can know exactly what they are, where they are, and what they're named. Think of it as a black box, one with a big warning label stating "NO USER SERVICEABLE PARTS INSIDE!" Or maybe a storage room with a locked door with a sign on it saying "PRIVATE - Calibre Only".

You don't need to find those files with your computer's file manager. Calibre is there to do that for you -- or, rather, calibre is there to find your books for you. It will display your books, put them on your ebook reader, whatever you need, without you having to care about the actual computer files, only the books. If you actually do need one or more files -- perhaps you want to pack up all of the Harvard Classics to give to a friend -- calibre can collect them from its black box and put them wherever you want them -- and in whatever format you need to have them in.

The transition can be a bit hard. You're used to thinking of your ebooks as files and your filesystem as their metadata. Using calibre means thinking of them as books, abstracted from their computer representation, and with much more comprehensive metadata. It's a different way of looking at them, yes. But once you've made the jump, you won't want to go back.

You see, I started out the same way you did. I had all my neat hierarchical folders, my metadata in the filesystem, and I liked it that way. Then I found calibre. It took me a while to get my head around the whole idea of dealing with books instead of files, but once I did, I've never gone back. And now I'm explaining it to people who are where I was a few months ago. Give calibre a try, and you may end up doing the same!

Knipfty · 05-30-2010, 10:36 PM

Bravo! Well explained!

Lady Fitzgerald · 05-30-2010, 10:47 PM

Hmmm... Normally, this old broad has trouble wrapping her mind around new technology but this basic concept came easy. Actually, it's not even new to me. I used to work in warehousing. I remember over 20 years ago trying to get our IE rep to come up with something similar for tracking stock (back then, we kept everything in the bins and racks in numerical order; we were constantly moving stock to make room so we could keep it in order). He kept claiming it was impossible even when I gave him published proof that GE and the U.S. Army had been doing it for 20 years (heck, if the Army can make it work, anyone can). Once we were rid of him, we finally got a locator system that worked similar to calibre. We parked our material where ever it would fit, with the fastest moving stock being kept in more accessable areas. We told the computer system where we put it. Later, when we needed it, the computer would tell us where to find it, how much was there, etc. When I retired from there 6 1/2 years ago, many people still didn't know how to use it. Instead of assigning location codes in a logical order, they would use discriptive terms, like "back wall," etc., then gripe a bluestreak when the computer pick lists would have them running back and forth all over the place. It couldn't have been rocket science if I could understand it.

I have my music cataloged in a program similar to this. The data (artist, album name, track names, etc.) get inputted (both manually and from the CD) and initially viewed in a tree form similar to windows folder filing system but one can bring up reports using pretty much any criteria one wants. Calibre just takes it a step further in that a book can be accessed from within any search.

I have duplicate files in Windows because of the flat file structure. I recently learned one can use shortcuts to do the same thing. That would eliminate duplicate files but requires more manual input than finding something by tags. Once I learn more about how to use Excel 2007 (or Open Office; constant upgrading of MS Office is getting too expensive), I may just have one folder or just a handful of folders) per drive and just drop the files in whatever order they land and use a homebrew (or commercial, if I find one) searchable data base that searches tags to find them.

Btw, if calibre's filing system makes people nervous, just keep two folders; calbre's and one that uses traditional file organization. When I scan one of my books to a PDF file, I save it with a file name structure that includes author, title, series name and number (if part of a series), and ISBN number to a folder. Then I import it into calibre. I edit the meta data using the ISNB. It takes less time to edit the meta data than it did to type the original file name. Since I have plenty of room on my drive (1TB) now and the case I'll someday import the innards of my present computer to has room for 8 drives (6 data, 1 hotswap bay I'll use for backups, and a place to put an SSD for a boot drive once I upgrade to Win7), methinks I'll keep the original files, too (yes, I am paranoid).

Heathen · 05-31-2010, 02:52 AM

Thanks for the explanation Worldwalker. I understand what you are saying, and maybe I will come around to your viewpoint. However, as a person who tends to like to collect things and arrange them just so, it may take a while

Worldwalker · 05-31-2010, 06:32 AM

Quote:

Originally Posted by Heathen

Thanks for the explanation Worldwalker. I understand what you are saying, and maybe I will come around to your viewpoint. However, as a person who tends to like to collect things and arrange them just so, it may take a while

Do your arranging with tags. Go nuts! You can do all sorts of arranging, categorizing, and organizing that way.

Let me ask you this: why do you need to access calibre's files directly?

Lady Fitzgerald · 05-31-2010, 09:38 AM

Like I said, Heathen, if calibre's folder system makes you uncomfortable, just keep your originals in another folder filed however you want. Personally, I prefer my books filed in one folder without subfolders (unlike the way calibre does) but that preference is just a hedge in case calibre ever becomes abandonware (not likely) and doesn't keep up with changing OSs.

kokoshmusun · 06-01-2010, 10:33 AM

I think for those who don't want any software to touch their files and folders, an independent tagging program would do it. I used to use TaggedFrog in Windows. Now, in Ubuntu, I still haven't arranged for this, but I found a java program called SCAN at sourceforge.net can do the trick cleanly.

And by the way, Worldwalker's great explanation of calibre should somehow be included on calibre website, so noobs like me don't just play with calibre for 5 minutes and then uninstall it.

Starson17 · 06-01-2010, 10:51 AM

Quote:

Originally Posted by kokoshmusun

I think for those who don't want any software to touch their files and folders, an independent tagging program would do it.

It is important to recognize that Calibre doesn't touch the user's files and folders. It makes its own copy and touches only the copy.

Lady Fitzgerald · 06-01-2010, 12:59 PM

True dat. One of the reasons I like it. I can even go back and change the original filenames and it won't mess up calibre so long as I stay out of the "black box" (mayhap I should look into how to hide the "black box" from prying eyes or just relocate it somewhere so I or anyone else won't be tempted). I wish my music cataloger was like that.

eboyhan · 06-04-2010, 02:05 AM

Microsoft originally faced a similar problem with their Sharepoint product. Early versions of the Sharepoint Team Services product kept all documents, calendars, etc in user files.

Eventually they moved it all into a SQL Server database. It might seem counter-intuitive to put all the eBook content into a SQL database, but Microsoft realized two advantages: with everything in the SQL DB it becomes a "black box" which no one attempts to monkey with (an oft avowed objective stated here by Kovid and others as to why file structure will not be changed) -- put it all in SQL and the requests for file system structure requests will go away. Microsoft also did a lot of tuning so performance with docs in a DB is far superior to when they used the file-system. Don't know but whether there might be some synergies possible if both metadata and eBook content are maintained in a SQL DB.

Just a thought...;-)

itimpi · 06-04-2010, 02:48 AM

My ebook collection is about 20GB and growing - I would not be happy with this much data in a database. Also backup becomes horrendous as this is now an amorphous blob that needs backing up in its entirety. Under the current system I can do differential file copies that means in most cases very few files need copying as part of a backup process.

An alternative solution which would probably work is to give up using the human readable folder names based on book title and author and simply revert to using the numeric ID's (that tie it to the Calibre database) as folder names. That would make it too hard to navigate the folder structure manually so users would probably give up trying and stick to using the Calibre GUI. I would also suspect that this is much easier to implement.

kovidgoyal · 06-04-2010, 03:01 AM

Then we'd see all the current "let me use my file structure posts" replaced by "why does calibre store files in numbered directories" posts.

Worldwalker · 06-04-2010, 03:38 AM

Hidden files?

Lady Fitzgerald · 06-04-2010, 08:51 AM

Wouldn't hidden files require a different extension? That would make the files unusable unless converted back to the original by calibre. Presently, even though the files are stored (organized doesn't really fit here) is a folder/filename structure unique to calibre, the files are still identifiable and usable outside of calibre should calibre become unusable in a future OS (unlikely) or the user decides to use something else. By being able to specify where calibre's folder goes, a user can ensure the file will be recognized by most, if not all, automated or semiautomated back up schemes.

theducks · 06-04-2010, 10:47 AM

Quote:

Originally Posted by kovidgoyal

Then we'd see all the current "let me use my file structure posts" replaced by "why does calibre store files in numbered directories" posts.

Use Roman Numerals

05-30-2010, 09:49 PM	#1
Worldwalker Curmudgeon Posts: 3,085 Karma: 722357 Join Date: Feb 2010 Device: PRS-505	Changing the Calibre File Structure The simple version: You can't. Not yours. The detailed version: You are probably accustomed to storing your metadata in your computer's filesystem. That is, you are using the names of the folders to keep track of information about your books. That's how we all started. Calibre is why we stopped. For example, you might have a folder called /ebooks, and under that you have /ebooks/nonfiction, /ebooks/fiction, etc. One of those, in turn, might have /ebooks/fiction/fantasy, and under that /ebooks/fiction/fantasy/lackey. The name of the folder gives you the type, genre, and author. So far, so good ... if you want all your Mercedes Lackey fantasy novels, you can find them right there. And you've probably got a folder called /ebooks/fiction/fantasy/flint where Eric Flint's books are stored. But where do you put Shadow of the Lion? Under /lackey, /flint, or /freer? How about a book that has both fantasy and SF short stories? Where does a SF mystery, a historical fantasy, or a near-future thriller go? And that's just fiction. Non-fiction gets even hairier. Filesystems are actually a pretty crappy way to store metadata. That's where calibre comes in. Calibre stores metadata as metadata, as much of it as you want. You can tag Shadow of the Lion as fiction, fantasy, historical fiction, and to-be-read, if you like, and it will come up in the list when you select any or all of those. It can be listed simultaneously under Lackey, Flint, and Freer. You can also note that you got it from Baen. And none of this has anything to do with the filesystem. Calibre organizes books. Calibre is not a file organization program. Calibre is a book organization program. It's more than just the card catalog to your library -- it's the librarian who goes back to the stacks and brings you the book you're looking for. The reason calibre copies all those ebooks over into its own set of folders is so that it can have exclusive access to them, and so it can know exactly what they are, where they are, and what they're named. Think of it as a black box, one with a big warning label stating "NO USER SERVICEABLE PARTS INSIDE!" Or maybe a storage room with a locked door with a sign on it saying "PRIVATE - Calibre Only". You don't need to find those files with your computer's file manager. Calibre is there to do that for you -- or, rather, calibre is there to find your books for you. It will display your books, put them on your ebook reader, whatever you need, without you having to care about the actual computer files, only the books. If you actually do need one or more files -- perhaps you want to pack up all of the Harvard Classics to give to a friend -- calibre can collect them from its black box and put them wherever you want them -- and in whatever format you need to have them in. The transition can be a bit hard. You're used to thinking of your ebooks as files and your filesystem as their metadata. Using calibre means thinking of them as books, abstracted from their computer representation, and with much more comprehensive metadata. It's a different way of looking at them, yes. But once you've made the jump, you won't want to go back. You see, I started out the same way you did. I had all my neat hierarchical folders, my metadata in the filesystem, and I liked it that way. Then I found calibre. It took me a while to get my head around the whole idea of dealing with books instead of files, but once I did, I've never gone back. And now I'm explaining it to people who are where I was a few months ago. Give calibre a try, and you may end up doing the same!

05-30-2010, 10:47 PM	#3
Lady Fitzgerald Wizard Posts: 2,013 Karma: 251649 Join Date: Apr 2010 Location: Tempe, AZ, USA, Earth Device: JetBook Lite (away from home) + 1 spare, 32" TV (at home)	Hmmm... Normally, this old broad has trouble wrapping her mind around new technology but this basic concept came easy. Actually, it's not even new to me. I used to work in warehousing. I remember over 20 years ago trying to get our IE rep to come up with something similar for tracking stock (back then, we kept everything in the bins and racks in numerical order; we were constantly moving stock to make room so we could keep it in order). He kept claiming it was impossible even when I gave him published proof that GE and the U.S. Army had been doing it for 20 years (heck, if the Army can make it work, anyone can). Once we were rid of him, we finally got a locator system that worked similar to calibre. We parked our material where ever it would fit, with the fastest moving stock being kept in more accessable areas. We told the computer system where we put it. Later, when we needed it, the computer would tell us where to find it, how much was there, etc. When I retired from there 6 1/2 years ago, many people still didn't know how to use it. Instead of assigning location codes in a logical order, they would use discriptive terms, like "back wall," etc., then gripe a bluestreak when the computer pick lists would have them running back and forth all over the place. It couldn't have been rocket science if I could understand it. I have my music cataloged in a program similar to this. The data (artist, album name, track names, etc.) get inputted (both manually and from the CD) and initially viewed in a tree form similar to windows folder filing system but one can bring up reports using pretty much any criteria one wants. Calibre just takes it a step further in that a book can be accessed from within any search. I have duplicate files in Windows because of the flat file structure. I recently learned one can use shortcuts to do the same thing. That would eliminate duplicate files but requires more manual input than finding something by tags. Once I learn more about how to use Excel 2007 (or Open Office; constant upgrading of MS Office is getting too expensive), I may just have one folder or just a handful of folders) per drive and just drop the files in whatever order they land and use a homebrew (or commercial, if I find one) searchable data base that searches tags to find them. Btw, if calibre's filing system makes people nervous, just keep two folders; calbre's and one that uses traditional file organization. When I scan one of my books to a PDF file, I save it with a file name structure that includes author, title, series name and number (if part of a series), and ISBN number to a folder. Then I import it into calibre. I edit the meta data using the ISNB. It takes less time to edit the meta data than it did to type the original file name. Since I have plenty of room on my drive (1TB) now and the case I'll someday import the innards of my present computer to has room for 8 drives (6 data, 1 hotswap bay I'll use for backups, and a place to put an SSD for a boot drive once I upgrade to Win7), methinks I'll keep the original files, too (yes, I am paranoid). Last edited by Lady Fitzgerald; 05-30-2010 at 11:11 PM.

05-31-2010, 09:38 AM	#6
Lady Fitzgerald Wizard Posts: 2,013 Karma: 251649 Join Date: Apr 2010 Location: Tempe, AZ, USA, Earth Device: JetBook Lite (away from home) + 1 spare, 32" TV (at home)	Like I said, Heathen, if calibre's folder system makes you uncomfortable, just keep your originals in another folder filed however you want. Personally, I prefer my books filed in one folder without subfolders (unlike the way calibre does) but that preference is just a hedge in case calibre ever becomes abandonware (not likely) and doesn't keep up with changing OSs. Last edited by Lady Fitzgerald; 05-31-2010 at 04:06 PM.

06-01-2010, 12:59 PM	#9
Lady Fitzgerald Wizard Posts: 2,013 Karma: 251649 Join Date: Apr 2010 Location: Tempe, AZ, USA, Earth Device: JetBook Lite (away from home) + 1 spare, 32" TV (at home)	True dat. One of the reasons I like it. I can even go back and change the original filenames and it won't mess up calibre so long as I stay out of the "black box" (mayhap I should look into how to hide the "black box" from prying eyes or just relocate it somewhere so I or anyone else won't be tempted). I wish my music cataloger was like that. Last edited by Lady Fitzgerald; 06-01-2010 at 01:01 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
File Structure Question	Tiburon	Calibre	2	01-11-2010 09:21 PM
file structure after Calibre upgrade	daesdaemar	Calibre	1	11-27-2008 02:01 PM
Why no user file structure?	MoSo	Amazon Kindle	4	04-14-2008 08:17 AM
Question about the File structure	bbusybookworm	Amazon Kindle	3	11-23-2007 06:45 PM
Flat file structure...ugh!	BobVA	Sony Reader	7	11-03-2006 01:05 AM

05-30-2010, 10:36 PM	#2
Knipfty Enthusiast Posts: 28 Karma: 14 Join Date: Apr 2009 Location: Long Island, NY Device: Kindle/Kindle PC/Kindle BB	Bravo! Well explained!

05-31-2010, 02:52 AM	#4
Heathen Junior Member Posts: 4 Karma: 10 Join Date: May 2010 Device: Sony PRS 505	Thanks for the explanation Worldwalker. I understand what you are saying, and maybe I will come around to your viewpoint. However, as a person who tends to like to collect things and arrange them just so, it may take a while

06-01-2010, 10:33 AM	#7
kokoshmusun Member Posts: 13 Karma: 10 Join Date: May 2010 Device: Sony PRS-600	I think for those who don't want any software to touch their files and folders, an independent tagging program would do it. I used to use TaggedFrog in Windows. Now, in Ubuntu, I still haven't arranged for this, but I found a java program called SCAN at sourceforge.net can do the trick cleanly. And by the way, Worldwalker's great explanation of calibre should somehow be included on calibre website, so noobs like me don't just play with calibre for 5 minutes and then uninstall it.

06-04-2010, 02:05 AM	#10
eboyhan PandaMuse Posts: 104 Karma: 104 Join Date: Nov 2009 Location: Florida Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9	Microsoft originally faced a similar problem with their Sharepoint product. Early versions of the Sharepoint Team Services product kept all documents, calendars, etc in user files. Eventually they moved it all into a SQL Server database. It might seem counter-intuitive to put all the eBook content into a SQL database, but Microsoft realized two advantages: with everything in the SQL DB it becomes a "black box" which no one attempts to monkey with (an oft avowed objective stated here by Kovid and others as to why file structure will not be changed) -- put it all in SQL and the requests for file system structure requests will go away. Microsoft also did a lot of tuning so performance with docs in a DB is far superior to when they used the file-system. Don't know but whether there might be some synergies possible if both metadata and eBook content are maintained in a SQL DB. Just a thought...;-)

06-04-2010, 02:48 AM	#11
itimpi Wizard Posts: 4,552 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	My ebook collection is about 20GB and growing - I would not be happy with this much data in a database. Also backup becomes horrendous as this is now an amorphous blob that needs backing up in its entirety. Under the current system I can do differential file copies that means in most cases very few files need copying as part of a backup process. An alternative solution which would probably work is to give up using the human readable folder names based on book title and author and simply revert to using the numeric ID's (that tie it to the Calibre database) as folder names. That would make it too hard to navigate the folder structure manually so users would probably give up trying and stick to using the Calibre GUI. I would also suspect that this is much easier to implement.

06-04-2010, 03:01 AM	#12
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Then we'd see all the current "let me use my file structure posts" replaced by "why does calibre store files in numbered directories" posts.

06-04-2010, 03:38 AM	#13
Worldwalker Curmudgeon Posts: 3,085 Karma: 722357 Join Date: Feb 2010 Device: PRS-505	Hidden files?

06-04-2010, 08:51 AM	#14
Lady Fitzgerald Wizard Posts: 2,013 Karma: 251649 Join Date: Apr 2010 Location: Tempe, AZ, USA, Earth Device: JetBook Lite (away from home) + 1 spare, 32" TV (at home)	Wouldn't hidden files require a different extension? That would make the files unusable unless converted back to the original by calibre. Presently, even though the files are stored (organized doesn't really fit here) is a folder/filename structure unique to calibre, the files are still identifiable and usable outside of calibre should calibre become unusable in a future OS (unlikely) or the user decides to use something else. By being able to specify where calibre's folder goes, a user can ensure the file will be recognized by most, if not all, automated or semiautomated back up schemes.

Advert

Advert