Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 05-30-2010, 10:49 PM   #1
Worldwalker
Curmudgeon
Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.
 
Posts: 3,087
Karma: 722357
Join Date: Feb 2010
Device: PRS-505
Changing the Calibre File Structure

The simple version:

You can't. Not yours.

The detailed version:

You are probably accustomed to storing your metadata in your computer's filesystem. That is, you are using the names of the folders to keep track of information about your books. That's how we all started. Calibre is why we stopped.

For example, you might have a folder called /ebooks, and under that you have /ebooks/nonfiction, /ebooks/fiction, etc. One of those, in turn, might have /ebooks/fiction/fantasy, and under that /ebooks/fiction/fantasy/lackey. The name of the folder gives you the type, genre, and author. So far, so good ... if you want all your Mercedes Lackey fantasy novels, you can find them right there. And you've probably got a folder called /ebooks/fiction/fantasy/flint where Eric Flint's books are stored.

But where do you put Shadow of the Lion? Under /lackey, /flint, or /freer? How about a book that has both fantasy and SF short stories? Where does a SF mystery, a historical fantasy, or a near-future thriller go? And that's just fiction. Non-fiction gets even hairier. Filesystems are actually a pretty crappy way to store metadata.

That's where calibre comes in. Calibre stores metadata as metadata, as much of it as you want. You can tag Shadow of the Lion as fiction, fantasy, historical fiction, and to-be-read, if you like, and it will come up in the list when you select any or all of those. It can be listed simultaneously under Lackey, Flint, and Freer. You can also note that you got it from Baen. And none of this has anything to do with the filesystem.

Calibre organizes books. Calibre is not a file organization program. Calibre is a book organization program. It's more than just the card catalog to your library -- it's the librarian who goes back to the stacks and brings you the book you're looking for.

The reason calibre copies all those ebooks over into its own set of folders is so that it can have exclusive access to them, and so it can know exactly what they are, where they are, and what they're named. Think of it as a black box, one with a big warning label stating "NO USER SERVICEABLE PARTS INSIDE!" Or maybe a storage room with a locked door with a sign on it saying "PRIVATE - Calibre Only".

You don't need to find those files with your computer's file manager. Calibre is there to do that for you -- or, rather, calibre is there to find your books for you. It will display your books, put them on your ebook reader, whatever you need, without you having to care about the actual computer files, only the books. If you actually do need one or more files -- perhaps you want to pack up all of the Harvard Classics to give to a friend -- calibre can collect them from its black box and put them wherever you want them -- and in whatever format you need to have them in.

The transition can be a bit hard. You're used to thinking of your ebooks as files and your filesystem as their metadata. Using calibre means thinking of them as books, abstracted from their computer representation, and with much more comprehensive metadata. It's a different way of looking at them, yes. But once you've made the jump, you won't want to go back.

You see, I started out the same way you did. I had all my neat hierarchical folders, my metadata in the filesystem, and I liked it that way. Then I found calibre. It took me a while to get my head around the whole idea of dealing with books instead of files, but once I did, I've never gone back. And now I'm explaining it to people who are where I was a few months ago. Give calibre a try, and you may end up doing the same!
Worldwalker is offline   Reply With Quote
Old 05-30-2010, 11:36 PM   #2
Knipfty
Enthusiast
Knipfty began at the beginning.
 
Posts: 28
Karma: 14
Join Date: Apr 2009
Location: Long Island, NY
Device: Kindle/Kindle PC/Kindle BB
Bravo! Well explained!
Knipfty is offline   Reply With Quote
Old 05-30-2010, 11:47 PM   #3
Lady Fitzgerald
Wizard
Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.
 
Lady Fitzgerald's Avatar
 
Posts: 2,013
Karma: 251649
Join Date: Apr 2010
Location: Tempe, AZ, USA, Earth
Device: JetBook Lite (away from home) + 1 spare, 32" TV (at home)
Hmmm... Normally, this old broad has trouble wrapping her mind around new technology but this basic concept came easy. Actually, it's not even new to me. I used to work in warehousing. I remember over 20 years ago trying to get our IE rep to come up with something similar for tracking stock (back then, we kept everything in the bins and racks in numerical order; we were constantly moving stock to make room so we could keep it in order). He kept claiming it was impossible even when I gave him published proof that GE and the U.S. Army had been doing it for 20 years (heck, if the Army can make it work, anyone can). Once we were rid of him, we finally got a locator system that worked similar to calibre. We parked our material where ever it would fit, with the fastest moving stock being kept in more accessable areas. We told the computer system where we put it. Later, when we needed it, the computer would tell us where to find it, how much was there, etc. When I retired from there 6 1/2 years ago, many people still didn't know how to use it. Instead of assigning location codes in a logical order, they would use discriptive terms, like "back wall," etc., then gripe a bluestreak when the computer pick lists would have them running back and forth all over the place. It couldn't have been rocket science if I could understand it.

I have my music cataloged in a program similar to this. The data (artist, album name, track names, etc.) get inputted (both manually and from the CD) and initially viewed in a tree form similar to windows folder filing system but one can bring up reports using pretty much any criteria one wants. Calibre just takes it a step further in that a book can be accessed from within any search.

I have duplicate files in Windows because of the flat file structure. I recently learned one can use shortcuts to do the same thing. That would eliminate duplicate files but requires more manual input than finding something by tags. Once I learn more about how to use Excel 2007 (or Open Office; constant upgrading of MS Office is getting too expensive), I may just have one folder or just a handful of folders) per drive and just drop the files in whatever order they land and use a homebrew (or commercial, if I find one) searchable data base that searches tags to find them.

Btw, if calibre's filing system makes people nervous, just keep two folders; calbre's and one that uses traditional file organization. When I scan one of my books to a PDF file, I save it with a file name structure that includes author, title, series name and number (if part of a series), and ISBN number to a folder. Then I import it into calibre. I edit the meta data using the ISNB. It takes less time to edit the meta data than it did to type the original file name. Since I have plenty of room on my drive (1TB) now and the case I'll someday import the innards of my present computer to has room for 8 drives (6 data, 1 hotswap bay I'll use for backups, and a place to put an SSD for a boot drive once I upgrade to Win7), methinks I'll keep the original files, too (yes, I am paranoid).

Last edited by Lady Fitzgerald; 05-31-2010 at 12:11 AM.
Lady Fitzgerald is offline   Reply With Quote
Old 05-31-2010, 03:52 AM   #4
Heathen
Junior Member
Heathen began at the beginning.
 
Posts: 4
Karma: 10
Join Date: May 2010
Device: Sony PRS 505
Thanks for the explanation Worldwalker. I understand what you are saying, and maybe I will come around to your viewpoint. However, as a person who tends to like to collect things and arrange them just so, it may take a while
Heathen is offline   Reply With Quote
Old 05-31-2010, 07:32 AM   #5
Worldwalker
Curmudgeon
Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.
 
Posts: 3,087
Karma: 722357
Join Date: Feb 2010
Device: PRS-505
Quote:
Originally Posted by Heathen View Post
Thanks for the explanation Worldwalker. I understand what you are saying, and maybe I will come around to your viewpoint. However, as a person who tends to like to collect things and arrange them just so, it may take a while
Do your arranging with tags. Go nuts! You can do all sorts of arranging, categorizing, and organizing that way.

Let me ask you this: why do you need to access calibre's files directly?
Worldwalker is offline   Reply With Quote
Old 05-31-2010, 10:38 AM   #6
Lady Fitzgerald
Wizard
Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.
 
Lady Fitzgerald's Avatar
 
Posts: 2,013
Karma: 251649
Join Date: Apr 2010
Location: Tempe, AZ, USA, Earth
Device: JetBook Lite (away from home) + 1 spare, 32" TV (at home)
Like I said, Heathen, if calibre's folder system makes you uncomfortable, just keep your originals in another folder filed however you want. Personally, I prefer my books filed in one folder without subfolders (unlike the way calibre does) but that preference is just a hedge in case calibre ever becomes abandonware (not likely) and doesn't keep up with changing OSs.

Last edited by Lady Fitzgerald; 05-31-2010 at 05:06 PM.
Lady Fitzgerald is offline   Reply With Quote
Old 06-01-2010, 11:33 AM   #7
kokoshmusun
Member
kokoshmusun began at the beginning.
 
Posts: 13
Karma: 10
Join Date: May 2010
Device: Sony PRS-600
I think for those who don't want any software to touch their files and folders, an independent tagging program would do it. I used to use TaggedFrog in Windows. Now, in Ubuntu, I still haven't arranged for this, but I found a java program called SCAN at sourceforge.net can do the trick cleanly.

And by the way, Worldwalker's great explanation of calibre should somehow be included on calibre website, so noobs like me don't just play with calibre for 5 minutes and then uninstall it.
kokoshmusun is offline   Reply With Quote
Old 06-01-2010, 11:51 AM   #8
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by kokoshmusun View Post
I think for those who don't want any software to touch their files and folders, an independent tagging program would do it.
It is important to recognize that Calibre doesn't touch the user's files and folders. It makes its own copy and touches only the copy.
Starson17 is offline   Reply With Quote
Old 06-01-2010, 01:59 PM   #9
Lady Fitzgerald
Wizard
Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.
 
Lady Fitzgerald's Avatar
 
Posts: 2,013
Karma: 251649
Join Date: Apr 2010
Location: Tempe, AZ, USA, Earth
Device: JetBook Lite (away from home) + 1 spare, 32" TV (at home)
True dat. One of the reasons I like it. I can even go back and change the original filenames and it won't mess up calibre so long as I stay out of the "black box" (mayhap I should look into how to hide the "black box" from prying eyes or just relocate it somewhere so I or anyone else won't be tempted). I wish my music cataloger was like that.

Last edited by Lady Fitzgerald; 06-01-2010 at 02:01 PM.
Lady Fitzgerald is offline   Reply With Quote
Old 06-04-2010, 03:05 AM   #10
eboyhan
PandaMuse
eboyhan doesn't littereboyhan doesn't litter
 
eboyhan's Avatar
 
Posts: 104
Karma: 104
Join Date: Nov 2009
Location: Florida
Device: kindle dx, kindle touch SO, kindle fire, kindle fire hd8.9
Microsoft originally faced a similar problem with their Sharepoint product. Early versions of the Sharepoint Team Services product kept all documents, calendars, etc in user files.

Eventually they moved it all into a SQL Server database. It might seem counter-intuitive to put all the eBook content into a SQL database, but Microsoft realized two advantages: with everything in the SQL DB it becomes a "black box" which no one attempts to monkey with (an oft avowed objective stated here by Kovid and others as to why file structure will not be changed) -- put it all in SQL and the requests for file system structure requests will go away. Microsoft also did a lot of tuning so performance with docs in a DB is far superior to when they used the file-system. Don't know but whether there might be some synergies possible if both metadata and eBook content are maintained in a SQL DB.

Just a thought...;-)
eboyhan is offline   Reply With Quote
Old 06-04-2010, 03:48 AM   #11
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,110
Karma: 780247
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
My ebook collection is about 20GB and growing - I would not be happy with this much data in a database. Also backup becomes horrendous as this is now an amorphous blob that needs backing up in its entirety. Under the current system I can do differential file copies that means in most cases very few files need copying as part of a backup process.

An alternative solution which would probably work is to give up using the human readable folder names based on book title and author and simply revert to using the numeric ID's (that tie it to the Calibre database) as folder names. That would make it too hard to navigate the folder structure manually so users would probably give up trying and stick to using the Calibre GUI. I would also suspect that this is much easier to implement.
itimpi is offline   Reply With Quote
Old 06-04-2010, 04:01 AM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,455
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Then we'd see all the current "let me use my file structure posts" replaced by "why does calibre store files in numbered directories" posts.
kovidgoyal is offline   Reply With Quote
Old 06-04-2010, 04:38 AM   #13
Worldwalker
Curmudgeon
Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.Worldwalker ought to be getting tired of karma fortunes by now.
 
Posts: 3,087
Karma: 722357
Join Date: Feb 2010
Device: PRS-505
Hidden files?
Worldwalker is offline   Reply With Quote
Old 06-04-2010, 09:51 AM   #14
Lady Fitzgerald
Wizard
Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.Lady Fitzgerald ought to be getting tired of karma fortunes by now.
 
Lady Fitzgerald's Avatar
 
Posts: 2,013
Karma: 251649
Join Date: Apr 2010
Location: Tempe, AZ, USA, Earth
Device: JetBook Lite (away from home) + 1 spare, 32" TV (at home)
Wouldn't hidden files require a different extension? That would make the files unusable unless converted back to the original by calibre. Presently, even though the files are stored (organized doesn't really fit here) is a folder/filename structure unique to calibre, the files are still identifiable and usable outside of calibre should calibre become unusable in a future OS (unlikely) or the user decides to use something else. By being able to specify where calibre's folder goes, a user can ensure the file will be recognized by most, if not all, automated or semiautomated back up schemes.
Lady Fitzgerald is offline   Reply With Quote
Old 06-04-2010, 11:47 AM   #15
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,292
Karma: 6022733
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by kovidgoyal View Post
Then we'd see all the current "let me use my file structure posts" replaced by "why does calibre store files in numbered directories" posts.
Use Roman Numerals
theducks is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
File Structure Question Tiburon Calibre 2 01-11-2010 10:21 PM
file structure after Calibre upgrade daesdaemar Calibre 1 11-27-2008 03:01 PM
Why no user file structure? MoSo Amazon Kindle 4 04-14-2008 09:17 AM
Question about the File structure bbusybookworm Amazon Kindle 3 11-23-2007 07:45 PM
Flat file structure...ugh! BobVA Sony Reader 7 11-03-2006 02:05 AM


All times are GMT -4. The time now is 03:20 AM.


MobileRead.com is a privately owned, operated and funded community.