Changing the Calibre File Structure

[kiwikobo]

I have a copy of Windows 3.0 release candidate floppies if they're of any use? :-)) Otherwise most of my soft-sectored mini-floppies are DOS...

[Nathanael]

Originally Posted by Worldwalker

You really want proof of my age?

Hollerith codes and 3-valued IFs. Also, punch cards.

I think I'll just go decay quietly in the corner now.

You'll have company.

OK, I don't go back QUITE as far, but most of my undergraduate career was spent submitting code on punch cards. I'll never forget the night before senior project was due -- the CS basement was jammed to the gills with procrastinators. A guy sitting next to me had just spent an hour checking his cards. He grabbed up his trays, stood up and turned to rush over to the submission window when it happened. Somehow, his foot caught on the table leg. He went down and 800 cards went up, settling all over that corner of the room. To this day when I read the phrase "a look of despair" I think of that guy's face.

I did work on PDP-11s in graduate school.

--Nathanael

[Worldwalker]

That's why you use sequence numbers!

[Nathanael]

Originally Posted by Terisa de morgan

I didn't know about Windows 2.1

I think they meant Windows 286. Back then we called Windows the bastard child of DOSShell. I was so excited the day I finally upgraded my computer so I could run Windows. Had a blast for about a week (so this is cutting edge computing) then the beastly instability began to grate on me. Finally, one day, after finding myself kicked out to the DOS prompt once too many times, I blew Windows away and went back to DOS. Didn't touch Windows again until '95 (by which point I was working for Microsoft providing DOS and Windows support).

Originally Posted by Terisa de morgan

And my first computer was a Z-80, where I make my Pascal practices for the university.

Mine, as well: a TRS-80 Model IV with the 32kb upgrade, running Montezuma Micro CP/M (later upgraded to NZ-COM). Two internal 5.25" and 2 external 8" drives. Gawd I miss that machine.

Turbo Pascal 3.02 -- still my sentimental favorite IDE. And WOW! was it fast.

--Nathanael

[theducks]

Originally Posted by Nathanael

I think they meant Windows 286. Back then we called Windows the bastard child of DOSShell. I was so excited the day I finally upgraded my computer so I could run Windows. Had a blast for about a week (so this is cutting edge computing) then the beastly instability began to grate on me. Finally, one day, after finding myself kicked out to the DOS prompt once too many times, I blew Windows away and went back to DOS. Didn't touch Windows again until '95 (by which point I was working for Microsoft providing DOS and Windows support).

--Nathanael

I think Win 286 might be a parallel version for 16bit CPU's?

http://en.wikipedia.org/wiki/History...rosoft_Windows
Shows a few I did not own.

[sassanik]

Okay so when I look into my Calibre files area, I noticed that some books are listed by author and some seem to be by book? I dunno the whole thing seems a bit untidy.

I guess the better question is: Why book information and author information in two different files? or some such thing? So for example I see a folder named: Anne McCaffrey then later down the list I find one titled Maelstrom (one of her books) Is there a reason for this? Because when I look in Anne McCaffrey's folder I see "Maelstrom" and when I open that folder I see it in epub and lit formats.

Its not a big deal as I have plenty of hard drive space and its not like ebooks take that much space up. Someone mentioned having 20 gigs of ebooks! What did they do download all of Project Gutenberg?

[Starson17]

Originally Posted by sassanik

Okay so when I look into my Calibre files area,I noticed that some books are listed by author and some seem to be by book?

This indicates that you have the book title accidentally listed in the author name for those books.

I guess the better question is: Why book information and author information in two different files?

All this info is in a single database file -metadata.db

So for example I see a folder named: Anne McCaffrey then later down the list I find one titled Maelstrom (one of her books) Is there a reason for this?

Do a search in the GUI for "Maelstrom" - you'll find a record with that as the Author. Fix it.

Because when I look in Anne McCaffrey's folder I see "Maelstrom" and when I open that folder I see it in epub and lit formats.

This means you have two records for Maelstrom: one correct record and one incorrect.

(You may want to merge them, or maybe, just delete one. - It's also possible that you had a crash. If you don't find two records, then you have an orphan folder - there are ways to fix it.)

[theducks]

Originally Posted by sassanik

Okay so when I look into my Calibre files area, I noticed that some books are listed by author and some seem to be by book? I dunno the whole thing seems a bit untidy.

I guess the better question is: Why book information and author information in two different files? or some such thing? So for example I see a folder named: Anne McCaffrey then later down the list I find one titled Maelstrom (one of her books) Is there a reason for this? Because when I look in Anne McCaffrey's folder I see "Maelstrom" and when I open that folder I see it in epub and lit formats.

Its not a big deal as I have plenty of hard drive space and its not like ebooks take that much space up. Someone mentioned having 20 gigs of ebooks! What did they do download all of Project Gutenberg?

Starson17 listed the most common reason (and the cure).

Other possibilities were:
1) at some time, you used a "Save to Disk" while it was pointed at your Calibre library.
2) Your Library folder was not empty, when you started or you accidentally saved downloaded books there.
3) You have a alternate library program that is using the same folder as Calibre
The ONLY top level folders in th Calibe library folder are the names from the author column. (hint: open the Tag browser window and expand the authors list and see if there art any title names there. Any folder names, not there got are problems to be fixed.)

[Starson17]

Originally Posted by theducks

Other possibilities were:
1) at some time, you used a "Save to Disk" while it was pointed at your Calibre library.
2) Your Library folder was not empty, when you started or you accidentally saved downloaded books there.
3) You have a alternate library program that is using the same folder as Calibre

Yes. All of these can leave folders that Calibre does not consider to be its responsibility mixed in with those it does. Rather than uninstall, one can merely Move the Calibre library. That separates those that Calibre owns from those it doesn't own. If you follow the move with a Check Library Integrity, you get a clean library. Of course, I think this was all done once, so unless you've done another Save to Disk, or one of the above, it should have been cleaned.

IIRC, you (the OP) were at one time confused that a Save to Disk was needed to save the metadata. I wonder if theducks has hit this correctly and you have Save to Disk pointed to your Calibre Library??? That would explain this. To check, try to go through the steps of Save to Disk, and stop at the point where it asks you to point to the folder to save in. See if you are pointed to your Library! Calibre remembers where the last Save was done, so if it's pointed to the Library that would confirm this.

If so, clean it up by the two steps above, then see if any problems reoccur.

[sassanik]

Originally Posted by Starson17

Yes. All of these can leave folders that Calibre does not consider to be its responsibility mixed in with those it does. Rather than uninstall, one can merely Move the Calibre library. That separates those that Calibre owns from those it doesn't own. If you follow the move with a Check Library Integrity, you get a clean library. Of course, I think this was all done once, so unless you've done another Save to Disk, or one of the above, it should have been cleaned.

IIRC, you (the OP) were at one time confused that a Save to Disk was needed to save the metadata. I wonder if theducks has hit this correctly and you have Save to Disk pointed to your Calibre Library??? That would explain this. To check, try to go through the steps of Save to Disk, and stop at the point where it asks you to point to the folder to save in. See if you are pointed to your Library! Calibre remembers where the last Save was done, so if it's pointed to the Library that would confirm this.

If so, clean it up by the two steps above, then see if any problems reoccur.

I don't think I ever did a save to disk thing, at least not on the scale we are talking about, there are a ton of book folders.

I went ahead and moved my library (after backing it up of course!) and then checked the database integrity. This seems to have gotten rid of all the extra book folders that were showing up.


I am guessing that the problem perhaps occurred when I did a mass import the first time? I am not sure. Ahh well it appears to be fixed currently so I am happy.


Amy

[Starson17]

Originally Posted by sassanik

I am guessing that the problem perhaps occurred when I did a mass import the first time? I am not sure. Ahh well it appears to be fixed currently so I am happy.

Great!

[phenomshel]

Originally Posted by sassanik

I am guessing that the problem perhaps occurred when I did a mass import the first time? I am not sure. Ahh well it appears to be fixed currently so I am happy.


Amy

That could very well do it... I tried doing a mass import the first time and got such a horrendous mess that I cleared the calibre library folder and started OVER. This time I'm adding it all folder by folder and cleaning up the metadata as I go. I'm not sure this is precisely what I'm supposed to be doing, but it's working...even though it's going to take me til I'm 80 to finish.

[FlorenceArt]

Originally Posted by Worldwalker

You don't need to find those files with your computer's file manager. Calibre is there to do that for you -- or, rather, calibre is there to find your books for you. It will display your books, put them on your ebook reader, whatever you need, without you having to care about the actual computer files, only the books. If you actually do need one or more files -- perhaps you want to pack up all of the Harvard Classics to give to a friend -- calibre can collect them from its black box and put them wherever you want them -- and in whatever format you need to have them in.

The transition can be a bit hard. You're used to thinking of your ebooks as files and your filesystem as their metadata. Using calibre means thinking of them as books, abstracted from their computer representation, and with much more comprehensive metadata. It's a different way of looking at them, yes. But once you've made the jump, you won't want to go back.

Thank you for this clarification, it does make sense. Maybe because I spend my working hours trying to convince people that they will get more of out of software if they work with it rather than against it

I'll give this a try. Grudgingly, because what I really want from Calibre is to convert my books, not to manage them. But since I don't have much of a choice (looks like Calibre won't convert my books unless I add them to the library first, or unless I use command line, which I won't), I might as well try and make some use of it.

I downloaded the latest version of Calibre and found that I can adjust the preferences so that it doesn't create too many files or folders when I use the "save to disk" function. I guess I can live with that. At least it will probably be a bit easier than deleting all my books from the library after converting them

Thanks again for the useful input

[Lady Fitzgerald]

I've run into this on other fourms in MobileRead. People will complain that they didn't like calibre basically because they don't understand the multidimensional organization advantages of tags over a folder and file tree. They just don't get it when I try to explain why tag organization is superior to folder and file and come back saying how their system, using Windows system works so much better for them. It's really hard to let go of what one has used for a long time. I'll admit to being guilty of that. In fact, I still hang onto my original files just in case I decide to leave calibre (not likely but I like to keep my options open) since the file names contain the minimal metadata I need to file a book (author, title, series title, series number, ISBN #) without having to open the file and dig the data out. It's pure paranoia but, since I know enough to leave calibre's folder alone and I have plenty of disk space, it's a harmless paranoia that keeps this old broad happy.

[ysbelman]

I am trying to import a random collection of ebooks with filename structure from hell. I have customized the regex to parse <author> - <title> the best i could based on 50% of what i found. But there are so many variations in my filenames, with dashes inside title themselves, with author/title flipped, with author first/last flipped, etc. The result is - i have a Calibre directory structure with meaningless author/title information. The fact that Calibre strips the original file info - makes it that much harder to correct it.

Here is an idea which i'm sure has been suggested a million times.
Instead of bothering with directory structure which no user should see anyway, it could just dump all files in original onto disk.
Let the database keep reference to that file.

There will be no need to rename folders, move files around. This will be a lot more efficient, performance wise.
In order to help people with bad original content like me, allow for an iterative reimporting with the following workflow -
1. user imports original set, say 10,000 ebooks.
2. 1/2 of them contained title/author information out of sync with the original regex pattern
3. user selects a swab of rows in calibre, right clicks, chooses menu option "reimport".
4. user is asked to create a new regex
5. calibre knows the original filename of each item in the table
6. calibre iterates thu the list, and reimports using the new regex pattern
7. user continues to "refine" his import, step 1.

What does the community think?