Added or changed features request? Questions.

[Disfrutalavida]

I've experimented, read the forums, calibre.epub and tried to do the things below with no luck.

I think all these are about the add books function or how the database/library is read/written to in Calibre.

If anyone besides Kovid knows the answers please let me know since I am sure he is busy making other changes.

The viewer, metadata, send to device, and library functions work great once I get the data into Calibre. I even get book covers automatically. Cool.

1) Where can I get the expressions to build the database from the file names as I have them setup? Is there a central post? It's not in a sticky.

I will sort my books into separate folders with file names like
Author - Series 09 - Title.lit
Author - Title - Series 04.lrf
Last name, first name - Series 02 - Title.html
Last name, first name - Title - Series 05.txt
Same without the series info.
Then 7 imports and I'm done.

This should just be a matter of finding the correct expressions.
Anyone have a list?

2) Is there any way to import the author's name from the folder containing the files instead of from the file name? Can there be?

3) Can the ebooks be cataloged without copying the files to a new location?
Please please please. It takes a LONG time to copy 8000 books as they are being cataloged. It also wastes a lot of space.
If this can be done there will be no need to save them elsewhere until I need to convert them to read in my portable device.
I used to work with dbase and other database software and it can be done easily with them. I hope the same is true here but I don't know the architecture of this program and what's possible.

4) Can the SAVED folder structure be Lastname, Firstname instead of First Last? {author_sort[0]} or {authors} doesn't do what I want and the docs imply it cannot be done.

I have my ebooks stored on my drive in the format
Author'slastname, Firstname/Series/Title.ext which is the way books are organized in a library and at my age is logical to me.
Yeah I know. I should change. Sigh.
My audio files are organized the same way and my iTunes substitute can import and save the way I wish. Please say it's also possible with Calibre.

It also makes it really easy to find any book/FLAC/MP3 I want to give it to a friend. Sorry, but I was trained to use hierarchal structures for databases instead of metadata. I use Rockbox on my iPod instead of the stock Apple software for just that reason.

Thanks for reading and here's hope Calibre can become even better than it is now.

[itimpi]

1). There is no central repository of regex expressions although there are lots of threads in the forum discussing possibilities for different scenarios.

2). No, not without changing Calibre

3). You could in theory create a Calibre database containing just book entries using the command line tools without copying the actual files. However Calibre would then not know about your files which probably defeats the object of cataloging them.

4). The Folder structure is not amenable to change without a significant re-write of Calibre. You should treat the Calibre folders as a "Black Box" and only access the books in them via the Calibre GUI. Calibre expects all categorisation to be done by the use of tags, not by the folder structure (which is always subject to change without notice although in practise it is unlikely to happen).

[Worldwalker]

You'll get answers regarding the import, etc., from gurus much gooier than I, but there's a bit of it I can address. Namely, the questions about the calibre library and folder structure.

You shouldn't be worrying about how calibre organizes its files. As it is usually put, treat the calibre library folder structure as a black box. Don't even peek in there. Once you're using calibre, you shouldn't have to even think about those folders existing. Just interact with them through calibre itself. Like most people, I was accustomed to using my filesystem for metadata before I started using calibre; it quickly became obvious that using calibre (and real metadata) was so much easier and more efficient that I would never want to go back to the old way.

The tag system alone can beat any folder-based model all hollow. Let's say, for example, that you have a book of short stories by a favorite author, and they are a mix of genres: some fantasy, some science fiction, even a couple of mainstream mystery stories. If you were using the filesystem for metadata, how would that be categorized? Fantasy? SF? Mystery? With calibre, you just tag it as all three, and it'll show up in each category.

Or how about a book by two or more authors? In your system, which folder does that go in? With calibre, you don't care, because you just do an author search for whichever author you want and all of his books show up, even the ones where some other author's name was first on the cover.

You want to give a friend something ... say, all the fantasies from the Baen Free Library. So you select the tags you've set up for "fantasy" and "BFL" and export the books to wherever you want them. Bingo! Ready to go! No need to rummage through your folders trying to find the right books, or remember if this particular Mercedes Lackey book is redistributable or not; it's all in your tags.

With regard to the waste of space from duplicating files (if you don't delete the originals), I have somewhere around 2200 ebooks at the moment, of assorted sizes. They take up about 1GB. So your 8000 files can be expected to fit reasonably well into 4GB of space. When you figure that a 2TB internal hard drive currently sells for $130 (and with free shipping) on buy.com, we're talking around 50 cents worth of storage space. Or you could go the luxury route and spend eight bucks on a flash drive.

I'd guess that you, like me, remember the days of the cake-box RMO5 disc packs with their then-unimaginable 300MB capacity. A few GB here or there doesn't matter much anymore.

Calibre is an amazing tool if you use it as it was intended and work with it, not against it. It's a way to manage your books, not your files. Just keep that distinction in mind and calibre will be your devoted ebook slave.

[Disfrutalavida]

Hi Worldwalker,
Thanks for the response.

Though I understand the points you make, I have other reasons for having the/file structure in a different format.
I agree about tags, etc but it's just another name for a field in a database and I've been creating databases since 1970.
Sometimes I forget what I know that others don't and subconsciously assume, yes ASS U ME, that they will know what I mean, and not what I say.

I wrote about them in another thread you had with Kovid, Starson17 and Pepak.
Writing files in what has become a de facto structure has lots of advantages.
With a standardized folder/file structure I can create a database in Access, Excel or another program in minutes to manipulate organize or view data any way I wish and export or exchange that data with just about any other program that uses a database.
By using a standardized structure I can use templates from many programs with few changes, and data exchange becomes a breeze.

I did not explain my reasons for not saving when adding new files.
The main reason is speed.
Even with a 7200 RPM USB 2.0 drive hooked to a dual core 2.2 GHZ laptop it took a long time to add files.
Even on my quad core with RAID it took a while to add just 100 records.
That is not something I want to do very often as I plan to use 4GB SD cards to hold the files since my Palm T3 my iPAQ and all my PC's R/W them at close to USB 2.0 max speed.

The other reason is data recovery if the metadata.db file gets corrupted.
Currently I don't see a way for Calibre to recreate the database without reading and saving all the files again.
I added 20 files, closed Calibre and renamed the metadata.db file.
Starting Calibre did not rebuild the database. It had to be saved all over again so now all the files are duplicated even if you add then from the default folders Calibre uses.
With a standardized structure it's much easier to recover data from a hard drive or rebuild a database with another program.
Use the metadata for what it is good at but don't depend on it as your only recovery option. As you well know it's not IF data will become corrupted but WHEN.
And since the average person, not US of course, does not back up all their data the more and easier options for recovery the better and cheaper.

In 1998 I was using 3 9GB SCSI drives with one being a backup and also backed up to a 8GB DAT drive. Twice. 3 backups total.
A power surge wiped out the controller, and the 3 drives connected to it.
The tapes had become corrupted somehow. I lost 4 years of travel photos, data, etc since Seagate wanted $1500 to recover the data.
I cheated and bought new drives and exchanged the platters in a friends clean room, but still lost some data.
Never underestimate Murphy.

I remember when the 360/mod 30's at Argonne Labs were lucky to have 80MB of total capacity connected to them using 4 drives. Waste is waste, and I was taught never to waste anything. My current collection of eBooks is just under 8GB so I probably have more than 8000 books, or some are in multiple formats.
When you input with punch cards or the 16 switches on the control panel you learn to write tight code and waste nothing.

Nice to meet another Mercedes Lackey fan though I only enjoyed the SERRAated Edge series and her collaborations with Ms. McCaffrey.
Have you read the Rowan series by Anne McCaffrey, or the Kingdom series by Terry Brooks? Quite enjoyable also.

Calibre has got so many features I like, and some which I am amazed it has and would never want to do without, which I could not program without a lot of work . . . if ever.
What an incredible project Kovid has taken on, and in 3, count em 3, OS's!
I'd just like to see the input and output done in a way which makes it super easy to get to the data in case the metadata.db has problems.

I don't trust Murphy. OK I do. I trust him to strike when it's least convenient.

[DoctorOhh]

Originally Posted by Disfrutalavida

The other reason is data recovery if the metadata.db file gets corrupted.

Currently I don't see a way for Calibre to recreate the database without reading and saving all the files again.

I wrote basic instructions the other day with the exact steps needed to back up and restore the Calibre library. While I wrote the steps to help someone clean up their initial trouble I actually did the steps myself to ensure it worked.

I know you have experience with data backup. Once the initial backup is complete you can backup books incrementally by using the save to disk feature to update your backup directory with any recent additions. On restore the metadata.db file is recreated and all metadata is restored.

Once you feel secure with your data then using tags and the other features might be more attractive. This is just a minor part of your concerns but it is the one area I thought I could help out with.

[kovidgoyal]

A standardized structure is possible when you are thinking about a relatively homogenous colelction on a single operating system and a single file system. calibre is designed to work across multiple OSes and multiple file systems each with different restrictions on valid filename characters, file name lengths, character encoding and case sensitivity rules. In such a scenario, using a filesystem to store metadata, which is what you are advocating is simply not a good idea.

Note that calibre can export its database in both XML and CSV formats, for interop with other software.

[Starson17]

Originally Posted by Disfrutalavida

1) Where can I get the expressions to build the database from the file names as I have them setup? Is there a central post? It's not in a sticky.

No, there is no list and it's not in a sticky. It probably should be, except that work is in progress to make it even easier than finding it in a sticky. Currently, you have to search here. We agree that's less than ideal, but despite the fact that calibre is the fastest improving software I know of, and gets improvements almost weekly, that's not at the top of the list. It will get done.

2) Is there any way to import the author's name from the folder containing the files instead of from the file name? Can there be?

No, it's not an option now. Yes, it could be done, but there's not a lot of demand for it, so you can write it yourself, or request it in the bug tracker. I suspect the request would not go to the top of the list.

3) Can the ebooks be cataloged without copying the files to a new location?
Please please please. It takes a LONG time to copy 8000 books as they are being cataloged. It also wastes a lot of space.
If this can be done there will be no need to save them elsewhere until I need to convert them to read in my portable device.

No, it's not possible. Those of us who have committed to calibre's design often find that it uses less space, not more. I no longer have to store duplicate book copies for multiple authors in author-named folders. I no longer store as many multiple formats - I convert on the fly. My originals have become backups. If you must have your original structure for other programs, then calibre and your other programs are incompatible, and you may need both sets, but space is very cheap.

4) Can the SAVED folder structure be Lastname, Firstname instead of First Last? {author_sort[0]} or {authors} doesn't do what I want and the docs imply it cannot be done.

Author name is a single field. If you lie to calibre and tell it the author's name is Lastname, Firstname, it will store it that way. That causes problems with sorting, but there is a tweak to help deal with that. Try searching for "tweak" in posts by chaley.

[Worldwalker]

The first personal computer I owned was a Sinclair ZX-81 with 1k of RAM (which also was the video memory); naturally, I wrote games on it. In machine code. Manually poked into memory because I didn't even have space for a hex loader. I still have a soft spot for the Z80 chip. My first professional programming job was in FORTRAN on a PDP-11/34 running RSX-11M. After I moved on to newer and better computers (with almost the power of a cheap cell phone of today!), it took me years to really internalize the idea that I now had more than a 16k partition to work in, and I could freely use booleans instead of flag bits. When I think back to the things I did back then, when my whole dataspace was smaller than some variables I've used in the years since, I am both awed at what I did, and ashamed of the sloppy, bloated code I've written since (I should probably buy some bad code offsets).

Thanks a lot, Disfrutador, you're making me relive my past. Now I feel ancient.

Oddly enough, I've never been fond of the SERRAted edge books (though I do have the free ones on my 505, and several dead-trees ones floating around my overloaded shelves, because I am a bookaholic). The concept felt forced and the save-the-children bit seemed to override good storytelling at times. I've never liked the Last Herald-Mage trilogy, either, possibly because its "fill-in" nature constrained it too much, something that hurts a lot of prequels. On the other hand, I really wish she'd write more Diana Tregarde books, and more Free Bards books ... for some reason, my favorites are all the series that don't sell. One great thing about Mercedes Lackey: if you don't like one of her series, there's always a different one. She is a remarkably versatile writer. Also, Skandranon is too awesome for words. Though I'm sure he'd find some.

Back to calibre, your performance issues seem atypical to me. I'll leave that for the heavy hitters, though; I'm still very much calibre newbie.

One thing you'll quickly realize about calibre: it is advancing at a remarkable rate. It adds non-trivial new features in almost every weekly build. What it is today is so far beyond what it was when I first started using it a few months ago (I lived without saved searches? non-collection tags? merge?) and the way things are going, who knows what it's going to look like by next year? It'll probably wash my dishes and change the oil in my car.

[Disfrutalavida]

Thanks to all for your responses.
It's nice, very nice to see all the changes happening with Calibre.

dwanthny, thanks for that pointer.
One of my wish list done already.

Worldwalker, sorry for making you feel old.. . . . . but I'm considerably older ;-)
I started with IBM 1130 assembler, BASIC and Fortran, then moved to IITran and if I remember correctly, and I may not, COBOL, on a System 7, IBM 360/30 and 50.

Bad code offsets. Hilarious. They make about as much sense as carbon credits!

Got into custom circuit design and away from computers for 10 years then back into the "mainframe" world about 82, retired in 95.

I like Mercedes Lackey, though I think I come from an earlier time with different authors, though some "classic" sci-fi is funny and outdated when read now.
I'll read just about anything except horror, and enjoy thrillers to romances along with a fair amount of "non-fiction" though I always remember the victors write the history books.
I think I'm going to organize my collection, then find the Regex's I need.
By that time Calibre will probably have all the features I want!

I see where you are coming from Kovid.
I just like to "store" metadata in at least 2 places, or in at least 2 ways for redundancy, and you did do that.

Your library file structure is hierarchal which I can easily access and find files.
You just chose to make Author_sort First Last instead of Last, First.
I was just trained differently. I'm a big boy and I'll adjust.

I am impressed at the breadth of knowledge and programming skills needed to write a program like Calibre. I doubt I could have done it in my best programming days, but then my skill sets were in different areas.

Funny, but one of the questions asked, and dwanthny answered, is in one of the new posts today.
Glad to see I'm not totally crazy in what I want to see as a "feature.

Thanks again to everyone.

Rich

[Worldwalker]

Originally Posted by Disfrutalavida

I started with IBM 1130 assembler, BASIC and Fortran, then moved to IITran and if I remember correctly, and I may not, COBOL, on a System 7, IBM 360/30 and 50.

I'm at least old enough to have dealt with an IBM 1130, although it was in high school and obsolete even then. FORTRAN IV ... remember three-valued IF statements? Actually, there are times I miss them. I've been accused of writing FORTRAN programs in every language I code in, particularly Perl. It's sort of how English speakers learning Spanish put adjectives before nouns, and Spanish speakers learning English try to assign gender to nouns: your first language influences your mental grammar, and that's as true of programming languages as it is of spoken ones.

As far as SF, I like a wide range of stuff, with heavy emphasis on such things as the classic Heinlein juveniles (they're too good to be limited to kids) and a lot of the classic writers (and I still can't go to a con without looking around for Hal Clement for a moment before I remember....). I think that's one of the reasons why I'm such a techno-junkie: I don't like being away from my ebook reader, my netbook, my Nintendo DS, my cell phone, my mp3 player, my assortment of laser pointers in several colors, my pocket video camera, and all the other doodads, because they are, to me, emblematic of the SF future that I'm now living in. Alvin Toffler got it wrong: the future is fun!

[Disfrutalavida]

That's funny since I started sophomore year of high school on the 1130.
I vaguely remember Fortran IV, which always made me think of a planet name in Star Trek, since I wrote in IITran which was required where I went to school, but I do remember using three valued IF statements.

Where I live in Mexico Spanish is not really spoken as the average schooling is about 6th grade, and about a third of the words are from the Huichol or other indigenous groups.
Very, very idiomatic. Some of my Mexican friends from other parts of Mexico have a hard time understanding the older locals.
I understand the analogy you're using and agree.
This is the eighth country I have lived in and the language influences the culture, and vice versa, more than most people would believe.

I like my gadgets, BUT I could live without all of them.
In fact after I retired I bought an RV and traveled for 4 years, only using my PC to store the photos I took with my digital camera, and to decrypt Direct TV.
Hmmmmm, looks like I had gadgets after all. ;-}

I guess I have always lived in the tech future but with the values of the past and I like it that way just fine.

But some days, sometimes a week or 2 at a time, I like to just turn off the phones, lay in the hammock and read a book, nap or cuddle with my favorite person.
All I/we can hear is the bees, hummingbirds and the wind, and that suits me/us fine.