Changing the Calibre File Structure

[speakingtohe]

Another example or two of programs that will find anything anywhere and add it to a mostly unuseable database to select from are Windows Media center and Windows Media Player.

Add a new device and they can go out there indexing for hours even after you get them set up so they don't do this (for a while).

Calibre is much better behaved.

And how could you have multiple libraries etc. if it just took everything from one giant directory.

Seems kind of middle ages in computer terms to have one giant directory. Like throwing all of your food in one giant bag.

Helen

[ysbelman]

Ewe - windows media player. just a horrible example for any subject.

I don't mean throw everything into one flat directory, although it shouldn't matter. But let's say someone is a book collector, publisher, reviewer, or average user with some distinct sets of readable material, say text books they bought online, technical manuals for work, children's books, wife's digital magazines - user may want to keep it in a structure like this (example):
/ebooks/textbooks
/ebooks/work
/ebooks/magazines

This is easy for a backup services to traverse, if structure doesn't change offten (with each book title rename).
Easy for windows index service to keep tabs on file location. (Google desktop looks like is no more - discontinued).
If you need to send a coworker a bunch of work related pdf's - you may need to go back to the file system, zip up a bunch and emai them over.
Easy for a user to replicate a library on another computer.

But then, author/title file structure "works" too. There are ways around it. I personally keep the original files separate from those procesed by calibre. One day there maybe other apps that I will need to use, on different OS's. Maybe one day all of the books I store locally will go into the cloud.

[avantman42]

Originally Posted by ysbelman

I don't mean throw everything into one flat directory, although it shouldn't matter. But let's say someone is a book collector, publisher, reviewer, or average user with some distinct sets of readable material, say text books they bought online, technical manuals for work, children's books, wife's digital magazines - user may want to keep it in a structure like this (example):
/ebooks/textbooks
/ebooks/work
/ebooks/magazines

Having three Calibre libraries named "textbooks", "work" and "magazines", all in /ebooks, would do this.

Originally Posted by ysbelman

If you need to send a coworker a bunch of work related pdf's - you may need to go back to the file system, zip up a bunch and emai them over.

Again, that can be done with Calibre. Use the "Save to disk" functionality to save the books you want. You can even e-mail from within Calibre.

Originally Posted by ysbelman

Easy for a user to replicate a library on another computer.

If you want to replicate the library, just copy the library directory and it's sub-directories.

[speakingtohe]

Easy for windows index service to keep tabs on file location. (Google desktop looks like is no more - discontinued).

Eeeuuuwww Windows index service

I too keep my originals. I have seperate calibre libraries for fiction, non-fiction and cookbooks/reference.

I don't quite trust the cloud, (have recently had two experience of garbled data) so I keep a backup or two of stuff I may someday need with other family members in other locations.

[Cybercrawler]

I think that everyone is missing the point, I would be be inclined to agree with the Calibre advocates if not for a few things.

1.) Calibre is database only in the since that it is tracking information on the metadata. Not in the traditional since of single file DB Storage, IE SQL or others.

2.) It is storing everything in a file system!!!

The fact is that the books are files and there is not going to be a way around that until Calibre ingests the files that are given to it and holds it in a structured data format. All that Calibre is doing is taking the files that I give it copying them to a second directory adding a metadata file and covers. Don't get me wrong I love the program. But it is really hard to say that it is not a file organizing system as well when this is the difference of two books.

Me - Books\A\Allston, Aaron\Fate of the Jedi [4] - Backlash - (2010)\Fate of the Jedi [4] - Backlash - (2010).epub

Calibre - CalibreBooks\Aaron Allston\Backlash (1591)\Backlash - Aaron Allston.epub

I would love to either keep my structure or add the calibre ID to my structure.

I just don't understand why you can not select the database storage format at install and use the variables that are user selected to create the file structure.
Think of programs like
Plex
Sonarr
Sickbeard
Headphones
all of these programs store the metadata of the media in a second smaller DB and leaves the file structure alone. There are some rules of course on how the data needs to be stored so that the scanners can scrape, but at least the file structure makes since.

I operate a ftp for my friends to grab media from me, And I am consistently asked why is the books the only media that is not organized correctly when they look for stuff and Movies, TV Shows, Comics, Music, Programs, Games, Documents are, and the answer always has been that is the way calibre stores it. And I know that Calibre is dealing with copies so I could have a section for them but storing 300 GB of epubs in 2 locations seem stupid when having a 300 MB or less DB linking to the 300GB of data would work.Would you except Plex making a copy of all of the movies and storing them in a second location so that it is organized in plex? All of the other collectors that I am using have the metadata DB on solid state with a NAS backend. This is not possible with Calibre.

I think that it makes since to re-engineer to stay relevant with the changes in storage capacities and technology that is being used in the home.

My opinion...
Again I am blunt but I do love the program, I love the device integration and the metadata scraping is great. Please do not give up everyone you have a good base. Thank you

[theducks]

You are assuming a Monolithic DB is the Only DB type allowed to exist.
It is not.

I started with Paradox DOS (Ansa) Relational DB where every part was a separate file that worked together.
One single table could have as many as 64 members of its 'Family'
DB, Pri-Index, n Sec-indexes, 15 Reports, 15 Forms, a entry field template ( I can't remember them all now, but those were the main ones)

An Relation was established across many tables, all of them could also be used separately (with care )
A lookup Table could be shared among diverse applications without the need to replicate within each MDB. ex The same Postal code, City/State Lookup was used by every application with an street address. The table did not even need to be on the same drive as the application DB

Advantage: You worked with small files and did not need to open a huge DB package. Fast, even on a 8080 (XT) PC with MFM drives

Disadvantage: The 'Family' members of a Table must be maintained as a paired-sets (backed up and restored as a unit)

[eschwartz]

Also note that calibre is the work of one single developer, and the current structure is based on assumptions deep within calibre's code.

If someone was willing to step up and work on calibre to allow it to (safely) keep track of files the way other media programs can, that might change.

Then again, maybe not -- there are certain things to be said for keeping people out of the media storage, because people have a tendency to mess around.
There are things to be said for both sides, but at the end of the day it is simpler to deal with users if you just tell them "hands off". No disappearing files when the database loses track of a networked drive, no moving files around and losing them, etc.

iTunes for one has the option to "consolidate files", which copies all loose files into the iTunes Music folder, just like calibre does by default! And I have had to deal with people who lost their iTunes music because they didn't do this.
In one case, all the music had been added from Internet Explorer's Temporary Files cache, and promptly disappeared on the next boot.


Anyway, I am a big fan of using metadata-based copying, just like Worldwalker and others. The Content Server does pretty well for that.

---

You mentioned keeping the metadata.db on a solid state drive -- that is possible with calibre, see: http://manual.calibre-ebook.com/cust...ment-variables

CALIBRE_OVERRIDE_DATABASE_PATH - allows you to specify the full path to metadata.db. Using this variable you can have metadata.db be in a location other than the library folder. Useful if your library folder is on a networked drive that does not support file locking.

And finally, some brand-new news. The long-awaited Content Server revamp is underway! Expect that within the next few months you will be able to manage calibre remotely via a writeable Content Server -- so you can support access/editing from multiple locations.



e.g. Install calibre to your NAS and setup the server for remote access, the NAS itself handles everything: management, storage, file serving... You know the classic FAQ entry warning against NAS use? No more!

[LadyKate]

Originally Posted by kokoshmusun

Pepak, that sounds interesting. I hadn't thought of using calibre at the CLI to edit metadata (which I occasionally would like to do). I was just figuring out that pdftk can do that, but I'll look into that aspect of calibre as well.

And also conversions... but .pdf serves all my needs. I even tried an .epub vs .pdf of the same book on sony reader prs-600 and pdf was so much better. So I decided to stick to pdf for everything.

I find that pdf is a pain if you using a reader and it is not properly formatted. I find it difficult if not impossible to resize the font and keep the proper format without needing to scroll sideways to read the text.

It could be not finding the optimal pdf reader but it's not for lack of trying.

[LadyKate]

Originally Posted by DoctorOhh

I'm going and I can't myself.



Those catalogs were like Christmas to me. When I studied electronics in high school I learned the complete workings, waveforms, frequencies, voltages of and AM and FM radio... using tubes.



I actually won a Commodore PET computer in a raffle. A sizzling processor and a whopping 8k of memory topped off by a built in tape cassette storage system.

The big news back then was increases in modem speeds from 300 to 1200 to 2400 to etc....

For me I won a vic 20 and purchased the tape drive and a memor upgrade. Problem was when the memory (a plug in module) was in it changed the addresses and something written for the lower memory threshold didn't work with the memory upgrade in. Lol what a piece of crap... but it did get me interested enough to get an apple II+ clone that ran cpm... from there to dos based systems was a tiny step.

[LadyKate]

Originally Posted by kokoshmusun

Ok I gave up and I'm loading everything into Calibre now. I'm really nervous.

When I want to highlight, I go to the folder within Calibre and open the pdf in Xournal. Then I save highlights in Xournal, they have nothing to do with the original pdf anyway. I save the Xournal highlights in their own folder.

I keep a backup of my books outside of Calibre, just in case.

I hope it works.

My first PC was a 80286. I think I had 1 MB RAM and 40 MB HDD. I remember when I heard that Windows was going to come out with a version that would directly boot at the GUI (no DOS command line), I thought "what? that's impossible, it can't be!". I was a kid.

I would suggest that the better way to do it would be to save the book you want to highlight in a "working" folder, make your modifications, save it, and then replace the book in calibre. (if the metadata etc is the way you like it you can drag and drop the revised edition on the original.)

[LadyKate]

Originally Posted by FlorenceArt

Thank you for this clarification, it does make sense. Maybe because I spend my working hours trying to convince people that they will get more of out of software if they work with it rather than against it

I'll give this a try. Grudgingly, because what I really want from Calibre is to convert my books, not to manage them. But since I don't have much of a choice (looks like Calibre won't convert my books unless I add them to the library first, or unless I use command line, which I won't), I might as well try and make some use of it.

I downloaded the latest version of Calibre and found that I can adjust the preferences so that it doesn't create too many files or folders when I use the "save to disk" function. I guess I can live with that. At least it will probably be a bit easier than deleting all my books from the library after converting them

Thanks again for the useful input

Actually, you can always add the books you want to convert, convert and save them, then delete them in calibre.

I did that for a few months until the management of my ebooks by Calibre stated to make sense to me.

I had been rather anal about the way I organized my books.

Now I have my calibre library, a mirrored backup of that library on an external drive (or two), and a "pass to my reader" directory of books saved from calibre with metadata updated and names changed to the format I prefer on my tablet.

[DoctorOhh]

Originally Posted by LadyKate

For me I won a vic 20 and purchased the tape drive and a memor upgrade. Problem was when the memory (a plug in module) was in it changed the addresses and something written for the lower memory threshold didn't work with the memory upgrade in. Lol what a piece of crap... but it did get me interested enough to get an apple II+ clone that ran cpm... from there to dos based systems was a tiny step.

I applaud your enthusiasm but please try to refrain from responding to posts that are 5 years old.

[LadyKate]

Originally Posted by DoctorOhh

I applaud your enthusiasm but please try to refrain from responding to posts that are 5 years old.

Ooops. I was just going through the recent threads and didn't look at the original post date... my mistake.