Keeping book on NAS but metadata.db on local (Ubuntu)

[chaley]

Where you are going is toward the question "why does calibre keep all the books in its own library structure."

This question has been asked innumerable times, and the answer is always the same. Calibre treats the content (the book) as a column in its database. However, instead of storing the book in the database, it stores it in a file. That fact does not change the basic design decision, that calibre controls the storage of the books. Calibre gets upset if the two are not in sync. When calibre is upset, it shows error dialogs. When error dialogs are shown, users get upset. Then it lands in Kovid's lap.

Yes, other programs made different decisions. I use MediaMonkey, which supports storing music files outside the library. And yes, I have been bitten more than once by the fact that MediaMonkey does not in fact control the storage of the files. In the end, I elected to use MediaMonkey's file management to avoid corruption. But that is my call. Your mileage will vary.

The salient point here is that calibre is designed to manage the entire library, books and all, as a database. It expects close-to-transactional behavior, and it expects to own the folder structure. If you use a NAS, you certainly violate the first, and you might violate the second.

Calibre has hundreds of thousands of users. Lets assume 365,000. Imagine what happens if each user posts one question per year. That would average to 1000 questions per day, which is a number that Kovid could not possibly cope with. His choice, which I think is reasonable, is to constrain documented functionality to something supportable, leading to (say) one question per .01 user per year. NAS, and the complexities that come with it, are far outside that range.

As for rebuilding the database, in general that doesn't work. First, some book files do not contain metadata -- text files are a prime example. Second, even if the books do contain metadata, there is no reason to believe that it is the metadata you have put into your library.

Apologies for raining on your parade, but calibre is built the way it is and is supported the way it is. Some things are hard or of no interest to the developers/supporters. You should feel free to fix them. I did that when I got annoyed at author_sort not working the way I wanted it to with ln,fn author names. You might be able to make libraries stored on a NAS work reliably across platforms and across NAS filesystem semantics, which would be great addition.

[q345]

I do understand ( and I mentioned it in my previous letter) that Calibre is different from let's say Media Monkey, so I realize that motivation to keep database file and library in the same directory is valid.

Kovid is doing a tremendous job by developing and maintaining an outstanding product, and I am the last to demand from him advanced features, lengthy explanations, multi-colored manuals (printed on the acres of rain forests) and the rest of goodies reserved for corporations which frequently charges outrageous prices and maintenance fees (like 30$ support calls) for much less robust products.

Regarding the particular NAS issue I simply would like to denote, that it would be a reasonable guess to say, that many people from those hundreds of thousands which you mentioned, are ALREADY using this undocumented/unintended/debug feature and the number will grow, since NAS are mainstream now. It's always possible to put a disclaimer saying

advanced feature !!!
no promise of future support and/or compatibility !!!
use it on your own risk !!!
no support !!!

After all it's free software with all the pros and cons and if somebody puts his library on NAS, then network collapses, metadate.db is out of sync so he decides to sue for 3,000,001$ and 56 cents for mental and physical abuse, decimating his life work , extreme stress and discomfort etc, Kovid simply can tell him to get lost. He owes nothing to nobody, but doing a huge favor to community.

All I am telling is that the functionality IS there, people USE it now, more people will ask for it and WILL USE it in the future, and the only problem is where to keep this knowledge : in Preferences, in FAQs, in Manual or leave it (as it now) on this forum in this thread. People will search in Google "Calibre NAS" and it will be one of the first hits. May be this is legitimate form of documentation in the digital age.

[chaley]

Originally Posted by q345

All I am telling is that the functionality IS there, people USE it now, more people will ask for it and WILL USE it in the future, and the only problem is where to keep this knowledge : in Preferences, in FAQs, in Manual or leave it (as it now) on this forum in this thread.

There is another problem beyond reliability. It is likely that the undocumented feature being depended upon will go away in 0.8, when (if things go well) the library structure is reorganized to use plug-in storage backends. This is what Kovid was talking about when he said that he won't guarantee the technique's availability. Even though it sometimes happens, encouraging people to use and depend upon something and then take it away is not a good idea.

The above notwithstanding, the project is open source. People will look for and find ways to use it that are not expected or supported. This is a good thing, encouraging a fountain of ideas like yours. If you want to write the manual for some of the hacks, you should do so. That makes the information available without creating commitments for Kovid. Instead, it creates commitments for you.

[q345]

Chaley, I wasn't aware ( being a new to Calibre - installed it a few days ago by accidentally "bumping" into it on the web) about future plans (plug-in storage), which might affect the design in a very substantial way. ( I am guessing one will be able to put the whole Calibre Directory on USB stick ).

There is a fundamental difference between Calibre and digital music apps, namely that metadata and data in mp3/flac/ogg are kept in the same file, so metadata in such a case is completely duplicated, hence trivially reconstructable by simply rescanning the directory tree with original files.

As you pointed out it's not the case with e-book/Calaibre where metadata is NOT part of the original file, so in case of metadata.db get corrupted/get out-of-sync, the restoration might be cumbersome at best and impossible at worst. For a person, who use Calibre to keep tracks on thousand of books/papers and has invested considerable amount of time over months and even years into creating metadata. that might be indeed a huge loss.

One of the possible solutions might be (obviously) to duplicate metadata by keeping all tags/info of a given instance of e-book in a separate small file in the Calibre directory where the e-book (all formats of it) itself is kept, IN ADDITION to metadata.db file itself. In case metadata.db file got corrupted, restoration will be a simple process of scanning the Calibre directory tree, reading all those small metadata files each of them is related to a single e-book only ( to be precise to all the formats of a single e-book) and rebuilding a "main" metadata.db file.

It would be interested to hear Kovid opinion if this idea has merit . In essence it's a back-up of metadata.db file "spreaded over" the whole e-book tree. It might also made the whole product more robust, regardless of NAS,
since even if few "small" metadata files are lost, it still will be possible to recreate the bulk of metadata.db automatically. This scheme is conceptually identical to digital music numerous apps , all of them simply put original music tags into database for performance/searchability reason, but it's also possible navigate a music collection by browsing original directory tree.

[chaley]

Originally Posted by q345

One of the possible solutions might be (obviously) to duplicate metadata by keeping all tags/info of a given instance of e-book in a separate small file in the Calibre directory where the e-book (all formats of it) itself is kept, IN ADDITION to metadata.db file itself. In case metadata.db file got corrupted, restoration will be a simple process of scanning the Calibre directory tree, reading all those small metadata files each of them is related to a single e-book only ( to be precise to all the formats of a single e-book) and rebuilding a "main" metadata.db file.

This is a good idea that I have been thinking about that for some time now. The file would be an OPF, and calibre already knows how to make them. It wouldn't be hard to write them into the calibre library along side the book formats.

The interesting part is maintaining them. If they are kept up-to-date, then performance of some operations could become unacceptable. For example, if setting a tag on 100 books requires generation of 100 files, performance would go from under a second to many seconds. My guess is no.

One alternative I have considered is a background task that trawls for changed books and writes the file. This would eliminate the performance penalty, but comes at the cost of some complexity and the possibility of the files being out of sync. For example, if I change 1000 tags and then quit Calibre, the OPF files wouldn't match the database. Calibre could delay exit, but then ...

Rebuilding the DB from this backup also isn't trivial. One problem is that the database IDs calibre uses would change, possibly breaking an association with books on devices. The new caching could make this problem go away, and perhaps SQLite permits setting the ID (like MySQL does). Another problem is that at the moment, the OPF does not contain all the metadata, in particular the custom columns. Work is underway to fix this, but it isn't done yet.

All of this is on my list to ask Kovid about once the 0.7 beta is released, custom column metadata is stored, and device collections support custom columns. However, you just asked him, so I might find out the answer early.

[kovidgoyal]

Why not just use backups to backup your library. That's what backups were invented for.

If you have thousands of documents and you care at all about them, you had better be backing them up. Backing up the metadata.db file in addition is a tiny step. And if you backup regularly you will lose only a finite amount of metadata, not the metadata of all your thousands of books.

I believe in using the proper tool for the job. Guaranteeing data safety is the job of backups. There is no way that a program that has write access to any file can ever guarantee data safety.

[Dopedangel]

Originally Posted by kovidgoyal

Why not just use backups to backup your library. That's what backups were invented for.

If you have thousands of documents and you care at all about them, you had better be backing them up. Backing up the metadata.db file in addition is a tiny step. And if you backup regularly you will lose only a finite amount of metadata, not the metadata of all your thousands of books.

I believe in using the proper tool for the job. Guaranteeing data safety is the job of backups. There is no way that a program that has write access to any file can ever guarantee data safety.

I think you are mixing two things backup of actual files and the backup of database file that is lost because of some error.
That is what he is asking something that would back it up so he won't have to re add the whole library and all the metadata changes back to calibre.

I also asked for a similar thing few months back you said it would add a lot of
reading and writing to the disk so will slow down calibre.
Well thats the actual thread.
And your last response is exactly what you are saying now
https://www.mobileread.com/forums/showthread.php?t=66768

[itimpi]

Perhaps one compromise might be that if you are editing the metadata for a single book (rather than a mass edit) then the .OPF file could be writen out to the Stanza folder as at that point any overhead would be minimal so the performance argument is not so important. I tend to go into editing individual book metadata when i first add them to my library.

I guess it could also be added as a (no-peristent) option on the mass edit screen for those (like myself) who already have a lot of books in the library and want this done and do not mind the once-off hit.

[q345]

I shouldn't have used the word back-up, since what I've suggested is not.
It's just an implementation of having data and metadata of an e-book in one place, like in mp3 file.

I think everybody agree that some of the numerous e-book formats are deficient because they don't have place for metadata, and that the source of the problems. So instead of file let's allocate one directory per e-book in all formats ( as it now in Calibre ) or even one directory per a particular format of a particular e-book and keep there e-book itself and an additional small file with it's metadata.

Chaley, regarding the performance, indeed if one uses Calibre to maintain a huge library and performs constant massive tag editing, it might be a slight drop in performance ( remember those additionl metadata file are very tiny),
but, on my uneducated opinion and I might be wrong, for majority of people working with Calibre, frequency of read request to database is much higher then the write requests. Obviously, if you try to use Calibre to manage Library of Congress collection it might be very slow, but I guess big libraries uses different tools anyway.


Kovid, just to clairify that what I suggest is completely independent/orthogonal of back-up. Of course regular back-up are a must for ANYTHING, including Calibre. Still, a reasonable efforts are made in any application to make it more robust and fault-tolerant and as somebody else mentioned it's not for the e-books per se.

You are the owner of the code and you have much better perspective on the issue, but I think implementation of this feature is not too difficult and it doesn't even change anything in the design. Simply if you modify some rows in metadata.db , you also modify some of these small files in correspondent directories ( The easiest way just dump SQL for a certain row into a file if this row has been changed). It might even be controlled by the option. These files are not used for regular search, but for recovery only.

Of course, one might ask what if people start tampering with this small metadata files and move them. Well, even now I can tamper with big metadata.db file and screw everything with a very little effort. A single warning in the manual "Files are automatically generated. Don't move/edit them" should suffice.

[kovidgoyal]

I understand you're concerned about your metadata. Don't be. Sqlite is an extremely robust piece of software. I've used calibre heavily for three years while simultaneously hacking on the code, and I have yet to corrupt the database, even *once*.

Newer versions of calibre will even automatically dump to SQL and re-create your database if it gets corrupted. Think about firefox, it uses sqlite to store its settings and various cached data. Have you ever lost your firefox settings?

[q345]

I think it goes circles and/or wrong direction.

The whole discussion was about placing e-books on NAS which is fairly reasonable considering people have large e-book libraries, want to access them across multiple computers/ over network and NAS is mainstream ( very exact reason why music collections are on NAS) People mentioned that one of the concerns what happens if metadata/ebooks got corrupted/got out of sync because they are placed in different places : metadata.db locally (because sqlite doesn't support NAS and for performance reason) and e-books on NAS.

I suggested a reasonable way to reduce penalty if it's happens (nothing new, just the same concept as with digital music files), which need to be implemented in such a way because of deficiencies of some of e-book formats).

I denoted that as a positive SIDE-EFFECT it also makes the whole scheme more robust because it simplify recovery if something wrong happens with metadata.db file REGARDLESS where it's located.


That's all point. The whole moto was NAS placement, not a reliability of metadata.db/sqlite (btw even very robust applications sometimes failed, sqlite include, so a little redundancy is OK I guess if doesn't cost a fortune/years to implement/maintain). I am perfectly fine with current scheme - it took me 30 seconds to put setenv OVERRIDE_METADATA in my .gnomerc - I am used to Unix/Linux so I can live without GUIs most of the time - no problem.

Of course since you've developed this product, you have infinitely better perspective on the whole issue. Regarding performance, I am new to Calibre and if people use it for many hours per day non-stop constantly modifying tags, then indeed additional read/write operations might degrade performance ( I have no idea for how much). My guess was ( and may be I am totally wrong) that overwhelming majority doesn't use Calibre in such a way, in that case performance degradation will be unnoticable.

Chaley was correct to mention that this scheme is not without shortcommings. Ideal ( and unrealistic ) solution would be that every e-book format will have a a place for metadata ( like digital music) and then the whole point became moot. Unfortunately it will not happen for a billion of technical and non-technical reasons

[kovidgoyal]

I'm confused. How does adding little OPF files to every book folder facilitate networked access?

The correct solution for networked access is a proper, concurrency supporting, network backend for calibre, which incidentally, is on my TODO list.

[kovidgoyal]

Wait a sec, I think I get it. You're saying that because you're keeping your metadata.db on a networked storage backend that doesn't support file locking you're worried your database will get corrupted?

Your database will only get corrupted in this scenario if you are performing concurrent writes to metadata.db and in that case you have much larger problems to worry about than metadata loss. You could lose the book files themselves.

For example when you change author or title calibre moves books into a new folder based on the new author and title. Now if you happened to concurrently change the metadata in two calibre instances that are using the same metadata.db, you will end up losing the book files (as well as the little OPF file if it existed).

calibre's current code base just isn't deigned to handle concurrent write access. Adding distributed metadata is not going to fix that.

[q345]

Kovid , now I am confused.

When I first (before going on this forum) tried to change directory in preferences to the something which was mount point to NAS directory, it failed to copy it. I checked the web and found other people complained about unability of sqlite to work with NAS but find no solution, so I put a question on the forum and Toddos recommended this environmental varaible settings, which worked fine for me. I thought thta since the functionality is already there, so I was advised to place the enhancement ticket, since I thought it's merely a small GUI fix. ( I mentioned that since it works for digital music it should work for digital books too)

Chaley correctly pointed me to the major difference between digital music and digital books format (lack of structure to hold metadata) and brought to my attention why it should be kept together ( I am new to e-books, I wasn't even aware before yesterday that you couldn't keep tags within the e-book itself). You also added that it's not a preferred mode of work and I got intrigued why, than I thought let's make e-book keeping in Calibre similar to mp3 file thru the "back door".

Now if the problem is not a placement of metadata.db on NAS but sqlite, would a solution be to use mysql ?

You mentioned networking support on your TODO list. Does it include access only (you have it now thru web access/server) or also keeping e-books on networked drives ? If you have some comprehensive solution, I am all for it.
(Take in account I don't have database/networking background, so I don't know what you are trying to do)

My main motivation to keep it on NAS was that I want to be able to access them and copy the e-books to my Palm TX which I use as my book reader, without keeping my PC turned on all the time and without need to manually copy e-books that I converted using Calibre, back to NAS. BTW I tried to access it thru Blazer web browser on palm TX using calibre server and it failed for some reason. With the whole library on NAS, I can just use FTP client. so for me the issue was not even the size ( I have a very small collection, I can keep it locally). I just don't want to keep my desktop turned on 24 hours per day, so I have NAS so I can access my photos, music, and e-book without PC. Many other people do the same.

I am sorry to waste your time on this discussion, but I admit I really got confused.

[solomon]

Originally Posted by q345

When I first (before going on this forum) tried to change directory in preferences to the something which was mount point to NAS directory, it failed to copy it. I checked the web and found other people complained about unability of sqlite to work with NAS but find no solution, so I put a question on the forum and Toddos recommended this environmental varaible settings, which worked fine for me.

You came in and stated that you couldn't get metadata.db onto your NAS, and toddos told you how to not do that.... Maybe we should back up and figure out what your actual problem is before solving it


I'm confused as to why you're unable to use the metadata.db on a NAS. I used mine that way for a few days with no issues.... What toddos and I were both seeing is that *OUR* NASes (which are really Windows Home Server boxes aka WHS) got mad about the metadata.db file being open all the time. I don't immediately see why a generic NAS would have any problem with the metadata.db file unless it's doing WHS's goofy file-based replication (which I believe to be fairly unique).


For WHS, it does background file replication instead of sector-level mirroring/RAID like most other implementations. For that it must have an unlocked file otherwise WHS is (rightly) concerned that it may make a bad copy - because the file is locked for writing and therefore either actively in flux or not yet been written to a coherent state.

Calibre is perfectly happy with its metadata.db right on the WHS share alongside the ebook directory, but *WHS* is unhappy because Calibre leaves the metadata.db file locked as long as it's running. (Toddos and I both also want full-time Calibre content servers running, and Kovid has said that his sqllite library has no way to open a db for read-only.) A "normal" user who stores his/her books on the NAS and opens and closes Calibre as needed (but doesn't leave it running all the time) should have no issues regardless of NAS type.

Richard