lost my books database with personalized columns

[Misachat]

Hello,
I reinstalled windows 10 on my SSD and kept all my ebooks on other 1 tera hard disk. I forgot that on my SSD there is my database stored and so I lost it. I used Calibre to recompose my metadata db but I realized that it did not restore my personalized column. How come ? because I though that my 3 personalized columns were store in the individual metadata file that is stored for each ebook on my tera hard disk. Can I recompose the database with these column ? are these column saved in my individual files ?
thanks for help,

[theducks]

You were using one of the 'speedup' tricks that relocated the DB?


If you have not tried re-importing via ADD Books

RESTORE on the Library Icon or from the commandline should read the data from the metadata.OPF file stored with each title

Unfortunately, the Structure backup (metadata_db_prefs_backup.JSON) is stored in the same place metadata.db was

[Misachat]

hello,

i don't know of ADD books. what is that ? does it allow me to import from the individual metadata file in each book's folder ?
you confirm that it should restore everything.
yet i don't understand your remark about the structure. is that structure with my additional 3 columns not automatically stored in each individual metadata file ?
thanks,

[BetterRed]

Originally Posted by Misachat

I reinstalled windows 10 on my SSD and kept all my ebooks on other 1 tera hard disk. I forgot that on my SSD there is my database stored and so I lost it. I used Calibre to recompose my metadata db but I realized that it did not restore my personalized column. How come ? because I though that my 3 personalized columns were store in the individual metadata file that is stored for each ebook on my tera hard disk. Can I recompose the database with these column ? are these column saved in my individual files ?

I assume by, "I used Calibre to recompose my metadata db but. . .", you're referring to this function.



If you mean something else, then can you explain exactly what you did.

Custom column definitions are stored in the library database, and the normal location for the library database is in the library folder, in a file named "metadata.db".

The only way I know of to relocate a library database anywhere other than in its library folder is via a symbolic link, which is a file system artefact. Could you confirm this (or something similar) is what you had.

Where did you keep the "metadata_db_prefs_backup.json" file, it is normally kept in the library folder alongside the metadata.db file. The Library Maintenance->Restore database function needs that file in order to restore custom column definitions. If you've 'lost' that file too, then that would explain why your custom columns are not being 'recomposed'.

In which case you could try the following (I've never done this - so I am not 100% sure it will work)

• Firstly, backup what you have.
• Create a new library, and add the three custom columns to it. The definitions would have to be EXACTLY the same as the one's you've lost.
• Add a book with 'non-authentic' Author and Book Title values (eg "Author XXXXXX", "Book XXXXXX") and check that the custom columns are as you remember them.
• Then COPY a subset of the author and their book folders from the 'lost' library into the newly created one.
• Run the Library Maintenance->Restore database.
• If the custom columns are restored then redo the previous two steps with all author and books from the 'lost' library.
• If they're not restored then I think you'll have to use one of the Add Books options.

Relocating the database to an SSD via a symbolic link will have very little affect on performance since calibre loads the library database into memory when it opens the database. Memory operations (real and virtual) are invariably faster than file system I/O operations, irrespective of the media the file system is using.

If you elect to persist with doing this on the reconstructed library, then you should include the actual metadata.db and metadata_db_prefs_backup.json files in your library backups. Some of the better folder/file synchronisation utilities have options to do that.

BR

[PeterT]

See https://manual.calibre-ebook.com/cus...ment-variables


CALIBRE_OVERRIDE_DATABASE_PATH - allows you to specify the full path to metadata.db. Using this variable you can have metadata.db be in a location other than the library folder. Useful if your library folder is on a networked drive that does not support file locking.

[BetterRed]

Originally Posted by PeterT

See https://manual.calibre-ebook.com/cus...ment-variables


CALIBRE_OVERRIDE_DATABASE_PATH - allows you to specify the full path to metadata.db. Using this variable you can have metadata.db be in a location other than the library folder. Useful if your library folder is on a networked drive that does not support file locking.

- forgot about that one.

But no matter how its done, if the database is disjoint from the library then it must be considered when doing backups and restores of calibre library folders. EnvVars and SymLinks are not silver bullets, nor are they sprinkled with fairy dust to make them everlasting

BR

[PeterT]

Correct. Backups should include all, and to be even more careful, backups are useless unless tested by doing restores.

[Misachat]

ok I have learned that lessons but still is there any way that I would be able to view in mine individual metadata files what my definition was for these 3 customized columns ? I know it approximately but not well enough anymore.
thanks,

[Misachat]

Also where can I find the global metadata db that I have erased from my SSD ? and how is it called as well as the one defining the structure and the one which is the backup. than I will make sure in the future that I have a real backup in a separate place for thoses files.
thanks,

[theducks]

Originally Posted by Misachat

Also where can I find the global metadata db that I have erased from my SSD ? and how is it called as well as the one defining the structure and the one which is the backup. than I will make sure in the future that I have a real backup in a separate place for thoses files.
thanks,

see my post #2
Those ARE the names in every Library
The Folder name IS the Library NAME

[BetterRed]

@Misachat - global Preferences located are wherever you land when you go into Preferences->Miscellaneous and click the Open configuration directory button.

It's location varies according the OS your using. If you want to put it elsewhere, then that can be achieved via the CALIBRE_CONFIG_DIRECTORY environment variable. It isn't a database of such, it's a collection of folders and files containing global preferences, plugins, customisation data etc. You should include it in your backup/restore and disaster recovery considerations.

In the library book folders you should see files named 'metadata.opf'. Copy one of them somewhere (e.g. desktop). Open it with a half decent editor (e.g. Notepad++), replace all the '"' strings with nothing; that will make the file easier for you to read. You should be able to work out your original column definitions by looking at a couple of metadata.opf files.

You didn't tell us why you separated the library database file and the library preferences backup file from the library folder, or how (via the environment variable, symbolic link or otherwise) you did it.

I assume you did a destructive reinstall/reset of Windows 10, and that you didn't take an system image beforehand - that's another lesson to learn.

If you do have a pre-reinstall system image then why don't you restore it, secure the critical data (I would copy the contents of the Users and Program Data folders somewhere safe.) and then do a non-destructive reinstall, in Windows 10 it's done via Settings->Update & security->Recover->Reset this PC

BR

[Misachat]

I have copie the metadata file and my two most important columns are like this :
<meta name="calibre:user_metadata:#type_of_book" content="{link_column: value, is_multiple: null, rec_index: 25, is_custom: true, is_category: true, label: type_of_book, is_editable: true, table: custom_column_2, colnum: 2, name: Type of book, datatype: text, #value#: bd, column: value, search_terms: [#type_of_book], kind: field, display: {use_decorations: 0, description: }, category_sort: value, is_csp: false, #extra#: null, is_multiple2: {}}"/>
<meta name="calibre:user_metadata:#classification" content="{link_column: value, is_multiple: null, rec_index: 22, is_custom: true, is_category: true, label: classification, is_editable: true, table: custom_column_3, colnum: 3, name: Classification, datatype: text, #value#: Humour, column: value, search_terms: [#classification], kind: field, display: {use_decorations: 0, description: }, category_sort: value, is_csp: false, #extra#: null, is_multiple2: {}}"/>

IN order to create a new library, I will need to add the above 2 custom columm. beyond the name which I underlined and the fact that these are text field what else do I need to use in the definition ?

I separated the book folder (150 gb) which I stored on my 1 tera hard disk from the structure and other files which I did not even notice were put on my SSD by Calibre under my user name. is this the answer to your question. I know in the future I will keep a copy of that, I did not realize

I did not have an image either before reinstalling windows (another lesson).

Thanks

[BetterRed]

I think they are both tag like Text with Lookup names = 'type_of_book' and 'classification', and Column Header values = 'Type of book' and 'Classification' respectively. If you add those columns, and then do a database restore, the columns will be populated from the metadata.opf files - I think Before proceeding make sure you have backups that you know how to restore.

I don't know where you got the term 'structure data'. It is not a term I'm familiar with in the context of calibre (or anything else) - it does not feature in the Calibre User Manual.

The nearest analogy I can think of the SQLite database schema, which is contained within the library database file (metadata.db), which calibre stores within each library folder, alongside the author and book folders. The database schema includes custom column definitions, as well as library specific information, such as book list layout - which columns, column order, and column widths; some plugins store data in metadata.db - e.g. View Manager and Reading List.

As already discussed there are techniques to separate the database from the library itself, but they are only worthwhile if you encounter problems in server/NAS environments. Apart from a very marginal improvement in startup times, moving the database to an SSD won't have any effect on calibre's performance - because once calibre has started the database is memory resident.

Calibre puts global preferences in your home directory in the 'calibre configuration directory', which includes things like plugins, spell check dictionaries, tweaks, keyboard shortcuts, last used window sizes/positions etc, etc - basically all things that are NOT library specific.

This calibre configuration directory can be accessed in the main GUI program (library manager) by going to Preferences->Miscellaneous and click Open configuration directory. It can be relocated via the CALIBRE_CONFIG_DIRECTORY environment variable.

BR

[Misachat]

ok, just to say that I could find somewhere in an old directory called "consolidate all calibre libraries" a backup called ISDBN.json under another directory called plugins. I don't how it worked out but by indeed restoring the db with a few books and having tested it out positively it worked quite well. I did a full restore and Hoera ! everything is back.
so great help. thanks for it.
i don't have any clue about these plug ins which are I guess plug in I choose to put myself in Calibre although I don't know what this "consolidate" function is. but I am not very IT litterate to understand naturally all of that.
bye