Metadata into book, again?

[enonod]

I have seen this question in one form or another and the answers leave questions for me.
When metadata is updated manually, say Author, Title etc. I presume an OPF file is lodged with the book in its folder.
I understand (I think) that when the book is saved to disk or sent to a device the opf file data is written into the book file in the same way as exif data in a photograph.
I am presuming that Save to Disk is making a backup to another disk (I don't know if it is simply the books or the whole folder for an author or the whole library.

1. If a book is sent to a device and the opf incorporated into the book file, is it only in the copy that arrives at the device (or disk) or is it the original as well?
2. If a book is requested via catalog is the result the same or only for pushing?

3. If it is both, why is it not incorporated immediately instead of producing opf?
4. If it is only the one despatched, why?

Thank you for any enlightenment.

[theducks]

Originally Posted by enonod

I have seen this question in one form or another and the answers leave questions for me.
When metadata is updated manually, say Author, Title etc. I presume an OPF file is lodged with the book in its folder.
I understand (I think) that when the book is saved to disk or sent to a device the opf file data is written into the book file in the same way as exif data in a photograph.
I am presuming that Save to Disk is making a backup to another disk (I don't know if it is simply the books or the whole folder for an author or the whole library.

1. If a book is sent to a device and the opf incorporated into the book file, is it only in the copy that arrives at the device (or disk) or is it the original as well?
2. If a book is requested via catalog is the result the same or only for pushing?

3. If it is both, why is it not incorporated immediately instead of producing opf?
4. If it is only the one despatched, why?

Thank you for any enlightenment.

Performance
You have 5,000 books: You change a Tag that is used in 3000 of those.

That is 3000 BOOK TITLES * formats to be opened and adjusted.

Or

simply a quick write a (BACKUP) to 3000 OPF files

[unboggling]

The OPF is mainly for calibre to use as a backup if you ever have to restore the database.

The metadata resides in the backend database tables and is stored in file metadata.db.

As I understand it, conversion or export (save, send, email, get from content server) update some metadata internal to the book format, depending on the type of format, of the exported or converted copy, but not in the original base format the convert or export used to make the copy. Polish Books (EPUB or AZW3 only) or plugin Modify ePub (EPUB only) can update metadata in the base format without a conversion.

In preferences for Saving to disk, there is a tick box to save the OPF file along with the book, or not.

[eschwartz]

When you update metadata in a book, record, the database records that, writes to the OPF as a backup, just in case, and leaves the original alone. ANY time you send the book anywhere -- email, save to disk, send to device, download via calibre content server, it updates all metadata fields that the format supports in the process. If you must see that in your original copy too, Polish ebooks can manually start that process, but depending on how many books you have, it will take quite some time...

There really is no need. It never leaves calibre without the update.

[BetterRed]

Originally Posted by unboggling

Polish books or plugin Modify ePub can update metadata in the base format without a conversion.

Polish Books only works on ePub and AZW3 formats, and Modify on ePub. Most other formats can have the metadata in the library format file updated with a like to like conversion, although that can have undesirable side-effects. Sometimes its better to enter the data using a format specific tool - eg Word for DOCX, PDF XChange or Acrobat for PDF.

BR

[eschwartz]

You could also do a save whole library to disk and import (using option 2 in Add Books) into a new library called "updated metadata" or something. All supported metadata is updated in a save to disk, with no undesirable side effects.

[unboggling]

Originally Posted by BetterRed

Polish Books only works on ePub and AZW3 formats, and Modify on ePub.

Thanks. I clarified that in #3 above, to avoid future confusion.

Originally Posted by BetterRed

Most other formats can have the metadata in the library format file updated with a like to like conversion, although that can have undesirable side-effects.

Does like to like conversion update the metadata in the input format and the output format? Or only in the output format, which is what I'd assumed?

[BetterRed]

Originally Posted by unboggling

Does like to like conversion update the metadata in the input format and the output format? Or only in the output format, which is what I'd assumed?

Only the output file, but that can overwrite the input file if the settings in the save_original_format tweak are set to False.

BR

[eschwartz]

Originally Posted by unboggling

Does like to like conversion update the metadata in the input format and the output format? Or only in the output format, which is what I'd assumed?

Only output. The input is saved as book of type ORIGINAL_FORMAT.

[unboggling]

Originally Posted by enonod

I am presuming that Save to Disk is making a backup to another disk (I don't know if it is simply the books or the whole folder for an author or the whole library.

Calibre stores the book files in a library folder, which generally is hands off. If you want one or more books to be in a different folder organized in a different way, use Save Books to Disk to save (copy) all of the selected books in the calibre library-view booklist out to wherever you want them. Same disk, other disk, anywhere except inside the hands-off calibre library folder.

The term "save to disk" is a bit misleading. Calibre automatically updates all the changes you make to metadata very soon after you make the changes, automatically updating database.db (as well as the OPF). [Edit] Calibre automatically "saves" formats to the relevant book folder in the calibre library folder, whenever user adds or removes a format through calibre interface.[/Edit]

[unboggling]

Originally Posted by BetterRed

Only the output file, but that can overwrite the input file if the settings in the save_original_format tweak are set to False.

BR

Originally Posted by eschwartz

Only output. The input is saved as book of type ORIGINAL_FORMAT.

Thanks. What I thought.

[BetterRed]

@enonod - The Save to Disk feature is not intended to be used as a means to take library backups. It is primarily provided to allow users to copy the format files into a different folder structure with different file naming rules. It is possible to reconstruct a library from those folders and files via Add Books, however this could result in metadata loss.

Calibre does not have a facility to backup and restore libraries, I suggest using the same process as is used for other data - documents, emails, images, music etc.

I do an incremental backup of all my data every day to an external drive and once a week to commercial FTP service. The daily backup rarely takes more than 5 minutes, mostly two or three, the weekly backup typically takes about an hour. I normally run the backup within a shutdown script.

BR

[enonod]

Thank you to everyone who responded and provided me (and some participants) with valuable information.
I felt that one question was not clearly answered until the final response from BetterRed.

in 1. I wanted to know whether the book was updated only in the dispatched book, or the original as well.
BetterRed's answer seems to be indicated by the fact that books would need to be imported back to get the changes registered.

In 4, as a result of that conclusion, I wanted to know why the change is made in the copy departing and leaves the original. Why not change the original and then dispatch a copy.

I understand the 'hands off' aspect of the library, but with photography the same type of library can be used but the metadata (keywords or categories) are usually saved direct to the photographs as well as the database. In that way it is always safe to assume that 'any' database could be constructed from the metadata contained in the images and a specific database is not required. At the instant the keywords are added to the database, they are written to the image and it takes a fraction of a second.
If a 'session' of key-wording takes place then the user equally suffers the same time penalty mentioned in the first reply.
I note that if an author is changed the (sometimes very large) books are moved, which is no less time consuming and makes the folders inviolable. Why is that not considered equally bad to updating metadata? In fact a faster operation in itself.

This is not criticism but a genuine attempt at fully understanding.

I wonder how many cries for help there have been in this forum because somebody 'made a mistake' or lost their 'presence of mind' or moved several books because they were doing some disk tidying etc. etc. and did not remember/realise that it would screw their Calibre, simply because that was farthest from their mind.

Question 5.
When the library is moved, by invoking the wizard, does the new library have the metadata written to books?
I am unable to tell, through lack of knowledge of book metadata. I did this and noted that opf files were left in the old library, along with any non essential files.

[DoctorOhh]

Originally Posted by enonod

in 1. I wanted to know whether the book was updated only in the dispatched book, or the original as well.
BetterRed's answer seems to be indicated by the fact that books would need to be imported back to get the changes registered.

True, Unless you do a conversion or go out of your way using Polish or Modify Epub plugin the metadata in books are updated upon export from the calibre library. It is explained well in post 3.

Originally Posted by enonod

In 4, as a result of that conclusion, I wanted to know why the change is made in the copy departing and leaves the original. Why not change the original and then dispatch a copy.

Only Kovid can answer this question without speculation.

Originally Posted by enonod

I note that if an author is changed the (sometimes very large) books are moved, which is no less time consuming and makes the folders inviolable.

Are the books moved or is the folder simply renamed? I'm guessing the folders are simply renamed by the OS. If that OS process requires a "move" it is still a single entry in the allocation table not actually moving a large file. Either way these are a blink compared to expanding a book writing to it and re-compressing it.

Originally Posted by enonod

When the library is moved, by invoking the wizard, does the new library have the metadata written to books?

No.

Originally Posted by enonod

I did this and noted that opf files were left in the old library

Keep in mind the old opf files have nothing to do with calibre storing metadata for books. These files are used to restore the database in case of database corruption.

[BetterRed]

Originally Posted by enonod

I understand the 'hands off' aspect of the library, but with photography the same type of library can be used but the metadata (keywords or categories) are usually saved direct to the photographs as well as the database. In that way it is always safe to assume that 'any' database could be constructed from the metadata contained in the images and a specific database is not required. At the instant the keywords are added to the database, they are written to the image and it takes a fraction of a second.

a) All the professional and many consumer image management packages I know of store the metadata in a database and optionally in the image or sidecar files. Many pro's don't like the original files being polluted with extra stuff like ratings, categories, titles etc, i.e they want the RAW kept in the same state as it left the camera.

b) The metadata definitions for imaging are much more mature than those available for ebooks, mainly via the EXIF, IPTC and XMP 'standards'. I suspect this is partly due to most image creators being more intimately involved with their final product - than are writers are with their books, the latter have traditionally required intermediaries, editors, publishers, distributors etc. Also imaging has broader direct usage than books generally have - eg medical, forensics, military, aerospace, manufacturing, search and rescue, surveillance, intelligence etc etc.

Originally Posted by enonod

I note that if an author is changed the (sometimes very large) books are moved, which is no less time consuming and makes the folders inviolable.

With advanced file systems, eg NFS, HFS+, NTFS etc, when a folder (directory) is renamed (or moved within a device) it all happens within the file system index/directory/master file table structures. The contents of the folder are not moved, so a folder containing 200GB of data can be renamed just as fast as folder with 2KB can be. This is not true of primitive file systems such as FAT or BDOS.

Originally Posted by enonod

I wonder how many cries for help there have been in this forum because somebody 'made a mistake' or lost their 'presence of mind' or moved several books because they were doing some disk tidying etc. etc. and did not remember/realise that it would screw their Calibre, simply because that was farthest from their mind.

Yes that happens a bit, but given there are 3.5 million active users, perhaps its not as often as you might be imagining - http://status.calibre-ebook.com/.

BR