calibre - Not using existing metadata from *.opf files when Add books

[rolandt99]

Hello,

I am a relatively new user of calibre and have run into a problem caused by what seems to me to be an inconsistentancy in the way calibre does some actions.

I am using calibre 0.9.0 on Windows.

Due to some external issues I needed to reload a bunch of books into my calibre library. These had all been previously saved elsewhere using "Save to Disk"/"save to disk in a single directory". So all of the books along with the matching <cover>.jpg and <metadata>.opf files were in the saved data location.

First I attempted to re-Add these books using "Add books"/"Add books from directories, including sub-directories (Multiple books per directory, assumes every ebook file is a different book)". Since the calibre documentation states this is the "inverse of the Save to disk action, i.e. you can Save to disk, delete the books and re-add them with no lost information except for the date", I thought that all of the books would be added with the correct associated cover/metadata. That is NOT what I got. Books that are epub format do have 'correct' over/metadata - but that is only because the save to disk process updated the actual epub file. Books that are pdf format do not have the correct cover/metadata - again because the save to disk process did NOT update the actual pdf file. So the "multiple per directory" option seems to ignore cover/metadata files and just processes ebook files.

I then tried to re-Add a few of these books using "Add books"/"Add books from directories, including sub-directories (One book per directory, assumes every ebook file is the same book in a different format)". The calibre documentation states this is also the "inverse of the Save to disk action". So I took a few ebooks and created a directory tree with each book in a separate directory. This time it worked correctly. All books, including pdf format, now have 'correct' cover/metadata.




So my question is how can I add books including cover/metadata without having to place them into individual separate directories?

And, if I HAVE to use individual separate directories then will someone please tell me a quick simple straight forward process to move thousands of books (and related files) from a "single" directory structure to thousands of subdirectories.

thank you in advance for any assistance

[kovidgoyal]

http://bazaar.launchpad.net/~kovid/c...revision/13429

[rolandt99]

Thank you Kovid, for the fix.

However, since I am not a developer type to recompile calibre, I will have to wait until next week for release 0.9.2 (which I am guessing will include this).

[rolandt99]

Hello - this problem has resurfaced again. I have tried with calibre 1.22 and 1.25.

PDF books, even though "Save to Disk" from calibre; do NOT get added with the correct (updated) metadata when using the Multiple Books per Directory option.

Also, a secondary and separate issue. Based on all of the various postings I could find here, I expected basic PDF metadata (i.e. title and author) to be updated when the PDF format is "Saved to Disk". However, that is not happening. The PDF file is completely unchanged from the external orignal. So the ONLY place that has the correct metadata is the .opf file. Even these PDF books written to an ereader have the old/incorrect title and author info.

[Adoby]

Calibre attempts to update basic metadata. However many PDFs are created in a way that means that this fail.

However when adding, the opf-file should be used to set metadata in calibre. There is an option to read metadata from the filename. Make sure that you don't have this activated, and try again. Also you obviously need to have PDFs together with their individual opf-files. Perhaps you have turned saving opf-files off?

Please also note that PDF isn't the best format to use if you want to make full use of calibre. PDFs usually aren't meant to be changed or updated. Epub, for instance, would be a better choice.

[BetterRed]

@rolandt99 - re: Save to disk - is the box ticked, see attachment

I fell into that hole too

BR

[rolandt99]

Adoby, thanks for a reply. However -
1) The option "read metadata from file contents rather than file name" is checked. AND, the 'invalid/old' metadata being propogated is NOT in the filename. If I were extracting from filename then that small part would be correct ;-) Which does nothing for all of the other metadata and cover, etc.......
1.1) Since there is no explicit option for metadata.opf - Presumably, this "read metadata from file contents" option actually means "if a metadata.opf exists, use it; otherwise use metadata from file contents."

2) I stated the problem has resurfaced. The problem was (and is AGAIN) that adding "Multiples" does NOT process metadata.opf files where as adding "singles" does process metadata.opf files. So... yes that statement, explicitly means, I have the metadata.opf files with the book files with the cover files. They (both metadata.opf and cover.jpg) are just ignored (bypassed) in "add multiple" mode.

3) I agree with you, EPUB is a more versatile and usable ebook format. However, a significant number of books are only available in PDF format (roughly 20% of my library). Again, as I stated I am not attempting to "update" PDF files - other than what calibre documentation states it does - update basic metadata (i.e. title & author) when "saved to disk". I have checked hundreds and absolutely NONE of the PDF files "saved to disk" have any updated metadata.


Thanks BetterRed for your response. Checking that "update metadata in saved copies" setting was one of the first things I did. Metadata is updated in EPUB books saved to disk, metadata is NOT updated in PDF books (none, even basic title/author) saved to disk. But, even so the main "Problem" I have re-raised here is as I stated in 2) above.

kind regards

[kovidgoyal]

As of 1.25 *all* metadata (except cover) is saved to PDF files. If metadata in your PDF files is not being updated, you most likely have either disabled the PDF metadata writer plugin in Preferences->Plugins, or the PDF files you are working with are somehow broken, so metadata update is failing. If it is the latter you can check by running calibre in debug mode and doing a save to disk.

As for adding books I cannot reproduce it, steps I tried:

1) Add a .txt file to calibre
2) Change the title to "Test title" author to "Something else" and series to "test series [3]"
3) Click save to disk. txt file alongwith opf is saved in a folder named else, Something/Test title
4) Click add books, multiple books per folder and choose the folder else, Something
5) Book is added with correct title, author and series

[rolandt99]

Kovid, thank you for your response. And, also, thank for this fantastice software suite.

I followed your example - and yes, the metadata for the text file is imported with the text book file. However, I can reproduce the problem --- when I place one of my PDF books (book + cover + metadata) (pdf has internal old/outdated metadata) in the same "saved" directory tree and then add "multiple". The text book metadata is correct, the pdf book has only the old/invalid metadata. When, I do the same thing using add "single" both the text and pdf books have the correct/updated metadata. So, it appears that for "multiple" mode - if the book has metadata the OPF is ignored, if the book does NOT have metadata the OPF is used.

Further - it appears that the updated PDF "save" process in 1.25 does work better than previous versions. The books I had been testing with had all been saved with v1.23 or older. I just -resaved one of the PDF books and the basic internal metadata (title/author/series) has ben updated this time.

However, the add "multiple" issue does exist. When I add the newly updated PDF using "multiple" the cover is the internal (outdated) cover. When I add the newly updated PDF using "single" the cover is the "saved" cover.

thank you

[kovidgoyal]

When there is metadata in the file it will overwrite what is in the opf. If you want to use the OPF preferentially disable reading metadata from file contents.

[rolandt99]

Note - all of this testing is with calibre 1.25. I just re-tested for this posting.


Your statement "metadata in the file it will overwrite what is in the opf" does not match my results.
When I "add single book per directory" ALL metadata comes from the opf file.
<invalid/outdated information IS in book file but is NOT used>, cover used is the saved one.
When I "add multiple books per directory" NO metadata comes from the opf file.
<invalid/outdated information IS in book file and IS used>, cover used is the internal one.
In both scenarios, I am adding the exact same set of books (book, opf, cover).

Also, I see no option to preferentially use OPF. I see just one binary option - "read metadata from file contents rather than file name. Normally I have that option 'checked' - ie read from file contents not filename.

with it checked - the above mentioned results.
adding "one book per directory" - works, all metadata/cover are current
adding "multiple books per directory - does not work, metadata/cover are old

I just unchecked that option -
adding "one book per directory" - works, all metadata/cover are current
adding "multiple books per directory - does not work, no metadata of any sort, no cover, title is filename.


SO, I reiterate - there is some process error difference when selecting "add, multiple books per directory" as compared to selecting "add, one book per directory".

[kovidgoyal]

That makes no sense. metadata processing for all file types is the same. If you get the correct results for txt, you will get the correct results for pdf. Note that the opf file has to have the exact same filename as the pdf file (except for the extension).

[rolandt99]

The set of book files I used for my testing were created using "Save to Disk" with the following template pattern "{author_sort:sublist(0,1,&)}/{author_sort:sublist(0,1,&)}~{series}{series_index :0>2s| |}~{title}". Which places books for the same primary author in one sub-directory. And, yes the file set for a book are all named exactly the same (excepting .extension).

But, you keep over-looking the pointer to where there is a problem. Doing adds using the EXACT SAME set of book files ---
Using "Add from directories, ... (One book per directory...)" works as expected. You get correct (from .opf/cover files) metadata/cover.
Using "Add from directories, .... (Multiple books per directory...)" does not. You get incorrect (only from internal book file) metadata/cover.

To get correct (updated) metadata I have to place every book file set in individually separate subdirectories and then "Add... (One book per...) option.

As soon as I get time, I will do some screen captures to show you that one way works and the other way does not work.

[kovidgoyal]

Originally Posted by rolandt99

Using "Add from directories, .... (Multiple books per directory...)" does not. You get incorrect (only from internal book file) metadata/cover.

I just demonstrated for you, using a txt file, and you agreed, that adding multiple books per directory does read metadata from the opf file.

Futhermore, writing metadata to PDF files has always updated title, author and keywords fields, so I dont see how its possible you have PDF files that were saved with old metadata, unless you explicitly turned off updating metadata. What was changed in 1.25 was that *additional* metadata was written to PDF files as XMP metadata, the writing of the fields in the info dictionary was not changed.

So, again, I cannot reproduce anything you claim. Posting screenshots is pointless. zip up a directory that contains the files you are failing to add and attach that.

I attach here a zip file containing two books in txt format saved using save to disk into a single directory with their matching opf files and covers. Unzip that file into adirectory and add it using multiple books per directory and you will see that the metadata is correctly read for both, including the series, publisher, isbn, publish date and cover.

[rolandt99]

Yes, I acknowledged and agreed with you that creating/saving/adding a TXT format ebook seems to pickup the metadata from the OPF/cover. Apparently, because TXT format ebooks do NOT have any internal metadata to confuse the process.

Again, the issue here is NOT that PDF processing has or has not been changed. The issue that I see quite consistantly (with many add operations testing with several calibre versions) is that "Add Multiple" produces different results (uses old INTERNAL instead of OPF/cover metadata) than "Add Single" (uses OPF/cover metadata).


And, while I am doing that which you ask of me -- I ask you to please try some ebooks that have INTERNAL metadata that is different than the external OPF/cover and you might possibly see the issue. Using PDF ebooks just happens to be an easy format to use to demonstrate the difference between "Add Single" and "Add Multiple". Since, for PDF ebooks, even if other metadata is (now) updated on "Save to Disk" the internal cover is not. So the PDF internal cover can easily be different than the external (cover.jpg) cover.