Importing books - getting all metadata

[parkher]

Sorry for a newbie question:

I am importing books into Calibre

Most of them have metadata information, but some do not.
Then they are imported as Unknown author, Unknown Title, and even their files renamed to unknown.
(yes, I know, the list of such books is given).


In import settings I see this

- Read metadata from file contents rather than file name.

Unfortunately, a bit different option is needed:

- Read metadata from file name only when metada from file content is not available

BTW, even better would be to use this option:
- get from file names the metadata fields that are not present in file content.

Or perhaps, if metadata in file content is often wrong or in wrong fields, the opposite option would be useful - to give priority to file names, but take other stuff from file content.

But I can find no such settings.


Anyway, I tried to get metadata from file content.
Unfortunately the ready-made regular expressions do not even cover the most popular e-book file naming convention.

So I made an improved regular expression, maybe it will be useful for others too:

(?P<author>[^_-]+) -?\s*(\[|\()?\s*(?P<series>[^_0-9\[\(-]*)(?P<series_index>[0-9]*)\s*(\]|\))?\s*-\s*(?P<title>[^_\[\(]+)\s*\[?\s*(?P<publisher>[^\]]+)\s* ?

It handles correctly most of the books that adhere to the naming conventions, even allows quite significant variety.
But there are cases that it does not cover yet:

1. when a range is given in a series:

Some Author - [Some Series 01-05] - Some Title Boxed Set (epub).rar

I don't know what to do here: series number can be only a single number,
perhaps not to enter any number and to try to move 01-05 to the end of the title?

2. when the name of a series begins with a number (this is a fake example):

Angel Devil - [666 Devil Street 01] - Devil Moves In (mobi).mobi

That is a serious complication.
Possible solution - do not support this format:

author - series nr - title
but only these:

author - [series nr] - title
author - (series nr) - title

That probably would easily allow to have series name starting with a number.
And besides, the naming convention calls for [series name] in brackets anyway.
Still, this expression covers most of everything else.

Some possible publisher info or publisher series numbers moved to publisher.
At least saved somewhere.
Again, a completely fake example:

Reeva Shmadams - The Smashing Dreamcrasher [HT-429, NHK-824] (v1.1) (epub).rar

So, in this example, I do this:

Publisher: HT-429, NHK-824

It could be changed to use a different field, but which one?
Because these are usually publisher's series with their numbers Publisher perhaps is OK.
Just to preserve this info somewhere.
Besides, sometimes there is only publisher's name in [] - so Publisher fits perfectly.

Anyway, maybe somebody would like to improve this regular expression further, or could offer suggestions what to do in those complicated cases.
Or can find other unsupported cases.

Any suggestions welcome

[parkher]

Other questions:

1. How to extract, inspect and possibly add metadata from the file content and/or from the file name (not from the internet) to a book which is already imported?


2. How to pause an import operation? I see only "Abort" button.
I suppose that an import operation may take days to complete, pausing it may be necessary for various reasons.

[theducks]

Originally Posted by parkher

Other questions:

1. How to extract, inspect and possibly add metadata from the file content and/or from the file name (not from the internet) to a book which is already imported?


2. How to pause an import operation? I see only "Abort" button.
I suppose that an import operation may take days to complete, pausing it may be necessary for various reasons.

1: Done is Done. The filename (within the Library) is no longer what it was

2: You don't.. Do smaller batches.

[parkher]

Originally Posted by theducks

1: Done is Done. The filename (within the Library) is no longer what it was

Yes, I see now that file names are changed - the author goes to the end, etc.
So in that case, it makes sense to try to get metadata from the file name while importing - while you still can - and that makes even more important to be able to add metada from the file contents later.
A book may contain some useful metadata information entered manually that is not available on the internet.

Or the initial internal metadata information is also lost and no longer present in the added book?
I mean, there is metadata.opf with new metadata, but there is also, for example, an epub file - maybe it still contains the initial metadata?
Or more than one file each with its own metadata still intact?

If not, that makes even more important to have an option to import metadata both from the file name and from the file content, maybe choosing priority to one of those sources.

But if the original metadata is still in the file, I should be able to fetch it and transfer fully, or partially to the library book metadata (metadata.opf)

I hope I don't have to import each file for the second time just to achieve that and then to delete the copies.

[theducks]

Calibre does not change internal metadata ON IMPORT

Conversion (to the same format) and a number of other tools Can update internal metadata. The blue I on the Edit Metadata form will retrieve any internal metadata for the selected format ( This will overwrite what you see. use with care )

[parkher]

Thanks
In other words, I need to copy what I see before extracting metadata from a file.
It would be better if it was just shown without overriting (similarly as when gettnig from the internet) - then I could copy only what I need and cancel the rest.

Anyway, because I now realized that the initial file name is not preserved, I want to save all the information from the end of the file name, not just the publisher, such as version numbers etc.
So I modified the regular expression, it just takes everything (including multiple braces and or parentheses) and copies to the "publisher" field, which later can be cleaned up manually, if so desired.

(?P<author>[^_-]+) -?\s*(\[|\()?\s*(?P<series>[^_0-9\[\(-]*)(?P<series_index>[0-9]*)\s*(\]|\))?\s*-\s*(?P<title>[^_\[\(]+)\s*(?P<publisher>[^$]+)\s* ?

[Sabardeyn]

Parkher,
If you still have the original files, you could create a new library and re-add all of the books. You could even do so twice, once reading metadata inside the ebook and once again based on filenames. Then look for duplicates, keep the better book record, and merge or delete the extra. Neither method is perfect, errors will occur. The suggestion to add small groups of books allows you to notice and correct any problems quickly.

You can also download metadata from the internet, but a warning: downloaded data overwrites existing field data. It's best (at first) to use this in a limited and per book fashion.

Regardless, you will probably want to manually tweak the information in your book records to reflect your personal needs.

Each user's needs and methodology differ, but essentially we're all doing something similar when it comes to managing our ebooks through calibre. This effort is made easier by new features to calibre, and its plugins, the programmers are able to devise.

[parkher]

Thanks, Sabardeyn
Yes, adding two times is a salution.

As to getting metadata from the internet, it is possible to get it, to inspect it, to copy to the clipboard and to discard, but to paste from the clipboard what you want.

------- BUG REPORT !!! -----------------

I think I found a bug in Calibre:

My settings for adding books:

v Automerge added books if they already exist in the Calibre library: Create new record for each duplicate format

So, if I add two books at once (let's say epub and mobi) - I get one line and metadata shows boths files there.

And this message is shown:

"The following 1 duplicate books were found and incoming book formats were processed and merged into your Calibre database according to your automerge settings"


If I add those two books one at a time, I get two lines with just one book in the corresponding metadata of each line.

And I get EXACTLY the same message:

"The following 1 duplicate books were found and incoming book formats were processed and merged into your Calibre database according to your automerge settings"

But clearly, this time the books were not merged !
I still have to merge them manually

Because by merging duplicate books the metadata, derived from the file name of each, is not preserved, and because it is not practical to add all the books at once (too many days and no Pause button) - I think I will switch that automerge feature off anyway.

[parkher]

-------------- Inconsistency in handling rar files ------

Previously I noticed that when I add a rar file, epub, mobi, etc. files are extracted,
but html, txt - rar file remains.
Now, when adding both epub and mobi - I see that epub is also not extracted.
Which is strange, because there is no point in keeping zip inside rar, while it perhaps would be useful to keep mobi rarred.
Then, experimenting with adding a bookt at a time, I sometimes get mobi not extracted too. No consistency. Even with the same files, does not depend on the file name.

And, turns out, rar is no good - I cannot extract metadata from rar or to open rar in the viewer or to convert to another format.

Yet, clearly Calibre supports rar format, it is listed and it is SOMETIMES extracted.

Seems like another bug?

Yes, rar is no longer extracted if inside is epub, but extracted if inside is mobi. It seems.

Maybe because I deleted all the books multiple times and db got corrupted somehow?
Deleted db - still rar.


UPDATE:

I tried 32-bit Calibre: it imports epub from rar correctly - I get epub, not rar.
When importing html, still useless rar. But it is possible to convert it later to epub.
So, I guess, it is ok then.

But that is 32-bit Calibre. Because it is acting differently than the main 64-bit Calibre, it only confirms that it is another bug, I think.

So, BUG REPORT #2 - see above.

[speakingtohe]

Originally Posted by parkher

Thanks, Sabardeyn
Yes, adding two times is a salution.

As to getting metadata from the internet, it is possible to get it, to inspect it, to copy to the clipboard and to discard, but to paste from the clipboard what you want.

------- BUG REPORT !!! -----------------

I think I found a bug in Calibre:

My settings for adding books:

v Automerge added books if they already exist in the Calibre library: Create new record for each duplicate format

So, if I add two books at once (let's say epub and mobi) - I get one line and metadata shows boths files there.

And this message is shown:

"The following 1 duplicate books were found and incoming book formats were processed and merged into your Calibre database according to your automerge settings"


If I add those two books one at a time, I get two lines with just one book in the corresponding metadata of each line.

And I get EXACTLY the same message:

"The following 1 duplicate books were found and incoming book formats were processed and merged into your Calibre database according to your automerge settings"

But clearly, this time the books were not merged !
I still have to merge them manually

Because by merging duplicate books the metadata, derived from the file name of each, is not preserved, and because it is not practical to add all the books at once (too many days and no Pause button) - I think I will switch that automerge feature off anyway.

Not a bug, just unclear documentation possibly. It is merging them according to your automerge settings. (second example)

First example what is done with the books merge-wise depends on how you add them and overrides the automerge setting.

Helen

[speakingtohe]

@Parkher

When I bulk download metadata I do only metadata so I can quickly see if the names match the cover. Often the new metadata is far removed from the original book Then I check them and download the ones that are wrong individually.

If I am doing a lot of books I import them to a temporary library.
Next I copy to main library,switch back to main library and download metadata.
Then I use the find duplicates plugin to find library duplicates in temp library.
Next I type Not in the search bar before marked:library_duplicate.
This gives me a shorter list of possible problem files

[parkher]

Thanks!
As I began to import to a completely empty library, I can consider it is a temporary one

[parkher]

Turns out, the previous regular expression does not handle fractional series numbers.
Such as:

Budung Kuleeeee - [Silly Horror 2.5] - Horror Horror Hor. epub

Also a fix so that the last letter of the title is not moved to Publisher in the case when the title is the very last part of the file name.

Here is the fixed one:

(?P<author>[^_-]+) -?\s*(\[|\()?\s*(?P<series>[^_0-9\[\(-]*)(?P<series_index>[0-9.]*)\s*(\]|\))?\s*-\s*(?P<title>[^_\[\(]+)\s*(?P<publisher>[^$]*)\s* ?


Still currently not supported:

when the name of the series itself contains a number followed by the series number:

Some Author - [Series Name Contains Number 21 02] - Title.rar

when the series name and numer are separated by a dash, so we have already three dashes:

Some Author - Some Series - 16 - Some Title.rar

[DoctorOhh]

Originally Posted by parkher

Turns out, the previous regular expression does not handle fractional series numbers.
...~~~...
Some Author - Some Series - 16 - Some Title.rar

It would be rare to have one expression handle all possible Add books scenarios. You might want to use the Quick preferences plugin. This plugin allows you to quickly switch between Add books expressions.

Also you might want to review this thread for some insight into the limitations of using RAR files.

So far I think your bug reports are features, but you can find out how to submit an official bug report here.

[parkher]

DoctorOhh,
Thanks for the links

The thing is, I am trying to add folders of books at once. Not one-by-one - that is not practical. So I am trying to come up with the regular expression which covers most of my books.

It is interesting to look over also what other plugins are available and what they do anyway.

As to my "bug reports" - the second one (epub not extracted from rar) is probably a real bug, because it occurs only with the 64-bit version of eclipse, with the 32-bit version - the result is different and correct.

As to the first bug (merging duplicate books when adding new folders of books) - the books are merged when they are in the same folder and are added together, but not merged when added later.
However, when adding later, such merging might be pretty dangerous anyway - some later added books might have very different metada, etc. So I don't know if I would use it anyway. Especially that I am adding older book versions later...