Metadata extract from Title

[507Tuli]

Hi,
Well I have finally given up trying to figure the regular expressions out. Hopefully someone can help.
I have renamed all my book filenames as

Author - Title

All I am trying to figure out is how to write this as a regular expression.

Sometimes I do have a series in the middle

Author - Series - Title

but I want to store it so that everything after the first dash is part of the title, just to keep it simple for me.

Thanks for your help!

[Format C:]

(?P<authors>.+) - (?P<title>.+)

Should work for both.
(I'm not sure about the plural "authors", you can try it without the "s" if it does not work...)

[kad032000]

When I try that solution, I get

Title = Title
Authors = Author - Series

Rather than the desired

Title = Title - Series
Authors = Author

(At least that's what the test data shows.)

[Sabardeyn]

Kad,

The regular expression that was suggested is operating correctly. Unfortunately, the results it is generating and what you wanted are not the same thing. The inconsistent data set (author - title vs author - series - title) creates a major problem in trying to get calibre to correctly import your books accurately.

Off the top of my head the easiest way to correct for this is simply to import your books in two different sets. If you split your books into Series or No Series groups, changed the expression accordingly, you would be fine. While this sounds wonderful as a theory, I'm sure that several of the authors have books that follow both file naming formats. So this would be a major hassle to separate them. Or to add groups of books at a time matching whichever regular expression (regex) you're using. Not to mention we might be talking about doing all this on hundreds of books.

What you need is a regex that determines whether the filename currently being tested contains two or three fields. The first field is always the author(s)/editor(s) name, so grabbing that straight off should be fine. But the next field is either the series or the title. If their is a way to determine if " - " occurs again in the filename, then you can assume that what lies between the first and second " - " literal string is the series (more likely, series & series index). Otherwise, anything remaining automatically becomes the title.

While this seems straight forward enough, potential issues remain. Hyphenated author names or titles, if they specifically contain " - ", will cause the import to be performed incorrectly for that individual book. Of course, with 99% of them entered automatically, you might find the remainder acceptable for manual entry / editing.

While I know what needs to be done, I'm in the same position you are, simply too much of a regex & calibre neophyte to generate something this complex.

Another website I came across has a Regular Expression Tutorial; I've listed it here just in case you might find it helpful. Keep in mind calibre uses the python "flavor" of regex. (The calibre referenced site should always take precedence.)

[Sabardeyn]

It occurred to me that you cannot be the only user with author - [series -] title filenames. I've seen too many references that follow that exact format in the forums. So I decided to do a search...

I found this topic, Regular Expression Help, which specifically addresses your exact needs. It even accounts for the Series Index. The reliance on " - " to separate fields was removed. This means hyphenated names, or titles with dashes, should not be a problem when importing books.

Of course, that doesn't mean you won't still have some issues using it. Other potential problem areas are:

• authors listed as surname, firstname;
• multiple authors per book;
• unicode characters (I believe calibre strips them out);
• symbols and numbers used in filenames ("&" between two authors, "leetspeak", etc)

I'm not sure about all of these, just guessing based on what I can understand of all the gobbledygook in the regex. And I know that something will come along and "break" the expression simply because it was not designed to account for every possible situation in every possible language. So a review of all automated imported books would still be in order. But it should do the trick on most of them!

Keep in mind, if metadata is defined within the files, it will take precedence over the regex filename info unless you uncheck Preferences>General>Read metadata from files.

[gwynevans]

This is what I use

(?P<author>[^-]+) - ((?P<series>.+) (?P<series_index>\d+) - )?(?P<title>.+)

[Sabardeyn]

Ouch! Looking at your regex and testing it against the formula from the other topic, I've discovered neither one is perfect.

So far all of the formulas are dependent on " - " being used as a field delimiter only. It cannot be used for hyphenated author's names nor as a part of the series (unlikely) or title (where it is likely to occur). When an extra " - " occurs the automatic import fails as the parts of the filename are separated incorrectly.

So, for instance, this example fails in all formulas to import correctly:

John D. Smith - Jones - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf

This can obviously be corrected manually before (change to "Smith-Jones", etc) or after (edit the book's record). Let me be the first to admit that I would rather have things entered accurately the first time, automagically. Editing is a hassle and easily forgotten. Luckily for me I'm already using "Smith-Jones" (no space) for hyphenated names. However, their is no good way around the potential problem in the title.

For my purposes I would prefer a regex that resolves all of the following correctly:

• John D. Smith - The Perfect Book.pdf
• John D. Smith - Bibliographic Perfection - The Perfect Book.pdf
• John D. Smith - Bibliographic Perfection 1 - The Perfect Book.pdf
• John D. Smith - Bibliographic Perfection 189 - The Perfect Book.pdf
• John D. Smith - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf
• John D. Smith - Jones - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf
• John D. Smith-Jones & Somebody Else - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf

It should also handle author names that are 133t5p34|< (leetspeak), numbers and/or symbols. Why? Because we're already headed down that path and I might as well get a jump on things. Unicode import and export would be good too - more books are being sold internationally and this trend will only grow. (I don't want much, do I?)

[darkmonk]

This is what I use, I asked around some unix forums to get the answer. I use the same dataformat. My regex does not, however, take the series number.

(?P<author>((?!\s-\s).)*)\s-(?:\s(?P<series>((?!\s-s).)*)\s-)?\s(?P<title>.*)

EDIT: I just took the regex from the link, and while it wouldn't work as is, after removing the last "?", it worked with all my books.

(?P<author>[^_-]+) -?\s*(?P<series>[^_0-9-]*)(?P<series_index>[0-9]*)\s*-\s*(?P<title>[^_].+)

The only think I'd like tho change in that regex would be to make there have to be a space before and after a "-" to use it as a delimiter.
Time to reimport my Library! Now, if only .6 was out so I could convert it into epub with perfection!

[ilovejedd]

Originally Posted by Sabardeyn

For my purposes I would prefer a regex that resolves all of the following correctly:

• John D. Smith - The Perfect Book.pdf
• John D. Smith - Bibliographic Perfection - The Perfect Book.pdf
• John D. Smith - Bibliographic Perfection 1 - The Perfect Book.pdf
• John D. Smith - Bibliographic Perfection 189 - The Perfect Book.pdf
• John D. Smith - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf
• John D. Smith-Jones & Somebody Else - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf

These are easily done using the same regular expression.

Originally Posted by Sabardeyn

• John D. Smith - Jones - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf

This one's bit of a stretch. You can play with the greediness to get this one to play nice. Don't have Calibre on this PC but I can play with it when I get home.

[darkmonk]

• John D. Smith - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf
• John D. Smith - Jones - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf
• John D. Smith-Jones & Somebody Else - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf

From what I see, what you have there would require two series; the first taking precedence above the other. As far as I know, the epub standard does not allow for the further defining of series, but you can have custom fields - which calibre certainly does use. I wonder if we should allow a subordinate series - that, I think, would be a good idea, really. An author may create a world, and with that have many subordinate series - for example, star trek. What do you guys think?

[ilovejedd]

Originally Posted by darkmonk

• John D. Smith - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf
• John D. Smith - Jones - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf
• John D. Smith-Jones & Somebody Else - Bibliographic Perfection 1 - The Perfect Book - A Bedtime Story.pdf

From what I see, what you have there would require two series; the first taking precedence above the other. As far as I know, the epub standard does not allow for the further defining of series, but you can have custom fields - which calibre certainly does use. I wonder if we should allow a subordinate series - that, I think, would be a good idea, really. An author may create a world, and with that have many subordinate series - for example, star trek. What do you guys think?

I thought "The Perfect Book - A Bedtime Story" was the full title of the book while "Bibliographic Perfection" is the name of the series?

[darkmonk]

Originally Posted by ilovejedd

I thought "The Perfect Book - A Bedtime Story" was the full title of the book while "Bibliographic Perfection" is the name of the series?

Ah... why didn't I think of it that way???!
oh well; my thought remains, it would probably be good to have a subordinate series, so that books like this:
STAR TREK - TOS - 085 - My Brother's Keeper, Book One - Republic.pdf
STAR TREK - TOS - 086 - My Brother's Keeper, Book Two - Constitution.pdf
STAR TREK - TOS - 087 - My Brother's Keeper, Book Three - Enterprise.pdf
Would register as
Title: Author: Series: Series #: Subordinate Series: Subordinate Series #
Republic STAR TREK TOS 85 My Brother's Keeper 1
Constitution STAR TREK TOS 86 My Brother's Keeper 2
Enterprise STAR TREK TOS 87 My Brother's Keeper 3

^^ Evil lack of tables... There are many books that are in a mini series, in another series, or a world or some such.

[Sabardeyn]

Testing the various regular expressions, so far all of them fail at some aspect. Most notably dealing with either hyphenated names, or dashes in the series or title name. Not to mention extra spaces around dashes munging the whole thing.

Gwynevan's regex has been slightly more robust than other samples. My attempts to expand on it are not meeting with success; but I'm not a programmer or particularly good at regex. I'm trying to account for hyphenated names first as accurately importing the author is most important. I have avoided using \s to avoid later issues with making the formula work with leetspeak. But as that is a lesser concern I can forgo it.


darkmonk & ilovejedd,

My original attempt to post that sample book was clearly defined. But I deleted it as being unnecessary. Now I stand corrected:

• author = John D. Smith - Jones
• series = Bibliographic Perfection
• series index = # (any numeric value from 1-999; keep in mind the comic/manga readers)
• title = The Perfect Book - A Bedtime Story

I think the idea of a series and subordinate series is great. If I remember correctly Alan Dean Foster wrote in a Universe & Series (Humanx & Flinx, etc) as do some of the manga authors (called "circles" - authors write around other members stories [A uses B's characters, etc]). Universe/Setting/(Subordinate) Series, is feasible.

This begs the question of input vs output. I mean, I'm trying to input what I have whereas you're establishing a valid output pattern. Ultimately this is not a problem. But it occurs to me working on the same thing might be more effective!

[Sabardeyn]

Originally Posted by darkmonk

(?P<author>((?!\s-\s).)*)\s-(?:\s(?P<series>((?!\s-s).)*)\s-)?\s(?P<title>.*)

I noticed a typo in the regex. I believe the highlighted area should be "\s-\s". Right now you're looking for a literal text string of "any white space character-s" instead of the equivalent of " - " between the <series> and <title> fields. Just FYI in case you copied & pasted this from calibre.


Try this combined regex, it should handle almost everything:
(?P<author>((?!\s-\s).)*)\s-(?:\s((?P<series>.+) (?P<series_index>\d+)((?!\s-\s).)*)\s-)?\s(?P<title>.*)

Playing around with things, I managed to insert the series index portion of Gwynevan's regex into Darkmonk's regex. So far it meets all of the criteria I posted with the following exceptions:

• Extra " - "s within the <author> or <series name> fields mangles importation
• Hyphenated names like "Smith-Jones" are fine, "Smith - Jones" mangles importation.
• Leetspeak has limited importation - depending on the exact character combination used. (Which is fine, there is no way to account for every combination of letters and characters with Unicode.)
• Titles generally are not affected by the above since the regex allows all characters once the series name and index is obtained.

[Sabardeyn]

Hey, Darkmonk, you there? ... ... ...

Now that the effort to find a great filename regex seems complete, I had a few minutes to really look at your Star Trek post, above. The lack of tables definitely hurts my understanding of what you want. Nor did I understand what you meant by saying that calibre uses custom fields and that they can be used for sub-series despite ePUB not supporting such a feature.

Despite that, how about a scheme something like the following?

• Original Filename: STAR TREK - TOS - 085 - My Brother's Keeper, Book One - Republic.pdf
  • universe = Start Trek
  • setting = TOS (means The Original Series?)
  • setting index = 085
  • series = My Brother's Keeper
  • series index = 1
  • title = Republic

I used <universe> and <setting> instead of <subordinate series> because it seems appropriate. These specific words might not adhere to the "flavor" of other books though. I avoided re-using <series> because I'm not aware if it is a reserved word in the programming language sense, either by calibre or ePUB.

<Universe> would be the over-all series name including all derivative works. It would not have a <universe index> because it is simply a container for all of the Universe's constituent parts.

<Setting> is the exact branch, or portion, of the Universe that is being written about. In Star Trek terms, this would be TOS, TNG, DS9, VOY, ENT. (If I understand correctly how things are arranged in the Trek books. I haven't ever read any of them.) The <setting index> reflects the publication order of the various books within this Setting (as opposed to timeline / chronological order).

<Series> and <series index> follow calibre's standard fields. I assumed the more finite series name of "My Brother's Keeper" belonged here because series are not very long. Right now I think the Wheel of Time is the largest series. I don't count Trek, Buffy, Babylon 5, etc because not all of the books are concerned with a single storyline arch.

Dang! I was halfway through generating the formula and something broke the whole thing. Not sure, but think I might have exceeded the max number of variables for regex.