Correct publication dates or at least an "edition" option

[MustardBallon]

Hello folks! I JUST sent this message to wikidude, but I might as well put it out on the public to see if anyone else has some words of wisdom... (thanks for reading!)

The set up: I'm very new to Calibre, downloaded it today in the AM & been messing w it all day long. I have a very large number of science textbooks (mostly PDF & DjVu) and most of them are the nth edition of the original. I need to be able to organize them in the correct chronological order (due to outdated scientific data and such)

The problem: I've been checking every option available in Calibre and in [wikidude's] plugins and I can't find a way to get the correct metadata nor an option to include the "edition" of the textbook! (very frustrating!)

The question: IS there a way to include the edition on a book's metadata? If so, is there a way to sync it from an online source? If the edition can't be included I could use the "published date" instead. HOWEVER, the dates that I'm getting back from the metadata search (even straight from the ISBN) are completely off! I'm assuming the ones I'm getting are the date for the original (1st ed.) publication (?)

The plead: Could [anyone] please shine some light on this issue. I am more than willing to put in the hard work (I was even considering looking at python code to see if I could do it myself!). I've been looking everywhere and I think I'm just missing something, an option I didn't check or a menu that needs revising... I don't know

ANY guidance would be immensely appreciated!! THANKS!!

[unboggling]

I handle edition info by manually appending it to the title (3rd Ed.) and manually putting correct edition date in the Published date field. Another possibility is to use a custom column for edition number, and one for edition date. Metadata Download date data and edition number often have to be corrected manually, which for accuracy usually means manually checking title page or copyright page inside that particular book format.

EDIT. There are several problems.
1. The data at the Metadata Download source may be inaccurate.
2. Varying Metadata Sources seem to handle edition data differently or not at all.
3. So calibre determines what date to select from all the metadata source returned data. Even if programming for that determination could be improved a little, that wouldn't solve the first two problems and the data would have to be checked and corrected manually anyway.
4. Possibly someone could write a plugin to extract the correct edition date and edition number from within the format. But edition data isn't presented in a standard way across all publishers and all languages and all book formats. So writing it wouldn't be easy and would probably take a lot of work.

[Divingduck]

I use as well a custom column. This is maybe a good way for those who have to handle a huge amount of publications.
Quality of Metadata is a problem. I check each manually to be sure to have the correct information in my library.

[kiwidude]

@MustardBalloon - there is no way to get this data without writing a bespoke plugin. The metadata download infrastructure does not support custom columns, so even if a plugin did scrape exactly the values you want it cannot store them anywhere except in a standard calibre field. And the problem with that is calibre does a collation process involving ISBN pools, Worldcat, and looking up first published dates etc which all combine to guarantee that no matter what value a metadata plugin might scrape for the likes of publication date it won't be that value on the book at the end of the process.

If the publication date of the edition is in the metadata for the PDF, then that can be grabbed by calibre at the time you add the book or via the edit metadata screen. You could then turn off the publication date column in the metadata download options, so that it won't get overwritten. Or just populate it manually.

Anything else like (xth Edition) you can store either in the comments, tags, a custom column or title if you prefer it as others have said. But you will need to do it manually.

[MustardBallon]

Thank you so much for the quick responses!

As I feared... it seems like I'm gonna have to do it manually.

I'm so surprised by the lack of metadata standardization across sources and publishing houses. With the resent proliferation of e-readers and places to buy the books themselves, it would seem like companies/publishers would want to get together and agree on some sort of standard; if not for consumer benefit, then perhaps to ease their own process of book conversion to various formats.

I'll probably use the "published date" as my main tool to ensure correct chronology. I'm gonna try to find the best source of that info from the list of "metadata sources" and turn off that option on all the others. And then, of course, check it manually... (seems doable, no?)

Any suggestions on the best source of "published date"? I'm thinking Amazon, since it sells different editions and must differentiate them somehow...

Thank you!

[kiwidude]

As I said above you cannot use metadata plugins to get published date, it is unreliable because of Worldcat overriding whatever value a plugin returns.

[MustardBallon]

Gotcha! I hadn't realized that Worldcat is NOT a plugin/option that you can manually adjust. I figured I'd at least get most of the dates right if I found a good source (& adjusted the settings just right) and have to change only some.

So what IS the point of giving us the option to check/uncheck whether we wanna download the "published date" from a certain source if Calibre is just gonna do their own collation process... I understand the whole concept of "Utilitarianism" (do the most good for the most people), which is what their collation process probably does, but it'd be nice to have the option to turn that off for the more "OCD" people who want to perform a much more customized process...

I had already accepted that I wasn't gonna be able to automate the process... but I was hoping to get at least 80% of the way there

[kiwidude]

Well only Kovid can say for certain, but I can submit a theory for you, and Kovid can shoot me down for talking nonsense.

IIRC before calibre 0.8 calibre did bring back the actual published date of an edition from the website. Unlike your needs however, I would suggest the vast majority of users are storing fiction/non-fiction novels and are more interested that a book was originally written in 1989, than that the particular edition they have was published in 2010. Certainly I fall into that category - I want to read books in the order the author wrote them, not caring in the slightest what edition I have.

However of the metadata sources that calibre pulls from "out of the box" (Amazon, Google and a few others), I don't believe any of them at the time certainly supported a "first published" date (unlike some of the plugins I have written like Goodreads and FantasticFiction). So when Kovid rewrote all the metadata download stuff for 0.8 he made the decision to instead hard-bake it into the calibre code to do a lookup based on ISBN against Worldcat as a source of first published dates.

Unfortunately there are two problems with this. The first is that people who want an edition date instead of first published date (like you in this case) are totally out of luck. The second problem is that Worldcat is an unreliable source, and from my experience gives me some very bizarre dates at times in comparison to what FF/Goodreads display. That is not to say that the latter have perfect data for every book, but right now it is immaterial what dates they have because calibre ignores their results in favour of Worldcat anyway.

You ask what is the point of the option for overwriting published date - the point is it allows you to either get "a" first published date (from Worldcat) or not get one at all and hence not overwrite whatever value you manually put in the column.

Kovid has said he will accept a patch to alter this logic but it isn't a priority for him to make the change himself. I started looking into it a while ago, but I'm afraid the complexity of the code managing the collation makes it a non-trivial change unless you can work out exactly how and when all the magic foo that is in there fires. It fell into the too hard basket and so I continue to manually correct published dates using values from FF (I have the search the internet plugin configured with a keyboard shortcut to open FF for an author). Perhaps one day when my plugins splurge reduces again I will revisit it, or maybe Kovid will take pity on us and fix it while he waits for a build like he did with another "issue" that had been lurking around for years a few weeks ago