Incorrect published data in downloaded metadata

[ASpence]

I'm having trouble getting correct published dates when downloading metadata. The published date I end up with doesn't seem to match the date that is contained in much of the resulting information that the process provides.

I'm using calibre 0.8.15. I've restricted the configured download sources to use only data from amazon.com for simplicity. For this example, I'm trying to get metadata for "Core Java(TM) 2, Volume II--Advanced Features (7th Edition)", ISBN:0-13-111826-9, which was published in 2005 (as best as I can tell). I've also cleared the book title and author fields so that the search query is based on the ISBN only.

After clicking "download metada" on the "Edit Metadata" screen, I get the following dialog:



The first thing to notice is the published data on the first and only entry in the table which shows 2001. This is the date that is ultimately chosen for the published date after clicking through and accepting the results.

Next, if I investigate the links provided to the right after "See at:", I find that the first link is for http://amzn.com/0132354799. When following that link, I end up at http://www.amazon.com/dp/0132354799/ref=cm_sw_su_dp which is the page for "Core Java, Vol. 2: Advanced Features, 8th Edition". If you check on that page, you will see the listed published date is "Publisher: Prentice Hall; 8 edition (April 18, 2008)" This is not the book I'm looking for but it also does not contain the published date that is displayed in the table.

The second link is for http://www.worldcat.org/title/core-j.../oclc/57378709 which contains the correct information for the book and a published date of 2005.

To conclude the mystery here is the log data from the search query:

Starting download 
Query: title:Unknown; authors:Unknown; isbn:0131118269 
Running identify query with parameters: 
{u'authors': None, u'identifiers': {u'isbn': u'0131118269'}, u'timeout': 30, u'title': None} 
Using plugins: Amazon.com 
The log from individual plugins is below 

****************************** Amazon.com ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1')] 
Found 1 results 
Downloading from Amazon.com took 3.71499991417 


--- 
Title               : Core Java
Author(s)           : Cay S. Horstmann & Gary Cornell
Publisher           : Prentice Hall
Languages           : eng
Rating              : 4.4
Published           : 2004-12-02T00:00:00+00:00
Identifiers         : amazon:0132354799, isbn:9780131118263
Comments            : <h3>Amazon.com Review</h3><p>The fifth edition of <em>Core Java 2: Volume II--Advanced Features</em> brings a classic Java text up to date for developers tackling JDK 1.3 and 1.4, with a fast-moving and example-based tutorial. Perfect for those who want to learn programming through small, complete demos, this new version is still an excellent choice for mastering the more advanced topics in Java.</p><p>The salient feature of this tutorial is that authors provide small, complete programs that demonstrate each area of functionality in "core" Java along with reference material on essential APIs. Beginning with using multiple threading (and synchronization, which allow threads to communicate), the authors provide an engaging tour of advanced Java. The focus here is on client-side Java based on the Java 2 Standard Edition (J2SE), both versions 1.3 and 1.4, including APIs for building applets and applications with Swing and JavaBeans. As in the previous edition, sections on the Java collections do little to simplify the complexity here. Coverage of JDBC is very good, though, with a fine mix of examples showing off basic SQL and queries, plus transactions and newer JDBC features. Later on, this text turns to Swing, including how to extend three powerful Swing controls (lists, trees, and tables), plus improved J2SE 1.4 support for progress controls, plus drag-and-drop and clipboard support.</p><p>Besides providing source code and APIs, this book is filled with tips and gotchas to avoid. A robust section on security covers the entire spectrum of security issues in Java, including signing JAR files (along with good practical details). Though it's probably rare enough today, the authors go into good detail about creating custom JavaBean components. Examples make use of Sun Forte Community Edition, a capable Java tool (bundled on the accompanying CD-ROM).</p><p>Standout later chapters include an excellent tour of Java's strong support for internationalization (illustrated with a "retirement saving calculator" in English, German, and Chinese). A final section on XML introduces this powerful new standard, plus some essential Java APIs for getting starting with parsing and transforming XML. Chock-full of examples and useful advice, the latest edition of <em>Core Java 2--Volume II</em> is a worthy update to an already successful Java tutorial. <em>--Richard Dragan</em></p><p><strong>Topics covered:</strong> Programming with multiple threads (states and priorities, daemon threads); thread synchronization; avoiding deadlocks; using threads with Swing (dos and don'ts); introduction to collections (including lists, sets, trees, and maps); the collections framework; algorithms (sorting and searching); legacy collections; Java network programming (sending e-mail, using sockets and URLs, basic Web programming); JDBC and databases (quick SQL tutorial, connecting and querying data, metadata, and transactions); new JDBC 3.0 features; remote method invocation (RMI); Java and CORBA; advanced Swing tutorial (JList, JTree, and JTable, including custom rendering options); advanced AWT and Java 2-D graphics (including image manipulation and graphics filters); clipboard and drag-and-drop support; JavaBeans (properties and events, property editors and customizers); Java security (class loaders and bytecode verification, digital signatures, signing JAR files, encryption); internationalization (including date and time and multiple character sets); the Java Native Interface (JNI); Java and C interoperability; and a quick tutorial to XML (SAX and XSL in Java). </p><h3>From the Publisher</h3><p>The #1 advanced guide for serious programmers is fully updated for JDK 1.3 release and JDK 1.4 release. An indispensable companion to the best-selling <em>Core Java 2, Vol. I--Fundamentals</em>, <em>Core Java 2, Volume II: Advanced Features</em> is now available in a revised and expanded fifth edition. It delivers the same real-world guidance you need to solve even the most challenging programming problems and offers an all-new chapter on XML and Java, plus thoroughly revamped coverage of many advanced features--from collections to native methods, security to Swing. </p><p>Cay Horstmann identifies the problems experienced Java platform developers encounter most often, and delivers insightful, expert-level guidance for addressing them--together with even more of the robust, sample code that have made <em>Core Java</em> an international bestseller for five straight years. You'll gain new insights into networking, remote objects, JDBC API, internationalization, and a whole lot more. </p><p>For experienced programmers, this book provides the answers that they need to take full advantage of the power of Java technology and to get the job done as efficiently as possible. </p> 

******************************************************************************** 
The identify phase took 3.84 seconds 
The longest time (3.715000) was taken by: Amazon.com 
Merging results from different sources and finding earliest publication dates 
We have 1 merged results, merging took: 0.00 seconds

Where is the date of 2001 coming from and why isn't it finding the correct date of 2005?

Thanks.

[kovidgoyal]

Look at this line in the log:

Merging results from different sources and finding earliest publication dates

[ASpence]

I see that, but what is it implying? There is only one result, what is being merged? Where is the date of 2001 coming from?

[kovidgoyal]

earliest published dates are obtained from worldcat.org

[ASpence]

Thanks! That would seem to be the information I'm missing. Sorry if I'm being difficult but I went to that site and used the advanced search to search that book by ISBN and I got 10 results with published dates of 2004 and 2005. Any ideas why Calibre is getting 2001?

[kovidgoyal]

calibre is open source, run it from source, insert debug statements into the source and find out for yourself.

[ASpence]

Sorry for trying to present clear and unambiguous information on a bug. I will refrain from doing so again.

[kovidgoyal]

Good.

[tweebee]

I am fairly new to Calibre, and I was just about to post this same question as I just noticed it last week that the published dates are off in some of my books. I am not familiar with the code that much to run and debug it. Is there a way to turn off search results from worldcat.org since Kovid had stated that the old published date came from that?

[ASpence]

That data from worldcat.org is pretty good. I've only encountered one book where they returned incorrect information (they listed a 2nd edition as having a published date a couple years before the first edition).

The query used by Calibre returns data (including published dates) for a specified ISBN and data for other editions related to that one. I'm not a Python programmer either but my testing would seem to indicate that Calibre is selecting the earliest published date for all editions. This could be considered a personal preference but different editions (with different ISBNs) are different books. It would be nice if Calibre had an option to restrict downloaded published dates to only the specified ISBN. To be fair, the code for this in Calibre looks like it is trying to accommodate a large number of possible outcomes and combinations of ISBNs, Author names and Titles so its current functionality is likely a tradeoff.

Here is the query that Calibre uses that you can type into your browser's URL bar:

http://xisbn.worldcat.org/webservices/xid/isbn/XXXX?method=getEditions&format=json&fl=form,year,lang,ed

Replace the XXXX with your ISBN. This will return the raw text data for that book and related books and you will be able to see the published date you are looking for.

Alternatively, you can install a Calibre plugin called Search the Internet and add a custom search with the following string:

http://xisbn.worldcat.org/webservices/xid/isbn/{isbn}?method=getEditions&format=json&fl=form,year,lang,ed

This will allow you to right click a book and open a search at worldcat.org using the ISBN for that book.

I have hope that there is a less manual way to do this that I have not yet discovered.

[tweebee]

Thanks for the response! I have quite a few math books in my collection and most of them are newer editions. And I just noticed this last week that those new editions are listed with old published dates from previous editions. These newer editions have different ISBNs and published dates, and I would think it makes more sense to have the correct dates of when the newer editions were published rather than what WorldCat listed, which are published dates of different previous editions.

Well, time to go fix those dates for my books. I hope there will be better solution or option for us in the future regarding this. I've only been using Calibre for a few weeks, so I'm trying to add my large library into it as fast as I can. I guess I should have paid better attention to what metadata is being downloaded.

[kovidgoyal]

Metadata download in calibre is optimized for general fiction books. In that context, per edition published dates are rarely useful. Indeed not just the earliest published date but the routines to match a result to a query are also designed to ignore edition information.

I have no interest in making this behavior optional personally, but I wont refuse a patch that does add an option to control it.

[tweebee]

Unfortunately for me that my whole library is pretty much 99% non-fiction books and mostly math, tech, and travel books which have many editions. I respectfully disagree with you Kovid, about the per edition published dates not being useful. The published date of each edition is also what makes each edition unique. If I own a 1st edition and 5th edition of a particular book. It doesn't look right to have them both shown with the same original published date when they are 20 years apart.

I'm not savvy enough to write a patch for Calibre, so I guess I'll just do it the manual way or until some kind soul can write a patch. Regardless, I came from Music Collector and find Calibre much more robust, so many thanks to you Kovid!

[HarryT]

Originally Posted by tweebee

Unfortunately for me that my whole library is pretty much 99% non-fiction books and mostly math, tech, and travel books which have many editions. I respectfully disagree with you Kovid, about the per edition published dates not being useful. The published date of each edition is also what makes each edition unique. If I own a 1st edition and 5th edition of a particular book. It doesn't look right to have them both shown with the same original published date when they are 20 years apart.

I appreciate that they aren't useful for you, but, as Kovid says, the most common use for Calibre is for fiction books, and for those, the original publication date is really all that most people are interested in. If I buy a copy of, let's say, "Dune", all I really care about is when it was first published (1965), not when the particular edition that I bought happened to be published.

It would be nice to have the option of selecting the original, or edition-specific publication dates, but I think that the current behaviour of Calibre is that one which provides the greatest benefit to the greatest number of people.

tweebee, to disable searching on xisbn.worldcat.org you can add the line

127.0.0.1 xisbn.worldcat.org

to C:\Windows\System32\drivers\etc\hosts