[GUI Plugin] Zotero Metadata Importer

[DaltonST]

@excaliber:

Your previous comments were predicated on ZMI Version 1.0.2, which is obsolete. ZMI has significantly changed as of Version 1.0.3. Please read the Original Post which has been greatly updated for new information and also read my previous 2 posts:

DaltonST @excaliber: The use of the... Today, 11:04
DaltonST Version 1.0.3 Posted Today, 14:48

Is this the normal procedure?

Absolutely not. Use Version 1.0.3's auto-add function.


DaltonST

[DaltonST]

Version 1.0.4 - 2016-06-24 New Identifier: DOI (Document Object ID) added from ZDoi. New messages while calibredb is adding missing books.


DaltonST

[DaltonST]

@excaliber:

The image named "ZMI_Import.png" that you gave me (see the attached) has some errors.

[1] The Zotero CSV file has no column 'Words'.

[2] The Zotero CSV file column 'Pages' is a range of pages,like 123-135. It is not always single integer. I cannot map non-integers to an integer column like #pages.

[3] ZMI already has the custom column #zotero_volume a.k.a. ZVolume.

[4] ZMI already adds the Identifier type "isbn" using the CSV "ISBN". Ditto for ISSN and DOI. In Calibre, ISBN, ISSN and DOI are "identifier types", not custom columns. The custom column "ISBN" you may see is a virtual custom column that shows what is in the Identifiers table of Calibre. You can safely delete the custom column ISBN without losing any ISBNs at all, because it is a virtual column, not real.


Regardless of the above, I am working on ZMI Version 1.0.5 which will allow you to easily copy any #zotero_xxxxx custom column to any target non-zotero custom column as long as the target custom column is not one of the following datatypes: composite; enumeration; series; rating.

Obviously I have to convert the ZColumn data into the proper datatype (integer or whatever). If the data cannot be made valid, it will not (cannot) be copied.

See the other attachment for an idea how the users will specify the from:to combinations. It saves them, of course.




DaltonST

[petzi]

@DaltonST: Thanks for providing a bridge between Zoteror and Calibre.

It seems to me that I have the same problem as @excaliber. I will use his explanation of the procedure he (and I) was using and qualifying some of the aspects with my experience:

I have a folder with books and I import them into Calibre as well as into Zotero.

(1) I fetch the metadata in Zotero.
(2) I export the metadata to a csv file.
(3) I import the csv file into Calibre via ZMI. (1st button)
(4) I highlight the new books in Calibre.
(5) I select the csv file (2nd button)
(6) I try to update the metadata of the new books in Calibre. (3rd button)

-> 0 books updated

Before starting the above procedure:

I am working on a OS X 10.10.5 (Yosemite).
I created in Zotero (Firefox Plugin, not the standalone Zotero) for test purposes a folder with just two books. I added exactly these two books into Calibre independently from Zotero via their ISBN and collected their metadata within Calibre.

(1) and (2): I'm exporting all data from a folder of my Zotero library into a CSV file. Maybe important(?): I have ZotFile installed.

(3) to (6) Same as @excaliber with the same result: 0 books updated.

But I could update the books when I used the ZMI button for manually updating a single book.

Other experiences:

• After doing (5) I got the message that three rows of the csv were selected (but I have only two books in it).

• After selected the two books in calibre I had to change to ZMI in order to press the button. It seems to me that ZMI deselected my selection.

Any idea what could be wrong?
Thanks in advance,
Peter

[DaltonST]

Any idea what could be wrong?

@petzi:

Yes, I know exactly what is wrong.

ZMI must import the books into Calibre for you. You cannot manually import them yourself because Calibre will change their filenames, and then there will be no way to match them back to Zotero.

You should delete all of the Zotero books that you manually added into Calibre, and then use ZMI to add them back. See the attached image that shows you the dialog that you should see. "Highly Recommended" really means "Highly Essential".

Better yet, create a new Calibre Library just for Zotero, and start fresh using ZMI exclusively to add the books for you.

If you do not use ZMI to add the books, then you will have to use the ZMI "manually update metadata" pushbutton to manually select the correct row from the CSV file for the correct Calibre book that you have selected. Tedious, but it works. I provided that capability as a last resort only.

I suggest you re-read the Original Post of this thread and review all of the attached images.




DaltonST

[DaltonST]

Version 1.0.5 - 2016-06-25 Major enhancements.

Since the Calibre developers will not do the right thing, I have added functionality to update the Book Detail View with the ZColumn Labels, and also to reverse that update.


DaltonST

[petzi]

@DaltonST:

You were completely right! Now it works!

Crucial for my success was your explanation about filenames:

It matches by finding a Calibre book Path with a book filename that is identical to the Zotero file attachment filename. It ignores everything in the Paths except the final filename.

After that I could solve my problem by dropping the Calibrefiles into Zotero, so that the filenames matches exactly. But changing the filenames manually from Calibre to Zotero is a tedious task. Maybe I still did not understand how to apply your plugin?

I am relatively new to Calibre, meaning that almost all my pdf-files are still inside Zotero. How can I get them to Calibre so that I can them convert and read as ebooks? What would be the preferred procedure of action steps?


BTW: Your link under [1] in Special Operating System Notes is missing an "e" (www.mobilread.com instead www.mobileread.com). As I had problem with installing calibredb I would suggest you refer not only to the general website but to a thread with more explanation (e.g. https://www.mobileread.com/forums/sho...d.php?t=264945, especially post #6)

And for your patience with a Newbie like me.
Peter

[DaltonST]

@petzi:

You are still doing it incorrectly. Follow the advice in my prior post about creating a fresh Calibre Zotero Library and using only ZMI to add the missing books from Zotero itself.

Please read this post again:

https://www.mobileread.com/forums/sho...5&postcount=20


Abandon your current Calibre Library and start over with a brand new created freshly by Calibre via the menu option for that. Only ZMI can import your Zotero books, not you via "Add Books".



DaltonST




DaltonST

[DaltonST]

@excaliber et al:

I came up with an easy way to automatically show the labels in the Detail View, but is also easily reversible since to use html in the Zotero custom columns corrupts them. They are not really "comments" at all. They are simple text that has to be stored in Calibre in a comments custom column because there is not a datatype of "non-normalized, simple text".

Because of data quality unreliability, inconsistency, user-defined columns and column usage, ZMI has to import them as text to avoid Calibre crashing due to datatype conflicts. They are static information of what Zotero knows about the books.

I have also changed ZMI to strip out the html that Calibre often automatically creates for a datatype=comments when ZMI does not want it to be there for the purposes of the Detail View.

Since your GUI has all of the datatype=comments columns shown in the Library View since they are not really comments, but are really simple text, you will have to endure having the Column Name at the beginning of every value in every row of every column once you have decided it is time to show the labels in the Detail View. Obviously that will not be until the very end of the metadata update process. If you cannot stand it for some reason, you can strip the labels (stripping all html) from the columns temporarily until you want the Detail View labels back again.

See an attached example.


DaltonST


n.b. do not forget to compress/vacuum/defragment your Calibre Library database via the "Library Maintenance" > "Check Library" menu option. Your metadata.db file will get enormous over time. That will compress it.

[petzi]

Sorry, I cannot find the problem.

• I created a fresh Calibre Zotero Library
• I started ZMI and the columns were auto generated. Calibre restarts and my empty database has many new Z-columns.
• After the Validation "All Zotero custom columns exist" I pressed the button "Select & Process Zotero CSV Export File…" and selected the .csv export file which I have generated by exporting a collection of two books from Zotero. It contains only two entries type "journalArticle" and "conferencePaper". I used english references to prevent some problem with German umlaute.
• Now I've got the messsage "2 Zotero book(s) have no matching Calibre book(s) with the same ZKey.…" My Calibre database is still empty
• I press "Yes" in order to add these 2 books. A message on the left bottom appears: "ZMI: calibredb has tried to add 2 missing book(s)" but nothing happens, no books were imported.

I have compared the filenames under "file attachement" in Calibre with the filenames in Zotero storage. They are the same. But the Zkey in the .csv file did not conform with the storage folder. Maybe this is the relevant issue? Additionally I noticed that the .csv started with the strange sequenece of three characters

"Key","Item Type","Pub…

I don't know what went wrong. I was using calibre 2.60 and ZMI 1.0.5. on OS X 10.10.5.

[DaltonST]

@petzi:

As I mentioned in a previous post in this thread, the CSV "file attachments" must exactly coincide with the path of your Zotero books. If your CSV file is corrupt or erroneous, then ZMI cannot possibly find your Zotero books.

If the "key" column (the very first column) is corrupt, it is just as fatal.

Perhaps you should determine why Zotero is exporting a corrupt CSV file. If you cannot, try doing a manual "search and replace" on the very first column, "key", to change it to remove the leading "". I have absolutely no control over what Zotero exports to its CSV file. It is up to you to ensure that the file is correct.

Use Libre Office or Open Office to change the .csv. Be sure to save it as UTF-8.

You should not change the encoding just because you have umlauts. UTF-8 encoding handles Deutsch just as it handles any other European language. If you changed it, then it is possible that the change caused the corruption.

Try exporting many Zotero books to the Zotero CSV file, not just two (2). That is not enough to draw a conclusion as to the source of the problem. Export at least twenty (20).

Good luck.


DaltonST

[DaltonST]

Version 1.0.6 - 2016-06-26 Major enhancements.

The functionality of adding Book Detail View labels has been refined, as has the ability to randomly copy a ZColumn to another Calibre Custom Column.

Most ToolTips have been changed to reflect the paradigm that only ZMI may add Zotero books to Calibre, and that the "Add Books" button in Calibre should never be used to add Zotero books unless you want to manually input their metadata into Calibre one-by-one.



DaltonST

[DaltonST]

Version 1.0.7 - 2016-06-26 Enhancements.

After attempting to add missing Zotero books to Calibre automatically, any CSV file attachments that were not able to be found within Zotero are listed in a new error dialog. That can be copied to the clipboard for research as to the cause.

The most common cause of the above errors is that the Zotero CSV file has encoding or other CSV format errors that are causing the File Attachment for a particular key to not be found in the Zotero storage folder.

Remember that Microsoft Excel 2010 (and perhaps later) does not properly support UTF-8 encoding of its files. If you make changes in Excel 2010 (and perhaps later), you likely will be corrupting the previously (and properly) UTF-8 encoded CSV file that was exported directly from Zotero.

Libre Office, Open Office, and Google Spreadsheets have always handled UTF-8 properly as a "save-as" encoding/filtering option.



DaltonST

[petzi]

I have tried again with the new version and now I've got some error message explaining was went wrong: For instance:

calibre, version 2.60.0
FEHLER: ZMI: Missing CSV Books Not Found within Zotero:

EX5MBGER -->> /Users/xxxxxx/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/E2CZVGF7/e-Learning Scott Midkiff.pdf

But the file is exactly where it should be. I copied the error message and took the local URL and tried it via the browser and I could reach the file.

*******

Another remark: Yesterday I tested ZMI with another Mac and on a windows machine. In fact, there is a difference as windows Excel does not produce these strange three letters I reported.

After your last message I started to open the csv.file with OO and the murky characters did not appear. But when I save the file with OO and try to take it as input with ZMI I got a very different error message: In that case ZMI did not report on any files but:

calibre, version 2.60.0
FEHLER: CSV File Error: Error: file atttachments column not found in dict. Cannot proceed.

I don't know what "dict." means. When I save the OO-File into a .ods-format, open it with OO and save it again with .csv then I receive the same error message as above "Missing CSV Books Not Found within Zotero:"

Any idea what could I try next?

Peter

[DaltonST]

@petzi:

[1] Does anything in the CSV work for you? 0%? 50%? 99%? How many books are in the CSV that you are testing?


[2]

FEHLER: CSV File Error: Error: file atttachments column not found in dict. Cannot proceed.

You should never see that message. I have never seen that message, although I programmed it as a theoretical error if the virtually-impossible happened. You made it happen. That means there is serious corruption of the CSV file. I cannot be sure, but I suspect that perhaps you saved it from OO without first changing the encoding/filtering option to Unicode-8.

Please export a CSV file from Zotero for 100 of your books. Immediately zip it. Do not touch it otherwise. I need the virgin CSV file exported from Zotero. Then, please attach it to your previous post.


[3]

When I save the OO-File into a .ods-format, open it with OO and save it again with .csv then I receive the same error message as above "Missing CSV Books Not Found within Zotero:"

Please do not save the .csv to .ods. Open the virgin CSV in OO, fix whatever is wrong, then save with Unicode-8 encoding directly back to a CSV. Changing formats introduces other variables into the equation, making it difficult to solve.

[4] You might try using Zotfile to rename the books that have the errors to use only basic characters to see if that fixes the problem.

The reasoning is that although Calibre's calibredb command-line utility that ZMI invokes to add the Zotero books is designed to be cross-platform (Linux/OSX/Windows), perhaps it cannot deal with file attachment filenames that "push the envelope" even for OSX itself. So, perhaps Zotfile can rename the file such that it becomes more "plain vanilla".

Calibredb generates a debug log message of "xxxxx.pdf not found" when it cannot find a file.

After calibredb finishes running, ZMI then makes a list of all Zotero keys used as an Identifier in Calibre (e.g. zkey:ABCDEFG), then compares all of the Zotero keys in the selected CSV file to that list. Missing keys are shown in the new error dialog that you referenced.

Try restarting Calibre in "debug mode", use ZMI the same way you have been using it with a CSV of 100 books, quit Calibre, zip the debug log file, and attach it to the previous post along with its related CSV.

To restart in "debug mode", right-click the "Preferences" button, then click "Restart in Debug Mode".

[5] Going forward, please attach the CSV file that you are referring to. Zip it as soon as it comes out of Zotero. Do not even open it to look at it in OO or anything else. The fewer variables, the better.

I am going to post a new version very soon that prints extra debug information to help me figure this out. Please wait to do the above until I have posted that new version, which will be shortly.



Thanks.


DaltonST