[GUI Plugin] Zotero Metadata Importer

[dzgarza]

Originally Posted by DaltonST

The Original Post, its attachments, and the ZMI plug-in's ToolTips contain the only up-to-date information that needs to be read. All old posts should be ignored as they often are deprecated or obsolete, add no value, and are a waste of time to peruse. To answer any questions you may have, simply download and carefully review the image attachments to the Original Post, then create a test Library, install ZMI, read the ToolTips (they are the documentation), and use it. It is all self-explanatory.


DaltonST

Thanks for the response!

I should have been more clear - I've already installed the plugin and have used it successfully, no real issues there.

The issue is that I noticed that my annotations to PDFs opened via Calibre were not appearing in when opening the corresponding Zotero entry imported via ZMI (and vice versa), and also that all of my references within Zotero seemed to break when my /tmp directory was cleaned.

I did follow most of the steps outlined in original post and tooltips, but it appears that it is still using the workaround of copying the entire library to "safe" names under /tmp and pointing to those files, instead of linking to the PDF in Calibre's library folder.

My question is mainly whether or not this is intentional, and if there is a workaround or fix that avoids this and links directly.

[DaltonST]

Originally Posted by dzgarza

Thanks for the response!

I should have been more clear - I've already installed the plugin and have used it successfully, no real issues there.

The issue is that I noticed that my annotations to PDFs opened via Calibre were not appearing in when opening the corresponding Zotero entry imported via ZMI (and vice versa), and also that all of my references within Zotero seemed to break when my /tmp directory was cleaned.

I did follow most of the steps outlined in original post and tooltips, but it appears that it is still using the workaround of copying the entire library to "safe" names under /tmp and pointing to those files, instead of linking to the PDF in Calibre's library folder.

My question is mainly whether or not this is intentional, and if there is a workaround or fix that avoids this and links directly.

ZMI uses no "workaround". I do not know where you are getting that unless it is from some old, obsolete post from when ZMI was being first developed.

Zotero is Zotero, Calibre is Calibre, and they do not "link" to each other.

When exporting from Calibre, all of the PDFs are temporarily stored in a temp folder until you open Zotero and import them into Zotero. Then, you have the PDFs in both Zotero and Calibre, separately and independently. The temporary folder is just that: a temporary place to keep the PDFs in the middle of the whole process, and is purged later like any other "trash" would be in your OS.

When you open a PDF in Zotero that originally came from Calibre, Zotero uses the Zotero copy of the PDF that it put in your Zotero database's "storage" folder.

Likewise, when you export from Zotero and import your PDFs into Calibre, Calibre has a copy of the original Zotero PDF along with the Zotero metadata. When you open your PDF in Calibre, it uses only the PDF in the Calibre Library.

You should be using the "embed metadata" tool in Calibre to update the PDF's internal metadata before you export them and then import them into Zotero. Otherwise, when you open the PDF in Zotero, it will not have all of the internal PDF metadata otherwise available in Calibre's metadata.db because it was not first embedded from metadata.db into the .pdf book format prior to exporting.

To reiterate, a Calibre PDF uses only Calibre, and a Zotero PDF uses only Zotero, and they do not "link" to each other once the Zotero import of the RIS file has completed.


Attached is an image from the Original Post that shows what in Zotero came from Calibre. Everything shown in Zotero is entirely stored within Zotero. You could then delete your entire Calibre Library with no impact whatsoever on Zotero once it has finished importing the RIS file.


Added: ZMI was designed to be used with Zotero installed on the same computer system as Calibre. If you are using some "cloud" version of Zotero, then it will probably not work. I tested with Zotero using the latest version for Windows 10 as of the latest release 1.0.59 of ZMI in April 2018. My Zotero database and storage folder were on my SSD.


Added: You mentioned "annotations". If you use the "embed metadata" tool in Calibre, all metadata, including Comments and any Custom Columns containing anything, including "annotations", will be embedded. How the "annotations" got there in the first place (manually or via another plug-in) does not matter. Everything in metadata.db (what you see in the Calibre "Library View" of its GUI) gets embedded. I believe there is a ToolTip in ZMI that says to always "embed metadata" prior to exporting to Zotero. That is true whether you have "annotations" or not. Also, your ZMI configuration for mapping Calibre metadata to the RIS export file must include the source of "annotations" in Calibre that you want in Zotero. Refer to the ZMI RIS Configuration ToolTips and also the images attached to the Original Post.



DaltonST

[dzgarza]

Originally Posted by DaltonST

ZMI uses no "workaround". I do not know where you are getting that unless it is from some old, obsolete post from when ZMI was being first developed.

Zotero is Zotero, Calibre is Calibre, and they do not "link" to each other.

When exporting from Calibre, all of the PDFs are temporarily stored in a temp folder until you open Zotero and import them into Zotero. Then, you have the PDFs in both Zotero and Calibre, separately and independently. The temporary folder is just that: a temporary place to keep the PDFs in the middle of the whole process, and is purged later like any other "trash" would be in your OS.

When you open a PDF in Zotero that originally came from Calibre, Zotero uses the Zotero copy of the PDF that it put in your Zotero database's "storage" folder.

Likewise, when you export from Zotero and import your PDFs into Calibre, Calibre has a copy of the original Zotero PDF along with the Zotero metadata. When you open your PDF in Calibre, it uses only the PDF in the Calibre Library.

You should be using the "embed metadata" tool in Calibre to update the PDF's internal metadata before you export them and then import them into Zotero. Otherwise, when you open the PDF in Zotero, it will not have all of the internal PDF metadata otherwise available in Calibre's metadata.db because it was not first embedded from metadata.db into the .pdf book format prior to exporting.

To reiterate, a Calibre PDF uses only Calibre, and a Zotero PDF uses only Zotero, and they do not "link" to each other once the Zotero import of the RIS file has completed.


Attached is an image from the Original Post that shows what in Zotero came from Calibre. Everything shown in Zotero is entirely stored within Zotero. You could then delete your entire Calibre Library with no impact whatsoever on Zotero once it has finished importing the RIS file.


Added: ZMI was designed to be used with Zotero installed on the same computer system as Calibre. If you are using some "cloud" version of Zotero, then it will probably not work. I tested with Zotero using the latest version for Windows 10 as of the latest release 1.0.59 of ZMI in April 2018. My Zotero database and storage folder were on my SSD.


Added: You mentioned "annotations". If you use the "embed metadata" tool in Calibre, all metadata, including Comments and any Custom Columns containing anything, including "annotations", will be embedded. How the "annotations" got there in the first place (manually or via another plug-in) does not matter. Everything in metadata.db (what you see in the Calibre "Library View" of its GUI) gets embedded. I believe there is a ToolTip in ZMI that says to always "embed metadata" prior to exporting to Zotero. That is true whether you have "annotations" or not. Also, your ZMI configuration for mapping Calibre metadata to the RIS export file must include the source of "annotations" in Calibre that you want in Zotero. Refer to the ZMI RIS Configuration ToolTips and also the images attached to the Original Post.



DaltonST

Ahh I see! It seems we have completely different workflows in mind, that's why things were a bit confusing. I don't really have a desire to move everything out of Calibre permanently, I simply wanted to add my current library as a bunch of references into Zotero without touching the PDF files (but still be able to open them through Zotero, e.g. for jumping to links in extracted annotations).

By "linking", I'm referring to the ability within Zotero to link to an existing file without copying the actual file into Zotero's own library folder. This is done through the GUI by holding Ctrl-Shift while dragging a PDF in, for example. It leaves the original location intact, and only stores its relative location.

If, for example, I navigate to Calibre's library folder and drop the PDF into Zotero in this way, this accomplishes what I want - it's just not viable to do that for a large collection.

I'd hoped that this linking functionality within Zotero would be exposed through its import system or by setting a certain tag to its file path, and it sounds like that's not the case. Thanks anyway!

[rerun]

Hi Dalton,
is ist or could it be possible to import fields like "num pages", "edition", "rights" etc from zotero csv to calibre?

[DaltonST]

In theory, any .csv column that is not already in Calibre could be added if it added value to a significant number of ZMI users, since it would require adding additional Zotero Custom Columns to the current 27 CCs that already are automatically created and populated for all ZMI users.

Number of Pages: Custom Column "zotero_pages", "A page number, or a range of page numbers, associated with the item", already exists in ZMI. You could easily copy "Number of Pages" into "Pages" in Zotero using a standard Zotero utility to do so. Also, Number of Pages in Calibre would be a page count created by the Count Pages plugin, which would update a user-defined CC such as #pages.


Edition and Rights were not included in the original 27 because the consensus by the Zotero users who contributed their needs and comments to my original development of ZMI in 2016 was that those two specific Zotero metadata elements (any many more) were of little or no value in Calibre, and not worth including, especially since the PDFs in Zotero can have their Zotero metadata updated into them ("embedded" in Calibre terminology) before importing them from Zotero into Calibre via ZMI. They can then be viewed with a PDF viewer that displays embedded metadata. Obviously, they can also be viewed in Zotero itself.


ZMI was not at all intended to totally reproduce Zotero inside of Calibre via Zotero's .csv export file, nor to reproduce Calibre inside of Zotero (although it can come pretty close if ZMI's RIS configuration is done thoroughly enough).




DaltonST

[DaltonST]

Having said that, see this Job Spy plug-in GUI Tool that would allow you to update any of the 87 columns in a Zotero export .csv file regardless of whether they correspond to one of the 27 Calibre custom columns that are natively automatically created by the ZMI plug-in.

https://www.mobileread.com/forums/sh...42118#poststop

Obviously, you would have to create new Calibre custom columns to contain the imported data random CSV file column values.


DaltonST

[sutzig]

I'm having issues with exporting an RIS files from Calibre/ZMI which I can then import into Zotero. I've installed the plug-in, select the books whose metadata I would like to export by ctrl+click, launched the ZMI application, selected the ZMI: Export RIS tab, clicked the Export RIS file [Selected Books] button, and get the error "No books were selected."

Could you please itemize the correct steps for exporting the metadata for selected content from Calibre into and RIS file that can then be imported into Zotero.

Thank you.

[DaltonST]

You said that you launched the ZMI application AFTER you selected the books. I bet that your "blue" selections disappeared. So, nothing selected. Launch ZMI and THEN select the books to export.

[sutzig]

I did it both ways. If I launch ZMI and then click to select the books to export, the ZMI dialog box disappears/closes and when I reopen it the books are no longer selected. Catch 22.

I discovered however that if I launch ZMI, hold down the Command key (Mac OS), and THEN click on the books, I can select the items. If I just launch ZMI and click on the title, , ctrl+click on the title, shift-click on the title, option+click on the title, the ZMI dialog box disappears/closes.

[DaltonST]

Originally Posted by sutzig

I did it both ways. If I launch ZMI and then click to select the books to export, the ZMI dialog box disappears/closes and when I reopen it the books are no longer selected. Catch 22.

I discovered however that if I launch ZMI, hold down the Command key (Mac OS), and THEN click on the books, I can select the items. If I just launch ZMI and click on the title, , ctrl+click on the title, shift-click on the title, option+click on the title, the ZMI dialog box disappears/closes.

I use Windows 10, and have never seen or heard of that problem before.




DaltonST

[sutzig]

I'm glad I could raise this bug/product enhancement for Mac OS (I'm using Mojave 10.14.3).

[DaltonST]

Version 1.0.60 - 2019-05-30 Python 3 compatibility. Minimum Calibre version now 3.41.3.

As always, upgrade Calibre (if necessary) prior to upgrading ZMI, and test the new version using a test Calibre Library.



DaltonST

[ksburgess]

I'm using the Juris-M variant of Zotero (specialized for legal research). I gave the plugin a go and it pulled in all of my cases, but the metadata didn't import properly. Each case was given a title/author of "Zotero/Zotero."

Do you think this is because it's coming from Juris-M rather than Zotero proper (Juris-M logs in using all of my Zotero credentials, accesses Zotero servers, etc.)?

Or maybe I did something wrong having it pull in papers from Juris-M?

Thank you for your help.

[ksburgess]

Okay maybe I missed a step before - tried again and the case titles came in. Authors all still show Zotero, but that's likely because they're cases and I don't have an author specified (rather, court name).

[pleiadesc]

Hi DaltonST!

I'm very happy to find and use this plugin, I use it to export books from Calibre into Zotero and it made my workflow easier

Would it be possible to export a network path / UNC instead of a mapped drive?

Extra field exports
E:/Calibre Library/John Schember/Quick Start Guide (1)/Quick Start Guide - John Schember.epub

can it export something like
\\SERVER\lib/John Schember/Quick Start Guide (1)/Quick Start Guide - John Schember.epub

Thanks!