[GUI Plugin] Zotero Metadata Importer

[DaltonST]

Originally Posted by partnoonein

I tried the 5.99.8 beta with new zmi and followed the steps to compare. Same result


Total Collection items in Zotero: 4
Total Collection items in Calibre: 0

Key: 4HNN39E7 type: book title: blahBlah1 content: None path: None

Key: AUWDQZG4 type: book title: blahBlah2 content: None path: None

Key: UBCBQHID type: webpage title: blahBlah3 content: None path: None

Key: UKM6K5VJ type: book title: blahBlah4 content: None path: None

Expected differences are equal to the actual differences listed above.

Okay, now I can start to help you. However, just saying "it doesn't work" is not helpful. I need the exact data that "did not work".



Please post the original .ris export file from ZMI for import into the Desktop Zotero application. I need to see the exact Calibre data being exported from ZMI.



Then, post a screen snip of the Zotero metadata for the item in Zotero imported from ZMI via the .ris export file that you posted just above, including the Calibre metadata imported plus the Zotero Collection to which it belongs.


I also need a screen snip of your PC's Zotero "Storage" folder for the document item imported above into Zotero from the .ris export file from Calibre ZMI.



Obviously, if you originally imported a document into Calibre from Zotero using ZMI, and then turn around and export a ZMI .ris file for the same document and import that .ris file into Zotero, there will be a problem. ZMI was not designed to go full-circle. Exporting to Zotero via a .ris file is an alternative to importing a Zotero .csv export file into Calibre via ZMI for each individual document. Hence, I need to see the screen snips and .ris file itself described above to figure out what is going on.


Thanks.




DaltonST

[DaltonST]

Originally Posted by partnoonein

I had actually followed additional steps after his step 3. I had indeed imported the book metadata through csv/ZMI into calibre. <<<--- See [2], below.

Okay, I studied the many files in the .zip that you sent me.


[1] Your exporting a .csv from your Zotero Libary and importing it into Calibre using ZMI to update all of the ZMI Zotero_* Custom Columns looks correct and proper.

[2] Your exporting a .ris from Calibre using ZMI for the identical books just imported from Zotero's .csv export file is not proper. See [5] & [6], below.

[3] If your Document Master Library is Calibre, then you move documents with their metadata from Zotero to Calibre via ZMI. You simply use Zotero as a conduit to Calibre/ZMI. Your documents stay in Calibre.

[4] If your Document Master Library is Zotero, and you simply want to use a particular Calibre ebook as a "source" for a citation or other purpose within Zotero, you do NOT import from Zotero into Calibre, but rather you export a .ris file for that particular Calibre ebook, and then import the .ris file into Zotero. You can track which Calibre ebooks are used in Zotero using ZMI to do so.

[5] Never export a .csv file for a particular Item from Zotero, then import that Item into Calibre as a new ebook using ZMI, and then turn around and export that new Calibre ebook to a .ris file that is subsequently imported as a new Item into Zotero. That makes no sense.

[6] Either import into Calibre using a .csv, or export from Calibre as a .ris, but do not do both for the identical document/item/ebook.

[7] You need to make a strategic determination as to the use of Zotero and Calibre in your document/item/ebook ecosystem based on the previous points.


Other comments:

You did not provide a screen snip of the "ZMI: Export RIS Tab" at the bottom where the pushbutton named "Retrieve ZKey from New Zotero Attachments (Just Imported)" is positioned for use immediately after the "Export RIS File ...." pushbuttons. So, I must assume that you skipped that step. Hover your mouse over that pushbutton to read its ToolTips.

If you do not Retrieve the ZKey (which was just created in Zotero for the RIS File item), then ZMI will not be able to connect the new .ris Item in Zotero with the Calibre ebook metadata that was just exported to that .ris file. If it cannot connect the two, then it cannot "Compare Collections" in the "ZMI: Comparison Tab". Matching a Calibre ebook to a Zotero Item is all about matching ZKeys.

Regarding the "ZMI: Comparison Tab", don't forget to push the various pushbuttons at the top so the Collections in Zotero are refreshed in Calibre in the ZCollection Custom Column. Refer to the ToolTips. The Collections in Calibre are "refreshed" every time you click the appropriate pushbutton in the "ZMI: Comparison Tab". So, if you change a Zotero Item's Collection, the new Collection value will be updated by ZMI in Calibre the next time you do a Compare by first pushing the top pushbuttons and clicking the Collection dropdown list.

All of the Z* Custom Columns in your Calibre "Zotero" Library must be set to be visible. It is also trivial to "Show" a hidden Custom Column in your Calibre Library by right-clicking your mouse in the Library View header row, and selecting "Show". The "View Manager" Plugin is excellent for that purpose too, plus has many other functions and features that ease the use of ZMI.



DaltonST

[partnoonein]

Dalton,
Thank you for your detailed reply. I feel something, a crucial point is missing in our communication that I have been unable to communicate. My use case is as follows sequentially.

1. I want to be able to use Calibre as a source for the PDF and Zotero for references for research writing. So your [3] assumption is correct. About [7] I hope it is clear that I want to use Calibre for PDFs and Zotero for references to be used in MS word.
   
2. I want to export this from Calibre as an RIS file using the ZMI plugin.
3. (Note: my starting point is not the export of .csv from my Zotero Libary, but the export of .ris from Calibre. In your response, [2] should come before [1])
   
4. I want to import the RIS file into Zotero as links.
5. (Note: I do not want to turn around and export from Zotero a CSV to be imported into Calibre unless you tell me it is necessary. I did that as according to one of your responses to MassiveAtoms member it appeared to me that is what I was supposed to do).
6. (Note: If I click on "Retrieve ZKey from New Zotero Attachments (Just Imported)" after import of RIS into Zotero, and closing Zotero, nothing happens. No keys are retrieved. That is the only reason I thought I was supposed to export a CSV file from Zotero because that seems to update the ZKey and other data; in other words, agree that the [5] sequence does not make sense. I hope it is clear that I did the opposite sequence described in [5] due to the inability for retrieving ZKeys and other metadata. [6] also makes sense and I did the RIS export but the keys are not retrieved.).
   
7. I want to use Microsoft Word and Zotero and use the citations/references from Zotero plugin of MS word.

Addressing other comments:
I did not skip the "Retrieve ZKey from New Zotero Attachments (Just Imported)" step. I always religiously click on that faithfully closing Zotero etc. No ZKeys are retrieved for me.

I wish there was a way I could show you what is going on because there seems to be a point that is being missed that I seem to be unable to communicate to you. Let me know if the use case I described above is supposed to work with the plugin. You have spent a lot of time developing this wonderful plugin and it would be great if we can clarify for once and all whether it is supposed to support this use case. Let me know how I can help.
Thank you

Originally Posted by DaltonST

Okay, I studied the many files in the .zip that you sent me.


[1] Your exporting a .csv from your Zotero Libary and importing it into Calibre using ZMI to update all of the ZMI Zotero_* Custom Columns looks correct and proper.

[2] Your exporting a .ris from Calibre using ZMI for the identical books just imported from Zotero's .csv export file is not proper. See [5] & [6], below.

[3] If your Document Master Library is Calibre, then you move documents with their metadata from Zotero to Calibre via ZMI. You simply use Zotero as a conduit to Calibre/ZMI. Your documents stay in Calibre.

[4] If your Document Master Library is Zotero, and you simply want to use a particular Calibre ebook as a "source" for a citation or other purpose within Zotero, you do NOT import from Zotero into Calibre, but rather you export a .ris file for that particular Calibre ebook, and then import the .ris file into Zotero. You can track which Calibre ebooks are used in Zotero using ZMI to do so.

[5] Never export a .csv file for a particular Item from Zotero, then import that Item into Calibre as a new ebook using ZMI, and then turn around and export that new Calibre ebook to a .ris file that is subsequently imported as a new Item into Zotero. That makes no sense.

[6] Either import into Calibre using a .csv, or export from Calibre as a .ris, but do not do both for the identical document/item/ebook.

[7] You need to make a strategic determination as to the use of Zotero and Calibre in your document/item/ebook ecosystem based on the previous points.


Other comments:

You did not provide a screen snip of the "ZMI: Export RIS Tab" at the bottom where the pushbutton named "Retrieve ZKey from New Zotero Attachments (Just Imported)" is positioned for use immediately after the "Export RIS File ...." pushbuttons. So, I must assume that you skipped that step. Hover your mouse over that pushbutton to read its ToolTips.

If you do not Retrieve the ZKey (which was just created in Zotero for the RIS File item), then ZMI will not be able to connect the new .ris Item in Zotero with the Calibre ebook metadata that was just exported to that .ris file. If it cannot connect the two, then it cannot "Compare Collections" in the "ZMI: Comparison Tab". Matching a Calibre ebook to a Zotero Item is all about matching ZKeys.

Regarding the "ZMI: Comparison Tab", don't forget to push the various pushbuttons at the top so the Collections in Zotero are refreshed in Calibre in the ZCollection Custom Column. Refer to the ToolTips. The Collections in Calibre are "refreshed" every time you click the appropriate pushbutton in the "ZMI: Comparison Tab". So, if you change a Zotero Item's Collection, the new Collection value will be updated by ZMI in Calibre the next time you do a Compare by first pushing the top pushbuttons and clicking the Collection dropdown list.

All of the Z* Custom Columns in your Calibre "Zotero" Library must be set to be visible. It is also trivial to "Show" a hidden Custom Column in your Calibre Library by right-clicking your mouse in the Library View header row, and selecting "Show". The "View Manager" Plugin is excellent for that purpose too, plus has many other functions and features that ease the use of ZMI.



DaltonST

[DaltonST]

The latest version of ZMI can be found at the link below.

https://www.mobileread.com/forums/sh...8&postcount=11

Partoonein, Zotero has had some technical database changes over the years which were causing your issue. See the attached results of my testing using ZMI with a few very minor tweaks. The Scalzi book was a simple Calibre import of a normal epub with no use of ZMI at all. It was then exported using the Link Option. Note that the Calibre screen snip shows no Z* Custom Columns are filled in, meaning it did not come from Zotero via an exported .csv file. Simple, normal Calibre ebook that was linked to Zotero.


DaltonST

[partnoonein]

WOW! IT WORKS with the latest update of ZMI. Thanks for the fix Dalton.
Appreciate it very much.

Originally Posted by DaltonST

The latest version of ZMI can be found at the link below.

https://www.mobileread.com/forums/sh...8&postcount=11

Partoonein, Zotero has had some technical database changes over the years which were causing your issue. See the attached results of my testing using ZMI with a few very minor tweaks. The Scalzi book was a simple Calibre import of a normal epub with no use of ZMI at all. It was then exported using the Link Option. Note that the Calibre screen snip shows no Z* Custom Columns are filled in, meaning it did not come from Zotero via an exported .csv file. Simple, normal Calibre ebook that was linked to Zotero.


DaltonST

[partnoonein]

Another use case - topic/Question/
I have a bunch of books 350 that I had exported from Calibre/ZMI as RIS Links and imported into Zotero earlier. I have been using these as references in my research, so I cannot just remove them from Zotero (and reimport the RIS since the retrieval of ZKeys now works.)

Since ZKeys were not updated for these books in Calibre, how do you propose I update the ZKeys in Calibre or reconcile them in both programs?

As a side note, I tried to experiment. Since these books were already in both, I tried the following to put the relevant ZKey in calibre. I was trying to figure out how to reconcile in both. Here are my steps
1. In Zotero, exported the library as csv file
2. In excel, I removed all except the header and one line for that book in csv
3. In Calibre/ZMI, I selected Add Zotero Books and Metadata -> Manual Mode - Single Select CSV Export and updated the Z columns in calibre and select the book. I can view the value of the ZKey that has been updated for this book in Calibre.
4. I try to do ZMI: Comparison, and it says
Total Collection items in Zotero: 1
Total Collection items in Calibre: 0
Key: AUWDQZG4 type: book title: ABCD content: None path: None
Expected differences are equal to the actual differences listed above.



Thank you.

[DaltonST]

Just pretend you just exported their .ris now, do a Retrieve, and then do a Compare. Those books are handled as special "legacy" books and will be fine.

DaltonST

[partnoonein]

Thanks. So pretending that for legacy I tried tressing the "Retrieve ZKey from New Zotero Attachment [Just Imported]". It does not update any Z custom columns for any book.

I checked calibre for #zotero_key:true as well which results in no books to be shown.

Originally Posted by DaltonST

Just pretend you just exported their .ris now, do a Retrieve, and then do a Compare. Those books are handled as special "legacy" books and will be fine.

DaltonST

[DaltonST]

You do not understand at all how ZMI works. Read the images attached to the OP and all of the ZMI tool tips.

Z* Custom Columns are ONLY updated by importing a Zotero .csv export file.

.ris exporting does NOT update any ZMI custom colums.

The Retrieve action adds 2 Calibre Identifiers. My post above clearly shows the Scalzi book with NO Z* custom columns BUT with 2 special ZMI Identifiers, including zcollection and the ebook's zkey for its file.


DaltonST

[partnoonein]

Dalton,
I do understand that

1. Z* Custom Columns are ONLY updated by importing a Zotero .csv export file.
2. .ris exporting does NOT update any ZMI custom columns.

My assumption has been that if there is no Z data in Calibre then the comparison will not work. Is that correct?

I saw your post and images, but I could not assume that it always updated only two columns (needed for comparison) and no other columns.

So going back to the legacy mode not working (books already existing in both Calibre and Zotero because they were exported from ZMI as RIS ), I was wondering if you had a fix.

For the past year, I have poured numerous times over the images and tooltips. Else I would not be writing detailed messages. I understand you are trying to help, and I am trying to help you by making this plugin better as well. As one can imagine, comments like "you don't understand" can come across as inflammatory.

Originally Posted by DaltonST

You do not understand at all how ZMI works. Read the images attached to the OP and all of the ZMI tool tips.

Z* Custom Columns are ONLY updated by importing a Zotero .csv export file.

.ris exporting does NOT update any ZMI custom colums.

The Retrieve action adds 2 Calibre Identifiers. My post above clearly shows the Scalzi book with NO Z* custom columns BUT with 2 special ZMI Identifiers, including zcollection and the ebook's zkey for its file.


DaltonST

[DaltonST]

Your assumption is False.

My Scalzi example proves your assumption is wrong.

Just do what I said to do. Retrieve then Compare for ALL books via a Calibre select-all.


DaltonST

[partnoonein]

Dalton,

You are the expert as you wrote the plugin so need to prove anything to me. I understand that my assumption is false. I also understand what you were telling me to do. I have done that multiple times.

What I am saying is that for legacy it does not work and that the retrieve ZKey in ZMI does not do anything, comparison does not work.

As you would agree, there was a bug that you fixed today after so much back and forth. One thing perhaps is to think about whether there is another bug that you might want to look into.

Thank you,

Originally Posted by DaltonST

Your assumption is False.

My Scalzi example proves your assumption is wrong.

Just do what I said to do. Retrieve then Compare for ALL books via a Calibre select-all.


DaltonST

[DaltonST]

Legacy books MUST have the identical path they had when you exported their .ris file and imported it into Zotero. Reexport all new .ris files for ALL of your legacy .pdf files.

The linked item in Zotero has the Calibre .pdf in its Zotero Extra field, which is used to match .ris-path books.

If you have moved your Calibre Library directory, then the Extra column is now wrong and will never match Calibre. Linking forces you to NEVER change absolute paths in the future.

That means that your use of Calibre Portable 5.99.8 for ZMI is a very bad idea if your real library is underneath Portable where Calibre always puts it.


My previous post of the Scalzi example proves that Linking via ris and Comparing works properly. The attachment next to it proves that importing from Zotero and then comparing works properly.


I suspect that your real ZMI Calibre Library is not on Calibre 5.99.8, and hence cannot use the latest version of ZMI. So, you cannot have done what I said to do, but rather are running a thought experiment.

[partnoonein]

Thanks for your thoughts, Dalton. You are right about my real library not being 5.99.8. But I have indeed been doing real hands-on experiments (and not thought experiments to begin with.) So much so that here is what I found out.

1. If I use CalibrePortable5.99.8\Calibre Portable\Calibre\Calibre.exe to point to the legacy library (not on 5.99.8) then it does not retrieve ZKeys etc. That is what I had been using.

2. About 50 min ago, I used the CalibrePortable5.99.8\Calibre Portable\calibre-portable.exe to point to the same legacy library (not on 5.99.8) as used in 1. It updated ZKeys for 51 books out of 350 (I understand that paths etc. may have changed so it is unreasonable to have it work 100%).

To be clear I am using the latest update from you (that you provided today) with Calibre 5.99.8.
Any thoughts on why these two Calibres in 5.99.8 be different?

[partnoonein]

I also noticed that the comparison works for several books if they are in a particular collection (with CalibrePortable5.99.8\Calibre Portable\calibre-portable.exe).
In other words, if a book called "ABCDE" is in two or more than two Zotero collections, the comparison might detect the book in one collection or the other but might report "Total Collection items in Calibre: 0" in another collection for the same book.

I do not know whether I have been able to communicate that to you, but would be happy to elaborate if needed.