[GUI Plugin] Zotero Metadata Importer

[DaltonST]

Zip posted in prior post.

[BetterRed]

Soon after I installed Calibre and I saw Windows PE in debug's Windows info I posted a query about it. Someone, probably Kovid, told me calibre gets that information from Trolltechs Qt package. Calibre reports Windows PE, on my home brew systems and on my Dell Alien. IIRC Speccy differentiates by slapping 'OEM' after the license number on the Alien - currently on loan so I can't check.

My knowledge of this is a bit rusty but I think the following still holds true:

No matter what the status of the Windows licensee, OEM or Retail, the first thing that will be installed is the 32bit Pre-installation Environment.

In the case of Retail the install process will then automatically and immediately proceed to installing the relevant full version of Windows.

If all you had was 32bit Windows PE you probably couldn't even install calibre, because amongst other things there's no Trusted Installer - msiexec and all that.

BR

[davidfor]

@DaltonST, OK, then lets track what everyone said:

Originally Posted by DaltonST

https://technet.microsoft.com/en-us/...(v=ws.10).aspx

Your very first post shows that you are using Windws 8 PE. See the above article.


Windows Preinstallation Environment (Windows PE) 2.0 is a minimal Win32 operating system with limited services, built on the Windows Vista kernel. It is used to prepare a computer for Windows installation, to copy disk images from a network file server, and to initiate Windows Setup.

Windows PE is not designed to be the primary operating system on a computer, but is instead used as a standalone preinstallation environment and as an integral component of other setup and recovery technologies, such as Setup for Windows Vista, Windows Deployment Services (Windows DS), the Systems Management Server (SMS) Operating System (OS) Deployment Feature Pack, and the Windows Recovery Environment (Windows RE).

There is no explanation with this post. You don't say why it is relevant to the problem. The only conclusion that I can make is that you were telling zoterocalibrnote that the reason things were not working was because they were running Windows 8 PE. If that is not the case, I will offer my apologies, but I would be very interested to know what you did mean. Of course, in your later quoting, you highlighted a line. That backs up my conclusion. A simple statement or question in this post would have helped everyone know what you are after.

Originally Posted by PeterT

Dalton; I'd suggest you look at the logs other people have posted over the years. The presence of WindowsPE in the log does NOT mean the user is running WindowsPE.

An easy statement to check as I'd seen it in the logs but had never bothered to track down exactly what it meant. And PeterT is just trying to point out that the entry is very unlikely to be part of the problem.

Originally Posted by DaltonST

Preinstalled.

Short and sweet unlike my "long winded" comment. But, the lack of explanation means that no-one reading the post knows why you think this is important.

Originally Posted by PeterT

Whatever. You obviously know everything so what the heck.

A bit rude, and I can understand why you would take objection to it.

Originally Posted by DaltonST

Research indicated that seeing WindowsPE means that the OS came pre-installed on the pc from the maker. Google knows everything.

Which is where I came in. I knew I shouldn't, but I wanted to know why this was important.

Originally Posted by davidfor

So? Why would the fact Windows was pre-installed have an affect on whether your plugin ran? Or calibre ran properly. Or any well designed Windows application worked. Or, for that matter, any badly designed Windows application. Remember, to most people, a computer is just a tool. They go into a shop pick one, take it home and plug it in. It works. Windows will be installed by the manufacturer, and they won't change it. They just want to use the computer. Most PCs out there are probably running a pre-installed copy of Windows.

And just to confirm, I have to older laptops here that I manually installed Windows on after replacing the hard-drive. One is running Window 7, the other Windows 10. On both, calibre shows 'WindowsPE'. And both run everything I have thrown at them with no problems.

Hmm, "longwinded", probably. "preachy" maybe, but I'd prefer the judgement to come from an impartial third-party. "Irrelevent"? Not to your statements. But, depending on exactly what your point was, maybe it is.

But, the question I asked is valid. I have no idea why using a preinstalled version of Windows would affect how an application ran. If there is a reason, I would be very interested in knowing what it is.

Originally Posted by PeterT

I'd like to add that on my PC which is hand built, and the o/s installed by me from Microsoft supplied media, the calibre log also shows WindowsPE.

Originally Posted by DaltonST

@davidfor:

Nobody said that. Ever.

So, what were you saying? You made two posts with no explanation of why they were relevant to the problem at hand. And one which was seemed to be explaining the others, but didn't really do that.

As usual, you interject comments that are out of context and have nothing to do with anything.

Hmm, I haven't made many posts in your threads, but I would have said all were in context. My post here was in context as it was a direct response to your posts. Are you saying your posts was out of context?

Since you jumped to the last post in the thread without reading the beginning, I will explain it to you now.

Now you are making claims about me that you have absolutely no knowledge of. I read most of the posts in this thread. When I read your post about "Windows 8 PE", backtracked to see if I had missed anything as I couldn't see the context. And I started a reply to that post. But, I took my time to check some things and watch an event on the Olympics, and the "discussion" had moved on. So, I moved my "longwinded, preachy" and, apparently, "irrelevant" comment to the last post and commented on that.

And my comment was a direct response to the post. I was asking why you thought that preinstalled Windows was relevant to the problem ZoteroCalbrNote was having.

My post to ZoteroCalbrNote:
If ZoteroCalbrNote were running PE itself, that would kill ZMI.

But, you didn't actually say that. You commented on an entry in a log and quoted something from elsewhere without actually showing it was quote. Then when someone pointed out (a bit clumsily) that the log entry didn't mean what you were suggesting, you replied with statements which seemed to be even less likely to be the issue.

And just to be clear: I think the likelihood of someone running"Windows 8 PE" also running calibre and the other applications necessary to be seeing this problem is tiny. And I think the probability of pre-installed Windows system causing this sort of problem is even smaller.

Then, PeterT said:

To which I replied:

To which PeterT replied:
To which I replied:
And to ZoteroCalibrNote I said:
To which you, davidfor, as usual, made longwinded, preachy, and irrelevant comments.

I make no apologies that my posts can be long winded. But, compared to the early version of some of them, including the one you are commenting on, they are concise.

I do take exception to "preachy". But, if an impartial third party comes along and tells me that my posts do come off that way, then I'll accept it. But, I would also ask for some examples so that I could decide what I had to do.

"Irrelevant", no. It is completely relevant to the series of posts I was responding to. I was making an apparently useless attempt to find out the relevance off those posts to the problem ZoteroCalbrNote was having.

Then, PeterT said:
To which I now reply: "That is because you installed it from Microsoft supplied media, just like a PC maker (including those who make it themselves) does".

So, neither of you two said anything of value whatsoever in solving ZoteroCalbrNote's issue.

And neither had you. You made a comment without explanation that I could only think YOU thought was relevant.

Quit wasting my time. Neither of you use Zotero and ZMI.

You're right, I don't use either plugin. That doesn't mean I don't understand what is happening or can't have a suggestion about how to fix a problem. Personally, I'm willing to take suggestions from any interested enough to make them.

Was I wasting your time? Not as much as you would have if tried to get ZoteroCalbrNote to look into "Windows 8 PE" and "Preinstalled" as a cause of the problem. And if you can explain how the latter could be relevant (the former would be but is a misinterpretation on your part of the log), will willingly apologise for poking my nose it.

And just so this isn't completely a waste of time...

Why don't you add some logging statements into a version of the plugin so that the debug log gives you a better idea of what is actually happening. Dumping the values of variables frequently explains to me why some code doesn't follow the path I expected.

And to everyone else, my apologies for being long winded. And if I have been preachy, please point it out so I know what I have done. As to relevance, it isn't relevant to the plugin or the problem ZoteroCalbrNote is having. But it is a direct response to a what I consider is an attack on me, so it is relevant in that context.

[DaltonST]

@ZoteroCalbrNote:

Only as a temporary workaround (we still need to get to the bottom of it), you can do this to jump-start your use of Calibre with your Zotero books and metadata:

[1] Set your ZMI options to create empty books if no file attachments are found.

[2] Run ZMI in auto-step mode with your 'real' CSV file.

[3] Finish the updating of the new empty books' metadata in ZMI by clicking the remaining pushbuttons etc.

[4] Navigate to your Zotero 'storage' folder.

[5] Drag-and-drop each .pdf onto the Calibre Book Detail pane (on the right) for the appropriately selected "empty book". That will be easy, since the Zotero storage files have the Zotero Key in their path, and Calibre has it in the ZKey Custom Column. You can also match on author and title, of course. Calibre will ask you if you really want to add that new format. Say "yes".

You will then have your Zotero .pdf files in Calibre with all of the appropriate ZMI-created Calibre metadata.

You can begin to work in Calibre with your Zotero files just as if your issue did not exist.

When you have the time, run the special ZMI plugin .zip file I uploaded previously just for you to use while running in debug mode, then upload its entire debug log for me.

I am going to figure this out. You have Windows 10. I have Windows 10. My ZMI works perfectly. User @petzi's works perfectly in OSX. User @excaliber's works perfectly in Windows 10. Yours should too.


Thanks.


DaltonST

[pleiadesc]

@DaltonST

helo sir! i found your plugin zotero import to calibre, will it be possible to reverse the process? i put all the books in calibre and update its metadata there, after that i export it in zotero for reading and annotations. i tried the bibtex trick, just maybe a plugin will streamline the process more.

TIA!

[BetterRed]

Moderator Notice
@pleiadesc - I moved your post from the plugin ideas thread to here, where you are almost guaranteed get a response from DaltonST, whereas in the ideas thread you're hoping he spots it

BR

[pleiadesc]

@BetterRed

Thank you sir! i was going to put it here but it was closed, thank you for moving it here.

Thank you!

[BetterRed]

Originally Posted by pleiadesc

Thank you sir! i was going to put it here but it was closed ...

That's odd I wonder who closed it, I didn't wittingly open it.

I think DaltonST must be taking a break, he's normally pretty quick to answer queries in his PI threads.

BR

[DaltonST]

Originally Posted by pleiadesc

@DaltonST

helo sir! i found your plugin zotero import to calibre, will it be possible to reverse the process? i put all the books in calibre and update its metadata there, after that i export it in zotero for reading and annotations. i tried the bibtex trick, just maybe a plugin will streamline the process more.

TIA!

@pleiadesc:

ZMI is a Calibre plug-in that uses a standard user-generated Zotero .csv export file as its primary source of data. It also reads Zotero's document storage folders plus does a scan of a few of its SQLite tables in order to compare what Calibre has to what Zotero has to show the user what is missing from Calibre and still needs to be imported into Calibre via ZMI. It does not update Zotero at all.

Updating Zotero itself entails using its ever-changing browser-oriented API, which constantly evolves as JavaScript, HTML, CSS, web browsers, and related technologies and standards evolve.

Zotero does not use Python, in which all 11 of my Calibre plug-ins are written. For that and other reasons, I lost interest in creating a Zotero plug-in several months ago after looking into doing so.

I suggest that you request that a native Zotero plug-in be created and maintained by a Zotero plug-in developer via https://www.zotero.org/support/ .



DaltonST

[pleiadesc]

Thank you DaltonST! its too sad that calibre and zotero does not work together, i think they complement each other.

I'll try asking the zotero support.

Thank you!

[bitnik]

Hi DaltonST,

got small problem importing pdf using ZMI, I need your support.

My configuration is as follow:

Windows 10 Pro 32 bit [language ita]
Zotero Standalone vers. 4.0.29.10 [language ita]
Calibre 32 bit vers. 2.77.0 [language eng]
ZMI plugin vers. 1.0.35 [language eng]

Locations:
Zotero profile : in standard location C:\Users\roby_\AppData\Roaming\Zotero\Zotero\Profi les\
Zotero library : in standard location C:\Users\roby_\AppData\Roaming\Zotero\Zotero\Profi les\xv7l7kj8.default\zotero\storage\
Calibre library : in standard location C:\Users\roby_\Documents\Biblioteca di calibre

Steps performed:
Zotero : Export library as CSV file
Calibre : Import CVS file using Auto-Step Mode
ZMI Options: Automatically validate... CHECKED
Enable Auto-Step.. CHECKED
Create an 'empty book'... CHECKED
Create Tags from ZAutomatic... CHECKED
Create Tags from ZManual... CHECKED

Results achieved:
all metadata are correctly in place BUT pdf file is not uploaded, instead the default .txt shows in book windows.

Further checks:
Zfile Attachments: shows the correct location for pdf file, checked via copy and past the address into windows explore, the pdf opens.
CVS file: shows the correct location for pdf file, checked via copy and past the address into windows explore, the pdf opens.

Other configuration tested:

• Win 10 32 bit [language ita] + Calibre [language ita], same result achieved, metadata in place, no pdf available
• Win 10 64 bit [language ita] + Calibre [language ita], same result achieved, metadata in place, no pdf available
• Win 10 64 bit [language ita] + Calibre [language eng], same result achieved, metadata in place, no pdf available
• Run Calibre as administrator (various combination as above) same result achieved, metadata in place, no pdf available.
• Tested different locations for zotero library , either inside o outside standard profile in C:\, and using different drives same result achieved, metadata in place, no pdf available.

Every alternative configuration has been tested rebuilding every time Zotero library and creating an empty Calibre library.

For the main configuration described at the begining of the post i provide the following:

• - CSV file
• - Zotero profile (library enclosed in "storage" folder
• - Calibre debug file

The question is:
Do you have any suggestion on how to make ZMI import pdf ?

Thank
Rob

[DaltonST]

@bitnik:

This phenomenon has happened to another Win10 user. I have never been able to reproduce it. The workaround is to be found in the Original Post in the section entitled "Workaround for File Attachment Issues:", which is a simple Drag-And-Drop procedure.

Given your excellent documentation of your problem, perhaps I can reproduce it this time.

I suspect it may have to do with a ZMI read-error of the C:\Users\roby_\AppData\Roaming\Zotero\Zotero\Profi les\xv7l7kj8.default\zotero\storage\ folder.

My storage folder is a much simpler and has a much shorter path, and none of my directories have a .default or any other file extension. Perhaps that is why I could not reproduce it. I will explore that scenario. In the meantime, please use the stated Workaround.

I will get back with you about this once I have solved it.


DaltonST

[bitnik]

Thanks DaltonST,

as mentioned in my post I already tried different Zotero Library location with a much more short path, however, this solution did not result in import pdf files.

I am aware of the workaround solution as i read the entire thread many times.

In my case this solution will be very time consuming as I have thousand on pdf to manage.

Waiting for your further feedback and possibile solution.

salut.

Rob

[DaltonST]

Version 1.0.36 - 2017-01-14 Technical changes to account for variability in Zotero .csv file data.




DaltonST

[bitnik]

DaltonST,

tested ZMI new version on the main configuration (on top of previous post).

It works !!

Great job!!

In the next few days I'll test it on other configuration.

For the time beeing thank a lot and have a nice weekend.

Rob