Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 06-27-2016, 11:13 AM   #31
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
@petzi:

I have attached a proper Zotero CSV file with only column headings.

Your CSV file apparently has different column headings, including perhaps a column with no heading at all. The latter is likely what caused: "Error: file attachments column not found in dict. Cannot proceed."

Please confirm that your CSV file is the generic Zotero export CSV file with a proper first row with the column headings, and that all columns have a heading.

If there are columns to the far right of column "Legislative Body", which is the last column in the generic CSV file, then ensure they have a column heading too.

ZMI reads the first row for the column headings, then maps each CSV column to the correct #zotero_xxxxx custom column in Calibre. It is okay to add new columns the CSV. They just will not be updated in Calibre until they are on the "official list", which is to say, there is a Zotero custom column to map it to. I am going to change ZMI to more gracefully handle corrupt columns. It will still issue the error message, but then will thereafter ignore that corrupt column.

Be sure to always use the most recent version of ZMI. Calibre updates its list of plug-ins to pick from only once an hour at the bottom of the hour plus a few minutes. If you want it sooner, you must go to the Original Post and manually download and install it via Preferences > Plug-ins > Load from file.


Thanks.


DaltonST


n.b. Mobileread.com will not allow you to upload a file with a file extension of .csv, so change it to .txt instead. It will also not allow a .7z extension, so you have to use .zip instead.
Attached Files
File Type: txt zotero_column_headings_only_csv.txt (1.1 KB, 80 views)
DaltonST is offline   Reply With Quote
Advert
Old 06-27-2016, 11:52 AM   #32
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
Version 1.0.9 New CSV-related error messages

Version 1.0.9 - 2016-06-27 New CSV-related error messages to assist in correcting CSV data issues.

I hope that you never see these new messages.


DaltonST
DaltonST is offline   Reply With Quote
Old 06-27-2016, 05:05 PM   #33
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
Version 1.0.10 - 2016-06-27 Major enhancements.

Version 1.0.10 - 2016-06-27 Major enhancements.

Everyone should upgrade to 1.0.10.


DaltonST
DaltonST is offline   Reply With Quote
Old 06-27-2016, 05:41 PM   #34
petzi
Connoisseur
petzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavens
 
petzi's Avatar
 
Posts: 69
Karma: 13884
Join Date: Dec 2009
Location: Austria
Device: PW2, KOA, iPad3, iPhone 6+
Quote:
[1] Does anything in the CSV work for you? 0%? 50%? 99%? How many books are in the CSV that you are testing?
No, never. Always 0%. The reported trial with the Windows machine I did over Skype with of friend of mine who has Zotero installed, but not Calibre. I'm out of town and have limited access to PCs and internet.

[2] I have attached the .csv file. I exported it from a collection and left the checkmark "Notes" (or Annotation) unchecked. It is a generic file, I have not touched it, opened it or whatsever. I just zipped it. And I have used the newest ZMI version 1.0.9. (It would be nice if you could include the version number in the ZMI dialog window.)

Quote:
[4] You might try using Zotfile to rename the books that have the errors to use only basic characters to see if that fixes the problem.
I have done this in some of my test trials. Just to be sure that there is nothing wrong with the file name. I also worked out some trials with just english filenames. (This was the reason why I exported sometimes only two/three Zotero entries – to have complete control.)

In some of my latest trials (starting with version 1.0.6 I believe) I sometimes got different error messages about a "TECHNICAL FAILURE".

Quote:
calibre, version 2.60.0
ERROR: ZMI: TECHNICAL FAILURE for: calibredb add --library-path=/Users/xxxxx/Documents/Meine E-Books/Calibre/Zotero --identifier zkey:S6IP87AQ "/Users/xxxxx/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/FCQ22QCV/dig_erlass_bl1.pdf"

TECHNICAL FAILURE for: calibredb add --library-path=/Users/xxxxx/Documents/Meine E-Books/Calibre/Zotero --identifier zkey:U48Z5ZX5 "/Users/xxxxx/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/S7JZ3SBF/Internetkompetenz_von SchuelerInnen.pdf"
In these cases I have the relevant files not included in my next trial, e.g. I exported another .csv file without these files. (Attention: these two files are included in the export file I have attached to this post.)

Columns: I have compared your file with my column names. They appear exactly as in my trials. I had no column without a header and I used a generic (default) column setting, when I created the new library. After each trial I deleted the (empty) library and also the folder where the files should go, because there were two metadata files in it. So essentially I started each trial with a complete new session: new library, new .csv-file and so on.

PS.: Due to other obligations I do not know when I will have time to look into this issue again. I hope that I will next Monday online again.
Attached Files
File Type: zip ZMI-Test.csv.zip (23.3 KB, 75 views)
petzi is offline   Reply With Quote
Old 06-27-2016, 08:27 PM   #35
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
@petzi:


Calibredb appears to take in the given filename, and "cleans it" to be harmonious with Calibre. It then adds it to Calibre. However, if several filenames become identical after the cleaning process, then only the very first one will be added to Calibre. Calibre will say "duplicate book", and ignore it. It really is not a duplicate in the Zotero storage folder, but remember that Calibre "cleans it", making duplicates dynamically. Example of leading characters in a filename that calibredb will pretend are not there when given to it via the command-line parameter: "." "_" "._" and similar.


[1] Try renaming your filenames so none start with "." or "_" or any other characters that calibredb will "clean" so that calibredb does not treat the path differently than it was given to it by ZMI per the Zotero CSV. When it treats it differently, it will of course not be able to find the file to import it!


[2] You have many file attachments that have multiple file paths in the same column of the same row, separated by a semicolon (";"). Example:

Code:
/Users/peterb/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/EVS5ZD7H/Austria_piaac-erhebung_2011_12.pdf; /Users/peterb/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/PV6N9NSG/idcplg.pdf
This will not work. One (1) .pdf per CSV row per CSV Key (first column).


[3] You have file attachments with single quotes in their filenames. Calibre will "clean" it, and hence never be able to find it. Example:

._[THESIS TITLE GOES HERE] - 'Georgia' Tech Thesis Office.pdf
is treated by calibredb as if it were:
[THESIS TITLE GOES HERE] - Georgia Tech Thesis Office.pdf


[4] Could you please test this very specific file attachment? I can find nothing wrong with it, and it should work perfectly. If it does not, please send me the debug log for that execution of ZMI.

Quote:
/Users/peterb/Library/Application Support/Firefox/Profiles/6kbcf8zv.Standard/zotero/storage/FG93WTHN/Murray et al_1998_Adult Literacy in OECD Countries.pdf

[5] You have a lot of .html files as attachments. Do you want those imported into Calibre too? Not just .pdf, .epub, etc.?




DaltonST
DaltonST is offline   Reply With Quote
Advert
Old 06-28-2016, 01:48 AM   #36
petzi
Connoisseur
petzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavens
 
petzi's Avatar
 
Posts: 69
Karma: 13884
Join Date: Dec 2009
Location: Austria
Device: PW2, KOA, iPad3, iPhone 6+
Quote:
[4] Could you please test this very specific file attachment?
Didn't work. How can I produce a debug log file? Never done. I invoked "calibre-debug --gui" via the terminal and after that I started the ZMI and produced the error. Then I looked into the terminal window again. But there was no interested information produced:

Quote:
calibre 2.60 embedded-python: True is64bit: True
Darwin-14.5.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '14.5.0', 'Darwin Kernel Version 14.5.0: Thu Apr 21 20:40:54 PDT 2016; root:xnu-2782.50.3~1/RELEASE_X86_64')
Python 2.7.11
OSX: ('10.10.5', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: DeDRM (6, 4, 3) && Zotero Metadata Importer (1, 0, 10) && Favourites Menu (1, 0, 4) && Goodreads Sync (1, 9, 0) && Find Duplicates (1, 6, 2) && EpubMerge (2, 0, 12) && Annotations (1, 4, 7) && Goodreads (1, 1, 11) && View Manager (1, 3, 2)
What exactly do I have to do for producing a debug log?


Quote:
[5] You have a lot of .html files as attachments. Do you want those imported into Calibre too? Not just .pdf, .epub, etc.?
No, just files like .pdf, .epub.
petzi is offline   Reply With Quote
Old 06-28-2016, 05:18 AM   #37
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
@petzi:

Calibre is cross-platform. Use Calibre itself to go into debug mode. Do this:

Right-click the Preferences button. Left Click "Restart in Debug Mode".

Calibre will restart, giving you a message. Run ZMI for that specific file. Then, exit from Calibre. Immediately afterwards, a text file with the debug log will pop open. Please zip it and send it.


Thanks.

DaltonST
DaltonST is offline   Reply With Quote
Old 06-28-2016, 07:18 AM   #38
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
Version 1.0.11 - 2016-06-28 Enhancements.

Version 1.0.11 - 2016-06-28 Enhancements.

Please remember to check Calibre status bar in the lower right corner for a message about which of your installed plug-ins have an update available. Click the message to update ZMI.

You should also add the "Plug-in Updater" to your Calibre ToolBar via Preferences.


CALIBRE_TEMP_DIR:

If you implement CALIBRE_TEMP_DIR, then as a minimum make TMPDIR = CALIBRE_TEMP_DIR.

If you want total flexibility, read the Original Post about Special OS Comments, especial item [5].



DaltonST

Last edited by DaltonST; 06-28-2016 at 03:40 PM. Reason: New information
DaltonST is offline   Reply With Quote
Old 06-28-2016, 01:46 PM   #39
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
Version 1.0.12 - 2016-06-28 Major Enhancements.

Version 1.0.12 - 2016-06-28 Major Enhancements.

I have changed ZMI so that calibredb never looks into Zotero for the books. Instead, ZMI copies them into a temporary directory, sanitizes their names while it still know what their Zotero Key is, and then points calibredb to that temporary directory using their Zotero Key to create an Identifier with which to match their metadata automatically.

This was necessary due to the complexity of Unicode symbols in some Zotero file attachments (books) that otherwise would never be able to be "found" by calibredb.




DaltonST
DaltonST is offline   Reply With Quote
Old 06-28-2016, 01:57 PM   #40
petzi
Connoisseur
petzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavens
 
petzi's Avatar
 
Posts: 69
Karma: 13884
Join Date: Dec 2009
Location: Austria
Device: PW2, KOA, iPad3, iPhone 6+
Quote:
Originally Posted by DaltonST View Post
@petzi:

Calibre is cross-platform. Use Calibre itself to go into debug mode. Do this:

Right-click the Preferences button. Left Click "Restart in Debug Mode".

Calibre will restart, giving you a message. Run ZMI for that specific file. Then, exit from Calibre. Immediately afterwards, a text file with the debug log will pop open. Please zip it and send it.


Thanks.

DaltonST
Thank you for the infomation how to switch into debug mode. Attached is the log file.
petzi is offline   Reply With Quote
Old 06-28-2016, 03:25 PM   #41
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
@petzi:

Thank you. Could you now upgrade to Version 1.0.12 of ZMI that was heavily enhanced using your Zotero CSV as test data? Then, do the testing again?


Thanks.



DaltonST
DaltonST is offline   Reply With Quote
Old 06-28-2016, 03:58 PM   #42
petzi
Connoisseur
petzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavens
 
petzi's Avatar
 
Posts: 69
Karma: 13884
Join Date: Dec 2009
Location: Austria
Device: PW2, KOA, iPad3, iPhone 6+
Sorry, again the same error. Log file attached.
Attached Files
File Type: txt calibre-dbug-zmi-2.txt (1.9 KB, 74 views)
petzi is offline   Reply With Quote
Old 06-28-2016, 04:29 PM   #43
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
@petzi,

Thank you. Excellent.

You need to purge all books from your Zotero Calibre Library before running the test book.

Remember, you must populate your Zotero Calibre Library exclusively with ZMI, and ZMI is evolving rapidly. So, you must do what I do. I delete all of my Zotero Calibre Library books and start over with the exported CSV file. Otherwise, you are mixing apples and oranges (different ZMI versions).

Once ZMI reaches steady-state, you will not have to purge the books every time you run a test. ZMI is less than a week old now, and your test data is what is driving its development at the moment. It can handle all of @excaliber's data, but yours...not yet. But soon.


Thanks again.


DaltonST
DaltonST is offline   Reply With Quote
Old 06-28-2016, 05:02 PM   #44
petzi
Connoisseur
petzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavenspetzi is a rising star in the heavens
 
petzi's Avatar
 
Posts: 69
Karma: 13884
Join Date: Dec 2009
Location: Austria
Device: PW2, KOA, iPad3, iPhone 6+
Actually during the test sessions with you I am always deleting the Zotero Calibre Library and then deleting the folder where still some metadata are stored. I always start form scratch. I do not understand the duplicate message…

Anyway I attached the debug log from the new test iteration.

There is now a more detailed error message. But the hint with the manual update I do not understand. There are no books in my new database to update manually.

[Just another small issue: Whenever I start from scratch I will get the notification from Calibre (calibre-notifier) that some column are missing and I should wait. But actually nothing will happen before I press the ZMI button for “Autovalidate/Generate Current Library…". Only then the missing columns will be generated. No big deal but could user confuse.]
Attached Files
File Type: txt calibre-debug-zmi-3.txt (1.9 KB, 77 views)
petzi is offline   Reply With Quote
Old 06-28-2016, 05:09 PM   #45
DaltonST
Deviser
DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!DaltonST rocks like Gibraltar!
 
DaltonST's Avatar
 
Posts: 1,098
Karma: 100494
Join Date: Aug 2013
Location: Texas
Device: 8" Win10 Tablet w/Calibre64
@petzi:

Your Calibre library metadata.db appears to need fixing. Calibre > Library Button > Library Maintenance > Check Library. Then, tell it to delete all of the invalid entries it finds.

Do this after you have deleted all of the books.

Quote:
But actually nothing will happen before I press the ZMI button for “Autovalidate/Generate Current Library…".
It would be dangerous to automatically do it, since the user may not want to automatically do it. So, it waits to continue. As long as the dialog with the "OK" and "Cancel" buttons is shown, ZMI is waiting for you to approve its next action.


DaltonST
DaltonST is offline   Reply With Quote
Reply

Tags
ris, zotero

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] KindleUnpack - The Plugin DiapDealer Plugins 412 04-24-2018 06:07 PM
[GUI Plugin] Consolidate All Library Metadata (CALM) DaltonST Plugins 188 06-09-2017 11:20 AM
[GUI Plugin] OPML Importer (RSS feeds from feedly/google reader) fem Plugins 14 07-19-2014 10:41 AM
Getting custom metadata field in GUI plugin SauliusP. Development 7 02-11-2012 01:31 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 05:52 AM.


MobileRead.com is a privately owned, operated and funded community.