[Conversion Output] KePub Output Plugin

[davidfor]

@jgoguen If you can pass the files on to me, I can test whether it is a Windows issue or not.

[Leonatus]

Originally Posted by davidfor

That error is exactly the same error you saw with the conversion. As you were using the extended driver, it was doing the work of changing the epub to a kepub. The driver and kepub output plugin share code, and the error occurred in exactly the same line.

That has been my suspicion.

To test if this is the temporary directory, open a command prompt and do the following:

SET CALIBRE_TEMP_DIR=C:\TEMP
calibre-debug -g

You will need to create the directory or change it to one that already exists. The seconds command runs calibre in debug mode. You can then try the conversion and see what happens.

Isn't this what I have done, following jgoguen's instructions from#205, by using Calibre's "restart in debug mode" option?

[davidfor]

Originally Posted by Leonatus

Isn't this what I have done, following jgoguen's instructions from#205, by using Calibre's "restart in debug mode" option?

The second line is the equivalent of restarting in debug mode. But, that won't change what is used for the temp directory. You need to set the environment variable followed by starting calibre in the same context. You can set it globally, but if the temporary directory isn't the problem, then there is no reason to.

[Leonatus]

Originally Posted by jgoguen

I'm not sure if it's the temporary directory since it's doing other write operations on files before it fails. Unfortunately, this looks to be Windows-specific (and possibly even 32-bit Windows-specific); on macOS 10.12.2 with calibre 2.77.0 the book I got converts perfectly fine.

The strange is that conversion works on my other Windows 10, 32 bit Computer. I tried it on a third Computer, Windows 10, 64 bit, and there, it did also work without issues.

I did see some older mailing list posts suggesting that lxml will break if you have anything other than Unicode or ASCII characters, and the file that's being referenced in the error you sent me has 0xfeff as the last (invisible) character on line 12. That character used to be a zero-width non-breaking space, but its use in that context is deprecated (use 0x2060 instead) and is currently standardised as the byte order marker which should only appear as the very first character in a file if it's present at all.

Since this is specific to Windows, there's not going to be much I can do. I can offer suggestions, and based on what you tell me I may be able to come up with something that works, but I can't promise anything. To start with, use calibre to edit the book (right-click, choose "Edit Book"), open the file "@public@vhost@g@gutenberg@html@files@14105@14 105-0-0.txt_split_000.html", and remove the last character on line 12; if you look at the bottom right you should see it say "ZERO WIDTH NO-BREAK SPACE : Line: 12 : 78". Backspace once, it should look like nothing changes but the bottom right should change. Save the book, try to convert, it'll probably fail again but it should mention a different file right before the UnicodeWarning line. If that's what happens, I may be able to work around it but it will be some additional pre-processing of each file. If that doesn't work, I guess it's back to the drawing board to figure out what to try next.

Well, I'll do my very best, but I doubt that this is the culprit, because the Problem occurs persistedly, with all other books in the library. I tested about 10 books to convert - always the same.

[Leonatus]

@Davidfor: This is the result following your instructions:

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. Alle Rechte vorbehalten.

C:\WINDOWS\system32>SET CALIBRE_TEMP_DIR=C:\TEMP

C:\WINDOWS\system32>calibre-debug -g
calibre 2.77  embedded-python: True is64bit: False
Windows-8-6.2.9200 Windows ('32bit', 'WindowsPE')
('Windows', '8', '6.2.9200')
Python 2.7.9
Windows: ('8', '6.2.9200', '', 'Multiprocessor Free')
Successfully initialized third party plugins: KePub Output (2, 7, 1) && Generate Cover (1, 5, 21) && Kobo Utilities (2, 5, 2) && Quality Check (1, 9, 11) && Diaps Editing Toolbag (0, 3, 2) && Hyphenate This! (0, 0, 8) && DOC Input (1, 0, 1) && Resize Cover (1, 0, 2) && Modify ePub (1, 3, 13) && Count Pages (1, 6, 9) && Open With (1, 5, 10) && Clean Metadata (0, 0, 6)
devicePixelRatio: 1.0
logicalDpi: 144.0 x 144.0
physicalDpi: 225.777777778 x 225.777777778
Starting up...
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.0 KoboUtilites::dialogs.py - loading translations
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    3.8 No Kobo Touch, Glo or Mini appears to be connected
DEBUG:    3.8 rebuild_menus - self.supports_ratings=None, self.supports_tiles=None
DEBUG:    3.8 KoboUtilities:set_toolbar_button_tooltip - start: text='None'
DEBUG:    3.8 device_driver_name - could not load extended driver. Exception= No module named kobotouch_extended.device.driver
DEBUG:    3.8 KoboUtilities:set_toolbar_button_tooltip - setting to text='Utilities to use with Kobo ereaders

Driver: KoboTouch'
DEBUG:    3.8 device_driver_name - could not load extended driver. Exception= No module named kobotouch_extended.device.driver
Started up in 6.21 seconds with 137 books

[davidfor]

@Leonatus: You need to try the conversion while running in that mode. That will tell us if the problem is related to the temporary directory.

[Leonatus]

@jgoguen: This is the error message that came after the changes you recommended. I think there has nothing changed, but you may judge better.

calibre, version 2.77.0 (win32, embedded-python: True)
Konvertierungsfehler: Fehlgeschlagen: Buch 1 von 1 (Im grünen Tann) konvertieren

Buch 1 von 1 (Im grünen Tann) konvertieren
C:\Program Files\Calibre2\pylib.zip\dateutil\parser.py:339: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
Resolved conversion options
calibre version: 2.77.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0.0,
 'book_producer': None,
 'change_justification': u'original',
 'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., '\\s*((chapter|book|section|part)\\s+)|((prolog|prologue|epilogue)(\\s+|$))', 'i')) or @class = 'chapter']",
 'chapter_mark': u'pagebreak',
 'comments': None,
 'cover': u'C:\\Users\\JRGEN~1\\AppData\\Local\\Temp\\calibre_d0gwsh\\5ltfz_.jpeg',
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_split_on_page_breaks': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_toc_at_end': False,
 'expand_css': False,
 'extra_css': u'@page {margin: 5pt}',
 'extract_to': None,
 'filter_css': u'',
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x064C5AB0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'kepub_clean_markup': True,
 'kepub_disable_hyphenation': False,
 'kepub_hyphenate': True,
 'kepub_replace_lang': True,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0.0,
 'linearize_tables': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.KoboReaderOutput object at 0x064C5ED0>,
 'page_breaks_before': u'/',
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': u'C:\\Users\\JRGEN~1\\AppData\\Local\\Temp\\calibre_d0gwsh\\ztxsfl.opf',
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': u'',
 'search_replace': '[]',
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': None,
 'sr1_search': None,
 'sr2_replace': None,
 'sr2_search': None,
 'sr3_replace': None,
 'sr3_search': None,
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': '[]',
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: EPUB Input running
on C:\Users\JRGEN~1\AppData\Local\Temp\calibre_d0gwsh\oypusq.epub
Parsing all content...
Parsing OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-3.txt.html ...
Parsing OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-4.txt.html ...
Parsing OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-5.txt.html ...
Parsing OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-1.txt.html ...
Parsing OEBPS/Styles/pgepub.css ...
CSSStylesheet: CSSCharsetRule only allowed at beginning of stylesheet. [8:1: @charset ]
Parsing OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-2.txt.html ...
Parsing OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-0.txt.html ...
Reading TOC from NCX...
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Found 1067 items of level: p_1
p_1  left margin stats: Counter({u'0': 1063, u'0%': 4})
p_1  right margin stats: Counter({u'0': 1063, u'0%': 4})
Cleaning up manifest...
Trimming unused files from manifest...
Creating KePub Output...
Splitting markup on page breaks and flow limits, if any...
	Looking for large trees in OEBPS/Text/%40public%40vhost%40g%40gutenberg%40html%40files%4014105%4014105-0-3.txt.html...
	No large trees found
	Looking for large trees in OEBPS/Text/%40public%40vhost%40g%40gutenberg%40html%40files%4014105%4014105-0-4.txt.html...
	No large trees found
	Looking for large trees in OEBPS/Text/%40public%40vhost%40g%40gutenberg%40html%40files%4014105%4014105-0-5.txt.html...
	No large trees found
	Looking for large trees in OEBPS/Text/%40public%40vhost%40g%40gutenberg%40html%40files%4014105%4014105-0-1.txt.html...
	No large trees found
	Looking for large trees in OEBPS/Text/%40public%40vhost%40g%40gutenberg%40html%40files%4014105%4014105-0-2.txt.html...
	No large trees found
		Splitting on page-break at id=id00008
		Splitting on page-break at id=id00009
	Looking for large trees in OEBPS/Text/%40public%40vhost%40g%40gutenberg%40html%40files%4014105%4014105-0-0.txt.html...
	No large trees found
	Split into 2 parts
Copying file 'C:\Users\JRGEN~1\AppData\Local\Temp\calibre_d0gwsh\tuw3b0_KePubOutputPluginInfo' to 'C:\Users\JRGEN~1\AppData\Local\Temp\calibre_d0gwsh\6hzgz2_epub_container' as 'plugininfo.kte'
{'clean_markup': True, 'replace_lang': True, 'smarten_punctuation': False, 'hyphenate': True, 'extended_kepub_features': True, 'no-hyphens': False}
KoboTouchExtended:common:modify_epub:Found cover image ID 'cover'
KoboTouchExtended:common:modify_epub:Setting cover-image property
Forcing cleanup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-0.txt_split_000.html
Forcing cleanup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-0.txt_split_001.html
Forcing cleanup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-1.txt.html
Forcing cleanup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-2.txt.html
Forcing cleanup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-3.txt.html
Forcing cleanup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-4.txt.html
Forcing cleanup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-5.txt.html
Forcing cleanup for file titlepage.xhtml
Cleaning markup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-0.txt_split_000.html
Cleaning markup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-0.txt_split_001.html
Cleaning markup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-1.txt.html
Cleaning markup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-2.txt.html
Cleaning markup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-3.txt.html
Cleaning markup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-4.txt.html
Cleaning markup for file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-5.txt.html
Cleaning markup for file titlepage.xhtml
Copying file 'C:\Users\JRGEN~1\AppData\Local\Temp\calibre_d0gwsh\kepub_rihxqj_hyphenate' to 'C:\Users\JRGEN~1\AppData\Local\Temp\calibre_d0gwsh\6hzgz2_epub_container' as 'kte-css/hyphenation.css'
Adding reference to kte-css/hyphenation.css to file OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-0.txt_split_000.html
C:\Program Files\Calibre2\pylib.zip\ntpath.py:532: UnicodeWarning: Unicode unequal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
Python function terminated unexpectedly
  All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 132, in main
  File "site.py", line 109, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 195, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 33, in gui_convert_override
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 1226, in run
  File "calibre_plugins.kepubout.conversion.kepub_output", line 131, in convert
  File "calibre_plugins.kepubout.common", line 103, in modify_epub
  File "calibre_plugins.kepubout.container", line 178, in add_content_file_reference
  File "lxml.etree.pyx", line 1445, in lxml.etree._Element.makeelement (src\lxml\lxml.etree.c:49848)
  File "apihelpers.pxi", line 133, in lxml.etree._makeElement (src\lxml\lxml.etree.c:14869)
  File "apihelpers.pxi", line 121, in lxml.etree._makeElement (src\lxml\lxml.etree.c:14748)
  File "apihelpers.pxi", line 268, in lxml.etree._initNodeAttributes (src\lxml\lxml.etree.c:16296)
  File "apihelpers.pxi", line 290, in lxml.etree._addAttributeToNode (src\lxml\lxml.etree.c:16701)
  File "apihelpers.pxi", line 1395, in lxml.etree._utf8 (src\lxml\lxml.etree.c:26485)
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

[Leonatus]

Originally Posted by davidfor

@Leonatus: You need to try the conversion while running in that mode. That will tell us if the problem is related to the temporary directory.

Surprise! This time, it worked. What do I have to do now?
I tried with other books in this mode, and it always worked.

[Terisa de morgan]

Originally Posted by Leonatus

Surprise! This time, it worked. What do I have to do now?
I tried with other books in this mode, and it always worked.

Not surprise, it is not the mode but the temp directory, I'm afraid

[jgoguen]

Unfortunately it looks like it's a temporary directory naming problem. Nothing I can do about that unless Kovid has another magical function I don't know about

You'll need to set the calibre temporary directory permanently (search your favourite search engine for "windows 10 set user environment variable") to a path without any characters outside the printable ASCII range.

[kovidgoyal]

Not clear to me why the temporary directory matters? I assume you are only placing relative paths into the XML, so why should the base matter? But if you are placing the temporary directory into the XML, then there should be no problem doing it, as long as it is a unicode object and not a bytesstring.

[Leonatus]

The only non-ascii characters I see is in my user account name:

C:\Users\Jürgen\AppData\Local\Temp

I suppose I would have to change my user account name before changing the environment variable, and as non-technic, this raises some fear in me.

[Terisa de morgan]

@Leonatus, you don't have to change your user name. I think that, if you set in your computer the environment variable CALIBRE_TEMP_DIR as you have done in the test, that would be fine.

[davidfor]

Originally Posted by kovidgoyal

Not clear to me why the temporary directory matters? I assume you are only placing relative paths into the XML, so why should the base matter? But if you are placing the temporary directory into the XML, then there should be no problem doing it, as long as it is a unicode object and not a bytesstring.

There is no intention to put the temporary directory into the XML. It should be using relative paths.

The code in question is (with error checking removed):

        for infile in self.get_html_names():
            root = self.parsed(infile)
            head = root.xpath('./xhtml:head',
                              namespaces={'xhtml': XHTML_NAMESPACE})
            head = head[0]
                 elem = head.makeelement(
                     "{%s}link" % XHTML_NAMESPACE,
                     rel='stylesheet',
                     href=os.path.relpath(
                         name, os.path.dirname(infile)).replace(os.sep, '/'))

"name" will be "kte-css/stylehacks.css" and infile is something like "OEBPS/Text/@public@vhost@g@gutenberg@html@files@14105@14105-0-0.txt_split_000.html" (that's one of Leonatus's examples.) And "self" is a subclass of EpubContainer.

My assumption had been that the path generated and passed as "href" was the issue. But, I can't get it to break And I've pulled it apart into the steps and there is no sign of the temporary directory. I've also created a temporary directory like the one Leonatus has, and it works. This is on both a full 64bit Windows and 32bit Windows.

I'm assuming now that it is something that has been set earlier, but I'll have to track it down tomorrow.

[Leonatus]

Originally Posted by Terisa de morgan

@Leonatus, you don't have to change your user name. I think that, if you set in your computer the environment variable CALIBRE_TEMP_DIR as you have done in the test, that would be fine.

I didn't do nothing but follow davidfor's and jgoguen's advice. the rest was automatic.
As my experience is very poor in these things, I would like to clarify: I go to
1. System
2. advanced system settings
3. Advanced
4. Environment variables
5. In "system variables" highlight "path"
6. Click "edit"
7. Got to "new"
8. and in the highlighted new line enter "C:\CALIBRE_TEMP_DIR"?

Is this correct?

Or, in step 6., should I click "new" instead. There opens a window where I would have to enter Name of the variable"(?) and, below, "value of the variable"(?)