Beta testers for calibre 3.0 needed

[kovidgoyal]

@JimmXinu: Thanks, good to know

@DivingDuck: Those PDF files generated covers from the first page for me in my windows calibre 64 bit install. Can you run calibre in debug mode and post the log when you add one of those PDF files. (Right click the preferences button to restart in debug mode).

[Divingduck]

Here it is:

Spoiler:

calibre Fehlerdiagnoseprotokoll
calibre 2.99.3 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.14393-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.14393')
Python 2.7.12+
Windows: ('10', '10.0.14393', 'SP0', u'Multiprocessor Free')
Successfully initialized third party plugins: DeDRM (6, 5, 3) && KoboTouchExtended (2, 8, 2) && Goodreads (1, 1, 12) && Goodreads Sync (1, 10, 1) && Open With (1, 5, 10) && Find Duplicates (1, 6, 2) && ScrambleEbook (0, 0, 5) && Job Spy (1, 0, 55) && food network Recipe Reader (1, 0, 2) && KindleUnpack - The Plugin (0, 81, 4) && Extract ISBN (1, 4, 4) && Import List (1, 1, 5) && Manage Sony x50 Reader Book List (1, 2, 11) && Generate Cover (1, 5, 21) && EpubMerge (2, 1, 0) && Count Pages (1, 7, 0) && EpubSplit (2, 2, 0) && Media File Importer (1, 0, 17) && Library Codes (1, 0, 29) && Wiki Reader (2, 1, 0) && Quality Check (1, 9, 11) && Modify ePub (1, 3, 13) && BOL_DE (0, 7, 5) && Search The Internet (1, 7, 6) && Diaps Editing Toolbag (0, 3, 4) && Annotations (1, 4, 11) && Favourites Menu (1, 0, 4)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 90.0253164557 x 90.1013513514
Starting up...
DEBUG:    0.0  ~~~~~~~~~~~~~~~~~~~~~ Annotations plugin 1.4.11 (Windows) ~~~~~~~~~~~~~~~~~~~~~~
DEBUG:    0.0 libiMobileDevice:__init__() 
DEBUG:    0.0 AnnotationsAction:genesis(ERROR) Error loading library libiMobileDevice: LoadLibrary() argument 1 must be string, not unicode
DEBUG:    0.0 AnnotationsAction:init_options(disable_caching: False) 
DEBUG:    0.0 AnnotationsDB:connect(db_version: 1) 
DEBUG:    0.0 AnnotationsAction:load_dynamic_reader_classes() 
DEBUG:    0.0   loading built-in class 'GoodReader'
DEBUG:    0.0   loading built-in class 'Kindle'
DEBUG:    0.0   loading built-in class 'KoboTouch-new'
DEBUG:    0.1   loading built-in class 'KoboTouch-old'
DEBUG:    0.1   loading built-in class 'KoboTouch'
DEBUG:    0.1   loading built-in class 'ParseKindleMyClippingsTxt'
DEBUG:    0.1   loading external class 'AGE31'
DEBUG:    0.1 AnnotationsAction:inflate_dialog_resources() 
DEBUG:    0.1 CompileUI:compile_ui() 
DEBUG:    0.1   recompiling C:\Users\Armin\AppData\Roaming\calibre\plugins\annotations_resources\dialogs\cc_wizard.ui
DEBUG:    0.2   recompiling C:\Users\Armin\AppData\Roaming\calibre\plugins\annotations_resources\dialogs\new_destination.ui
DEBUG:    0.0 HttpHelper::__init__: proxy=None
Calibre, and hence Job Spy, was gracefully shut down last time?  True
Last time daemon started:  Thu Feb 16 09:34:56 2017
Last time daemon failed:  never
Total daemon starts inception_to_date:  1
Total daemon failures inception-to-date:  0
Job Spy has finished initialization...
EpubMerge: DEBUG: 2017-05-12 18:54:20,940: calibre_plugins.epubmerge.epubmerge_plugin(152): macmenuhack file_path:C:\Users\Armin\AppData\Roaming\calibre\plugins\fanficfare_macmenuhack.txt
Started up in 8.90 seconds with 95 books
pdfinfo returned no UTF-8 data
Traceback (most recent call last):
  File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\customize\ui.py", line 416, in get_file_type_metadata
  File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\customize\builtins.py", line 342, in get_metadata
  File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\ebooks\metadata\pdf.py", line 115, in get_metadata
ValueError: Could not read info dict from PDF
Added UTP1007 to db in: 1.2
Added 1 books in 2.6 seconds
MFI self.tmp_directory never used; returning True for shutdown.
DEBUG:   30.4 AnnotationsAction:shutting_down()

[kovidgoyal]

The problem seems to be pdfinfo is returning non UTF-8 data on your system, despite passing the -enc UTF-8 option to it. Probably a bug in this version of poppler (which is where pdfinfo comes from).

Can you run the followoing in the direcotory containing the pdf file

calibre-debug -c "from calibre.ebooks.metadata.pdf import *; pdfinfo, pdftoppm = get_tools(); subprocess.call([pdfinfo, '-enc', 'UTF-8', 'UTP1007.pdf'])" > output.txt

and post the output.txt file for me to look at.

[theducks]

There is a weird Count pages issue
I have it set to auto update the columns (no user needed )
The PI ends its run, but the columns do noy show the values
UNTIL
I click a new selection
(I think this is a screen refresh issue)

[Divingduck]

Never mind, cool.

Here it is:

Spoiler:

Title:          Daring Duckies (Design Pack) - Thread List
Subject:        These duckies are real characters! Cute on bath decor and more.
Keywords:       machine embroidery design pack daring duckies rubber duck ducky toy toys
Producer:       Winnovative HTML to PDF Converter 6.4
CreationDate:   Sun May 25 19:32:46 2014 Mitteleuropäische Sommerzeit
Tagged:         no
UserProperties: no
Suspects:       no
Form:           none
JavaScript:     no
Pages:          9
Encrypted:      no
Page size:      612 x 792 pts (letter)
Page rot:       0
File size:      730347 bytes
Optimized:      no
PDF version:    1.4

[kovidgoyal]

I need the actual file not a copy paste as I have to check why UTF-8 decoding is failing.

[Divingduck]

I attach it in post #140

[kovidgoyal]

Ah the culprit is the localized date string, this should take care of it: https://github.com/kovidgoyal/calibr...66becc9e0cc062

[Divingduck]

Thanks. I will test as soon you release the next beta. (Or can I runt it from source?)

[kovidgoyal]

You can run from source, but you have to checkout the vs2015 branch.

[Divingduck]

Perfect, works like charm. Thanks again.

[Martina Schein]

I just tried to insert a Word document (docx) into Calibre. I received the following error message:

Spoiler:

calibre, version 2.99.3 (win32, embedded-python: True)
Konvertierungsfehler: Fehlgeschlagen: Buch 1 von 1 (Grundgesetz (GG)) konvertieren

Buch 1 von 1 (Grundgesetz (GG)) konvertieren
C:\Program Files\Calibre2\app\pylib.zip\dateutil\parser.py:60 1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
Conversion options changed from defaults:
verbose: 2
remove_paragraph_spacing_indent_size: 1.0
no_default_epub_cover: True
margin_left: 0.0
output_profile: 'tablet'
minimum_line_height: 100.0
epub_flatten: True
replace_scene_breaks: u'\u2217 \u2217 \u2217'
margin_right: 0.0
base_font_size: 16.0
flow_size: 4096
chapter_mark: u'none'
extra_css: u'body {page-break-before: always;}\np div {orphans: 3; widows: 3;}\nh1, h2, h3, h4, h5, h6 {-webkit-hyphens: none !important; hyphens: none !important;}'
font_size_mapping: u'5.0, 7.0, 9.0, 12.0, 13.5, 17.0, 20.0, 22.0, 24.0'
filter_css: u',font-family'
insert_metadata: True
margin_top: 0.0
read_metadata_from_opf: u'C:\\Users\\DARKAN~1\\AppData\\Local\\Temp\\calib re_edb0mz\\wcqt_w.opf'
preserve_cover_aspect_ratio: True
margin_bottom: 0.0
Resolved conversion options
calibre version: 2.99.3
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 16.0,
'book_producer': None,
'change_justification': u'original',
'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., '\\s*((chapter|book|section|part)\\s+)|((prolog|pr ologue|epilogue)(\\s+|$))', 'i')) or @class = 'chapter']",
'chapter_mark': u'none',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'docx_inline_subsup': False,
'docx_no_cover': False,
'docx_no_pagebreaks_between_notes': False,
'dont_split_on_page_breaks': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': True,
'epub_inline_toc': False,
'epub_toc_at_end': False,
'expand_css': False,
'extra_css': u'body {page-break-before: always;}\np div {orphans: 3; widows: 3;}\nh1, h2, h3, h4, h5, h6 {-webkit-hyphens: none !important; hyphens: none !important;}',
'extract_to': None,
'filter_css': u',font-family',
'fix_indents': True,
'flow_size': 4096,
'font_size_mapping': u'5.0, 7.0, 9.0, 12.0, 13.5, 17.0, 20.0, 22.0, 24.0',
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x00000208AA328BA8>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': True,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0.0,
'linearize_tables': False,
'margin_bottom': 0.0,
'margin_left': 0.0,
'margin_right': 0.0,
'margin_top': 0.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 100.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': True,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.TabletOutput object at 0x00000208AA332630>,
'page_breaks_before': u'/',
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': True,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': u'C:\\Users\\DARKAN~1\\AppData\\Local\\Temp\\calib re_edb0mz\\wcqt_w.opf',
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.0,
'renumber_headings': True,
'replace_scene_breaks': u'\u2217 \u2217 \u2217',
'search_replace': '[]',
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': None,
'sr1_search': None,
'sr2_replace': None,
'sr2_search': None,
'sr3_replace': None,
'sr3_search': None,
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': '[]',
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: DOCX Input running
on C:\Users\DARKAN~1\AppData\Local\Temp\calibre_edb0m z\lkxdhj.docx
Settings word/word/settings.xml do not exist
Styles word/word/styles.xml do not exist
Numbering styles word/word/numbering.xml do not exist
Converting Word markup to HTML
Python function terminated unexpectedly
'Styles' object has no attribute 'default_paragraph_style' (Error Code: 1)
Traceback (most recent call last):
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site.py", line 101, in main
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site.py", line 78, in run_entry_point
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\utils\ipc\worker.py", line 195, in main
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\gui2\convert\gui_conversion.py", line 33, in gui_convert_override
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\ebooks\conversion\plumber.py", line 1088, in run
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\customize\conversion.py", line 245, in __call__
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\ebooks\conversion\plugins\docx_in put.py", line 34, in convert
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\ebooks\docx\to_html.py", line 122, in __call__
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\ebooks\docx\to_html.py", line 405, in convert_p
File "C:/t/t/frozen-n_3rl3/winfrozen/app/Lib\site-packages\calibre\ebooks\docx\styles.py", line 225, in resolve_paragraph
AttributeError: 'Styles' object has no attribute 'default_paragraph_style'

Addendum:
I have tested a few more Word documents. The error I have mentioned occurs when I have previously created this file from a PDF document. For this I use the software "PDF Converter Professional 8".

[eschwartz]

Originally Posted by kovidgoyal

@eschwartz: There isn't -- I decided to add support for Virtual libraries in the UI instead. Trying to mix that with also having per user VL restrictions is too messy.

You can restict users to libraries, but not restrictions within a library.

Thanks for the explanation.

[eschwartz]

Originally Posted by rashkae

This is actually pretty big breakage to my exiting use case. I'll work around it simply by creating a server start script that clones a new copy of the library just for the server, but a server option, (or alternate binary) that does not modify database would be ideal.

Originally Posted by kovidgoyal

I'm afraid that is not going to happen. What is your use case? Remember that you can now connect to a running server with calibredb. ABout the only thing you cant do now, that you could before is run calibre-server and the GUI at the same time. Instead you have to run the server embedded in the GUI.

Originally Posted by rashkae

The server auto starts and runs as a service.. The gui is run when I'm logged in as desktop app.. pretty clear distinction there.

I dealt intend to deal with this by creating wrapper scripts that run things against the server I have launched as a systemd service.

Though I have always used a wrapper script for `calibre`, to make it run calibre-debug -g and log to a /tmp file.

...

Kovid, I cannot figure out how to run either `calibre` or `calibre-debug -g` with the necessary options passed.

calibre-debug: error: no such option: --with-library

calibre: error: no such option: --username

Also, I keep getting errors when I try running calibredb list as a test against the server.

[eschwartz@arch ~]$ calibredb list --with-library "http://0.0.0.0:8080/#Amazon" --username XXXXXX --password XXXX
Traceback (most recent call last):
  File "/usr/bin/calibredb", line 20, in <module>
    sys.exit(main())
  File "/home/eschwartz/git/calibre/src/calibre/db/cli/main.py", line 251, in main
    return run_cmd(cmd, opts, args[1:], DBCtx(opts))
  File "/home/eschwartz/git/calibre/src/calibre/db/cli/main.py", line 55, in run_cmd
    ret = m.main(opts, args, dbctx)
  File "/home/eschwartz/git/calibre/src/calibre/db/cli/cmd_list.py", line 324, in main
    for_machine=opts.for_machine
  File "/home/eschwartz/git/calibre/src/calibre/db/cli/cmd_list.py", line 150, in do_list
    ans = dbctx.run('list', fields, sort_by, ascending, search_text, limit)
  File "/home/eschwartz/git/calibre/src/calibre/db/cli/main.py", line 189, in run
    return self.remote_run(name, m, *args)
  File "/home/eschwartz/git/calibre/src/calibre/db/cli/main.py", line 209, in remote_run
    res = self.br.open_novisit(rq)
  File "/usr/lib/python2.7/site-packages/mechanize/_mechanize.py", line 239, in open_novisit
    url_or_request, data, visit=False, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/mechanize/_mechanize.py", line 284, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "/usr/lib/python2.7/site-packages/mechanize/_opener.py", line 195, in open
    response = urlopen(self, req, data)
  File "/usr/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 352, in _open
    '_open', req)
  File "/usr/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 340, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 1185, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 1154, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 880, in _send_output
    msg += message_body
UnicodeDecodeError: 'utf8' codec can't decode byte 0x95 in position 0: invalid start byte

[kovidgoyal]

@Martina: I dont think that;s a bug in the beta, it would have happened in calibre 2.x as well. Regardless, this should fix it: https://github.com/kovidgoyal/calibr...c70f08cd89936b