Calibre beta available that fully integrates custom fields

[atjnjk]

Testing v0.7.903
-Creating and editing a custom column built from other columns: works great.
-Editing a custom column (created by standard release):
ERROR: ERROR: Unhandled exception: <b>KeyError</b>:u'bool'
Traceback (most recent call last):
File "site-packages\calibre\gui2\preferences\columns.py", line 121, in edit_custcol
File "site-packages\calibre\gui2\preferences\create_custom_co lumn.py", line 87, in __init__
KeyError: u'bool'

[chaley]

Originally Posted by kovidgoyal

The problem is the writing to dropbox, which takes a long time, I've actually moved that into a separate thread, so the problem should go away.

Using dropbox doesn't intrinsically slow down the write operations to the filesystem. It does slow down the processor. The dropbox daemon wakes up for every file, chatters over the network, computes a send package, and then sends it. All of this takes reasonable CPU and I/O bandwidth.

On windows, dropbox also (I think) inserts itself into the notify chain, which means that touching a file will wake up the daemon, consuming yet more cycles.

I turn off dropbox when I am making a big change, such as restoring my library, running calibre2opds, or adding lots of books. The daemon really gets in the way, otherwise, making the operations take significantly longer. I turn it back on when I am done with the large updates.

[chaley]

Originally Posted by atjnjk

Testing v0.7.903
-Creating and editing a custom column built from other columns: works great.
-Editing a custom column (created by standard release):
ERROR: ERROR: Unhandled exception: <b>KeyError</b>:u'bool'
Traceback (most recent call last):
File "site-packages\calibre\gui2\preferences\columns.py", line 121, in edit_custcol
File "site-packages\calibre\gui2\preferences\create_custom_co lumn.py", line 87, in __init__
KeyError: u'bool'

Thank you.

The problem will occur only when editing yes/no columns, and is, of course, fixed.

[chaley]

Originally Posted by Perkin

I don't know if this was beta's fault or not.

I just used the beta 0.7.903 to send a load of books to my device, and it wiped the existing collections, and only wrote the newly sent collections.

I'm pretty sure they were intact beforehand.

Edit: I just sent some more books to device and the existing collections were wiped, and only new ones written

Edit2: Metadat management set to 'Only on send', whether that would make any difference?

Thanks for testing.

Turns out that 'only on send' didn't work at all. I fixed that. This problem is serious enough that there is a good chance the we will re-release the beta today.

If necessary, you can get all calibre-generated collections back by switching to automatic management.

[Perkin]

Thanks chaley.

look forward to new beta.

I've noticed another problem, converting a pdf with some settings throws an error, but with others full converts the same pdf

Whereas, the 'normal' calibre, converted the file with either settings.

This is the log of settings throwing the error

Spoiler:

ERROR: Conversion Error: <b>Failed</b>: Convert book 1 of 1 (Falls the Shadow)

Convert book 1 of 1 (Falls the Shadow)
Resolved conversion options
calibre version: 0.7.903
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 11.0,
'book_producer': None,
'change_justification': u'original',
'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., 'chapter|book|prologue|epilogue|glossary|section|p art\\s+', 'i') or re:test(.,'^\\d+$',i)) or @class = 'chapter']",
'chapter_mark': u'none',
'comments': None,
'cover': 'c:\\users\\leigh\\appdata\\local\\temp\\calibre_0 .7.903_tmp_g3k3tz\\calibre_0.7.903_v26evu.jpeg',
'debug_pipeline': None,
'disable_font_rescaling': False,
'dont_split_on_page_breaks': False,
'extra_css': u'body {text-align: justify; text-indent: 0.8em;}',
'extract_to': None,
'flow_size': 2260,
'font_size_mapping': None,
'footer_regex': u'(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'header_regex': u'(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'html_unwrap_factor': 0.40000000000000002,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x046D6E50>,
'insert_blank_line': False,
'insert_metadata': False,
'isbn': None,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 12.0,
'linearize_tables': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'max_toc_links': 50,
'new_pdf_engine': False,
'no_chapters_in_toc': False,
'no_default_epub_cover': True,
'no_images': True,
'no_inline_navbars': False,
'no_svg_cover': True,
'output_profile': <calibre.customize.profiles.SonyReader300Output object at 0x046E0210>,
'page_breaks_before': u'/',
'prefer_metadata_cover': False,
'preprocess_html': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': 'c:\\users\\leigh\\appdata\\local\\temp\\calibre_0 .7.903_tmp_g3k3tz\\calibre_0.7.903_6jvhrn.opf',
'remove_first_image': False,
'remove_footer': False,
'remove_header': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 0.80000000000000004,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'unwrap_factor': 0.0,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: PDF Input running
on D:\Calibre_BETA\Doctor Who\Daniel O'Mahony\Falls the Shadow (150)\Falls the Shadow - Daniel O'Mahony.pdf
Converting file to html...
pdftohtml log:

Retrieving document metadata...
Generating manifest...
Rendering manifest...
Parsing all content...
Parsing index.html ...
Failed to parse content in index.html
Traceback (most recent call last):
File "site-packages\calibre\ebooks\oeb\reader.py", line 158, in _manifest_prune_invalid
File "site-packages\calibre\ebooks\oeb\base.py", line 1060, in fget
File "site-packages\calibre\ebooks\oeb\base.py", line 789, in _parse_xhtml
File "site-packages\calibre\ebooks\conversion\preprocess.py", line 431, in __call__
UnboundLocalError: local variable 'length' referenced before assignment

Spine item 'id1' not found
Python function terminated unexpectedly
Spine is empty (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 103, in main
File "site.py", line 85, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 103, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 24, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 841, in run
File "site-packages\calibre\ebooks\conversion\plumber.py", line 968, in create_oebbook
File "site-packages\calibre\ebooks\oeb\reader.py", line 72, in __call__
File "site-packages\calibre\ebooks\oeb\reader.py", line 593, in _all_from_opf
File "site-packages\calibre\ebooks\oeb\reader.py", line 288, in _spine_from_opf
calibre.ebooks.oeb.base.OEBError: Spine is empty

Where the same pdf with different settings, converts it fine.

Spoiler:

Convert book 1 of 1 (Falls the Shadow)
Resolved conversion options
calibre version: 0.7.903
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 11.0,
'book_producer': None,
'change_justification': u'justify',
'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., 'chapter|book|prologue|epilogue|glossary|section|p art\\s+', 'i') or re:test(.,'^\\d+$',i)) or @class = 'chapter']",
'chapter_mark': u'none',
'comments': None,
'cover': 'c:\\users\\leigh\\appdata\\local\\temp\\calibre_0 .7.903_tmp_g3k3tz\\calibre_0.7.903_zvcgub.jpeg',
'debug_pipeline': None,
'disable_font_rescaling': False,
'dont_split_on_page_breaks': False,
'extra_css': u'body {text-align: justify; text-indent: 0.8em;}',
'extract_to': None,
'flow_size': 2260,
'font_size_mapping': None,
'footer_regex': u'(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'header_regex': u'(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'html_unwrap_factor': 0.40000000000000002,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x046F6E50>,
'insert_blank_line': True,
'insert_metadata': False,
'isbn': None,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 12.0,
'linearize_tables': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'max_toc_links': 50,
'new_pdf_engine': False,
'no_chapters_in_toc': False,
'no_default_epub_cover': True,
'no_images': True,
'no_inline_navbars': False,
'no_svg_cover': True,
'output_profile': <calibre.customize.profiles.SonyReader300Output object at 0x04700210>,
'page_breaks_before': u'/',
'prefer_metadata_cover': False,
'preprocess_html': True,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': 'c:\\users\\leigh\\appdata\\local\\temp\\calibre_0 .7.903_tmp_g3k3tz\\calibre_0.7.903_ctnr4z.opf',
'remove_first_image': False,
'remove_footer': False,
'remove_header': False,
'remove_paragraph_spacing': True,
'remove_paragraph_spacing_indent_size': 0.80000000000000004,
'series': None,
'series_index': None,
'smarten_punctuation': True,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'unwrap_factor': 0.40000000000000002,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: PDF Input running
on D:\Calibre_BETA\Doctor Who\Daniel O'Mahony\Falls the Shadow (150)\Falls the Shadow - Daniel O'Mahony.pdf
Converting file to html...
pdftohtml log:

Retrieving document metadata...
Generating manifest...
Rendering manifest...
Parsing all content...
Parsing index.html ...
Initial parse failed:
Traceback (most recent call last):
File "site-packages\calibre\ebooks\oeb\base.py", line 816, in first_pass
File "lxml.etree.pyx", line 2532, in lxml.etree.fromstring (src/lxml/lxml.etree.c:48270)
File "parser.pxi", line 1545, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:71812)
File "parser.pxi", line 1417, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:70608)
File "parser.pxi", line 898, in lxml.etree._BaseParser._parseUnicodeDoc (src/lxml/lxml.etree.c:67148)
File "parser.pxi", line 539, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:63824)
File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:64745)
File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64088)
XMLSyntaxError: Opening and ending tag mismatch: META line 9 and head, line 10, column 8

Parsing file 'index.html' as HTML
Forcing index.html into XHTML namespace
Generating default TOC from spine...
Merging user specified metadata...
Detecting structure...
Detected chapter: Prologue
Auto generated TOC with 20 entries.
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Cleaning up manifest...
Trimming unused files from manifest...
Parsing stylesheet.css ...
Creating EPUB Output...
Looking for large trees in index.html...
No large trees found
EPUB output written to c:\users\leigh\appdata\local\temp\calibre_0.7.903_ tmp_g3k3tz\calibre_0.7.903_wkm9t8.epub

[DoctorOhh]

Originally Posted by Perkin

I've noticed another problem, converting a pdf with some settings throws an error, but with others full converts the same pdf

Whereas, the 'normal' calibre, converted the file with either settings.

This is an error separate from the beta and introduced in 0.7.20 also. ldolse is relatively sure of the fix and should be submitting it to Kovid for the next release.

You might want to go to the other thread and supply some input so the error can definitely be fixed.

[Perkin]

thanks dwanthny, tha's nice to know.

I hadn't tried the new 7.20 yet, so that's why I missed it and blamed it on the beta.

[DoctorOhh]

Originally Posted by Perkin

thanks dwanthny, tha's nice to know.

I hadn't tried the new 7.20 yet, so that's why I missed it and blamed it on the beta.

The developers only have 12 balls in the air at once, I don't know how anyone could have missed that.

[Perkin]

So there's six male developers and their all doing handstands

[Sweetpea]

I've got a problem with the custom fields:

I've got two books:

Book 1:
Author: Thea Beckman
Title: Kinderen van Moeder Aarde
Series: KvMA
Series Index: 1.0 {I still can't figure out why the series index must be a float instead of integer, but ok...)

Book 2:
Author: Het Helse Paradijs
Title: 02
Series: [empty]
Series Index: [empty]

(this is how Calibre default reads my PDF files, as it won't read my PDFInfo).

I created a new custom field:

{Author} - {Series} {Series_index:0>2s} - {Title}

Book 1 is filled with: Thea Beckman - KvMA 1.0 - Kinderen van Moeder Aarde
Book 2 is filled with: Het Helse Paradijs - 00 - 02

So, the custom field series_index is correctly filled for those books that don't have a series index (at least, leading zero's wise), but the book that has its meta data filled out, doesn't apply the template correctly.

When I fill out the full meta-data for the second book, I get:

Thea Beckman - KvMA 2.0 - Het Helse Paradijs.

so, it loses the leading zero and makes it a silly float again...

[chaley]

Originally Posted by Sweetpea

I've got a problem with the custom fields:
...
so, it loses the leading zero and makes it a silly float again...

Series indices were not being formatted properly in composite fields. They should use the same rules as used in save templates.

Fixed. Thanks for testing.

[PatNY]

I usually stay away from betas, but really wanted to try out the new feature that excludes certain custom data from the content server so I took the plunge. I'm happy to report the feature works perfectly.

Thanks so much, chaley. I really appreciate your putting that option there.

FYI, I was wrong about Stanza not seeing all custom data. It does. What I think happened is that Stanza caches the book info data from the content server. And if you make changes to the book's metadata from within Calibre but don't back out of the download screen in Stanza and query the server again, it will serve up the old book info. But Stanza always retrieves the latest copy of the book itself. That's what threw me off in the testing. And I didn't realize that's how Stanza worked until a few days ago.

BTW, there is a minor bug within this beta which you may already have noticed. When you copy a book to another library, the cover art is not transferred over with the book.

[chaley]

Originally Posted by PatNY

BTW, there is a minor bug within this beta which you may already have noticed. When you copy a book to another library, the cover art is not transferred over with the book.

Did this work before? Library to library copies is one of the very few areas I haven't touched.

Hmmm... perhaps Kovid has. We will see.

Edit: And you are welcome. That was a very reasonable request to provide a feature I should have thought of in the beginning.

[PatNY]

Yes, in the standard releases I've used the cover was always copied over to the other library. It's not a matter of the second library simply not showing the cover. The jpeg is actually missing from the book folder in the target library.

[Starson17]

Originally Posted by chaley

- the first half of an automatic 'backup' scheme. Calibre now writes an opf to the book directory whenever metadata changes. At some point a facility will be provided to rebuild the database from a library's folder hierarchy, using these OPF files. (Kovid)

FYI, I'm seeing opf files only in records that have epub formats. Books with only .txt or .pdf don't get an opf, even if metadata changes. (As I've seen others report, all epubs have opfs, even if no change was made to metadata.) I saw no slowdown during the creation of the opf files. I assume this isn't a bug, just the current design.