Calibre beta available that fully integrates custom fields

[kacir]

Originally Posted by kovidgoyal

Does anyone else notice the auto backup slowing their calibre installs in 904? You can test it by going to preferences->advanced->misc and clicking backup.

I have clicked on backup. The HD started to blink and Calibre performance got a little bit sluggish. Not much, barely noticeable.
Then I did something ... an user shouldn't do ;-)
I selected tome 3000 books and updated one of my custom tags. At the moment the Calibre Window is greyed out (and has been for the last 10 minutes. I will let you know how it came out.
I am on Mint Linux Isadora (a newest Ubuntu based distro)

[chaley]

Originally Posted by kacir

I have clicked on backup. The HD started to blink and Calibre performance got a little bit sluggish. Not much, barely noticeable.
Then I did something ... an user shouldn't do ;-)
I selected tome 3000 books and updated one of my custom tags. At the moment the Calibre Window is greyed out (and has been for the last 10 minutes. I will let you know how it came out.
I am on Mint Linux Isadora (a newest Ubuntu based distro)

Hmmm....

That is almost certainly happening because the backup is doing database commits. A few releases ago we went to a lot of trouble to ensure that bulk update committed at the end, because re-creating the transaction log was expensive.

Tomorrow I will look at whether I can detect if a transaction is open, and if so, have the backup code wait until it is committed.

[DoctorOhh]

Originally Posted by kacir

I have clicked on backup. The HD started to blink and Calibre performance got a little bit sluggish. Not much, barely noticeable.
Then I did something ... an user shouldn't do ;-)
I selected tome 3000 books and updated one of my custom tags. At the moment the Calibre Window is greyed out (and has been for the last 10 minutes. I will let you know how it came out.

Windows XP SP3 | Calibre 0.7.904

I tried the same thing. I clicked to run the backup and things were fine. I could cruise through the cover browser and the library list with only minor glitches. I selected 9 books and updated a custom column metadata. Then I grabbed the same books and removed the same data. I opened some books and closed them. Everything was within acceptable standards. then I selected all 4018 books and added one tag. 90 minutes later and calibre is still unresponsive but churning up cpu time so I'll leave it alone.

Update: It's been 200+ minutes now and calibre is still churning. Time to kill it and restart. I restarted and the tag I was adding had been successfully added to each of the 4018 books.

[kacir]

Originally Posted by chaley

Hmmm....

That is almost certainly happening because the backup is doing database commits ...

It is 8 hours later and Calibre still does not react.
I will let it run another 10 ...

[Manichean]

I didn't notice anything the first time around. Just let it regenerate the OPFs and played around in the program a bit. System load isn't noticeably higher, and responsivity seems normal.

Edit: XP SP3, 0.7.904, P4 2.6 GHz, 1 GB RAM, should it matter.

Edit 2: Just did some search & replace in some 25 or so books. That did seem to be a tad more sluggish than normal, but barely noticeable.

[kacir]

Originally Posted by kacir

It is 8 hours later and Calibre still does not react.
I will let it run another 10 ...

I came home from work. Calibre was still greyed out. I closed the program and started it again.
It looks like Calibre recovered from the situation very gracefully. I did not have to kill it, I did not have to restart computer, the database does not seem to be damaged, the change I have tried to commit is done.

Thank you dear developers for your hard work.

[Bullwinkle]

Have been using Calibre for approx 1 month, no problems until new 7.2 upgrade.
All of a sudden, file conversions (in this case pdf to ebook) are failing. I am converting individual files; it will correctly do a couple and them I get error messages on the rest.
This is a sample error message: (NB: I have removed specific author & title name)

ERROR: Conversion Error: <b>Failed</b>: Convert book 1 of 1 (TITLE)

Convert book 1 of 1 (TITLE)
Resolved conversion options
calibre version: 0.7.20
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0.0,
'book_producer': None,
'change_justification': u'original',
'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., 'chapter|book|section|part|prologue|epilogue\\s+', 'i')) or @class = 'chapter']",
'chapter_mark': u'pagebreak',
'comments': None,
'cover': 'c:\\docume~1\\mk\\locals~1\\temp\\calibre_0.7.20_ tmp_4jektx\\calibre_0.7.20_3cl4fv.jpeg',
'debug_pipeline': None,
'disable_font_rescaling': False,
'dont_split_on_page_breaks': False,
'extra_css': None,
'extract_to': None,
'flow_size': 260,
'font_size_mapping': None,
'footer_regex': u'(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'header_regex': u'(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'html_unwrap_factor': 0.40000000000000002,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x03F577B0>,
'insert_blank_line': False,
'insert_metadata': False,
'isbn': None,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0.0,
'linearize_tables': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'max_toc_links': 50,
'new_pdf_engine': False,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_images': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.OutputProfile object at 0x03F57990>,
'page_breaks_before': u"//*[name()='h1' or name()='h2']",
'prefer_metadata_cover': False,
'preprocess_html': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': 'c:\\docume~1\\mk\\locals~1\\temp\\calibre_0.7.20_ tmp_4jektx\\calibre_0.7.20_hueki4.opf',
'remove_first_image': False,
'remove_footer': False,
'remove_header': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'unwrap_factor': 0.45000000000000001,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: PDF Input running
on C:\Calibre Library\AUTHOR\TITLE\TITLE, AUTHOR.pdf
Converting file to html...
pdftohtml log:

Retrieving document metadata...
Generating manifest...
Rendering manifest...
Parsing all content...
Parsing index.html ...
Failed to parse content in index.html
Traceback (most recent call last):
File "site-packages\calibre\ebooks\oeb\reader.py", line 159, in _manifest_prune_invalid
File "site-packages\calibre\ebooks\oeb\base.py", line 1060, in fget
File "site-packages\calibre\ebooks\oeb\base.py", line 789, in _parse_xhtml
File "site-packages\calibre\ebooks\conversion\preprocess.py", line 431, in __call__
File "site-packages\calibre\ebooks\conversion\preprocess.py", line 152, in __call__
File "site-packages\calibre\ebooks\conversion\preprocess.py", line 133, in dehyphenate
File "re.py", line 190, in compile
File "re.py", line 245, in _compile
error: unbalanced parenthesis

Spine item 'id1' not found
Python function terminated unexpectedly
Spine is empty (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 103, in main
File "site.py", line 85, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 99, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 24, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 841, in run
File "site-packages\calibre\ebooks\conversion\plumber.py", line 968, in create_oebbook
File "site-packages\calibre\ebooks\oeb\reader.py", line 72, in __call__
File "site-packages\calibre\ebooks\oeb\reader.py", line 594, in _all_from_opf
File "site-packages\calibre\ebooks\oeb\reader.py", line 289, in _spine_from_opf
calibre.ebooks.oeb.base.OEBError: Spine is empty

[chaley]

Originally Posted by Bullwinkle

Have been using Calibre for approx 1 month, no problems until new 7.2 upgrade.
All of a sudden, file conversions (in this case pdf to ebook) are failing. I am converting individual files; it will correctly do a couple and them I get error messages on the rest.
This is a sample error message: (NB: I have removed specific author & title name)

0.7.20 is not the beta release. Are you using the beta? This thread is for discussing that.

[chaley]

Originally Posted by kacir

I came home from work. Calibre was still greyed out. I closed the program and started it again.
It looks like Calibre recovered from the situation very gracefully. I did not have to kill it, I did not have to restart computer, the database does not seem to be damaged, the change I have tried to commit is done.

We found a couple of things.

1) There was definitely some fighting going on between the backup and the bulk edit. Fixed by disabling the backup during bulk edit, and also changing the way bulk edit reports progress on the screen. On my test library of 1000 books, I don't have any problems with performance of bulk edit. Changing tags on all books takes less than 2 seconds. Changing authors and titles is slower (around 1/4 second per book), which is caused by the file system operations.

2) However... the code we added to reselect the books seems not to be optimal. It takes over a minute on my machine to reselect the 1000 books after the bulk edit finishes. This isn't 10 (or however many) hours, but it is annoying. Kovid has fixed it.

3) I had one case (before fixes) where the replace finished, and then everything hung. I am unable to repeat it. It is possible that the former progress-reporting code (multiple threads) could deadlock (although it isn't obvious how), and perhaps that is what was happening. With the thread gone, this problem should be gone.

Kovid is preparing a release that contains all these fixes.

Thank you dear developers for your hard work.

You are very welcome.

[chaley]

Originally Posted by Sweetpea

Oh, in 904 it works, kind of... But no error messages. It only slows down Calibre something horrible.

I found the case where I didn't correctly detect composite fields that reference each other. Fixed. Future definition mistakes shouldn't be so painful.

[Perkin]

Hi guys, I don't know if it's a beta issue, or in the main version as well, but saving to disk has been taking a lot longer than it used to, I think once it starts writing the files it's just as quick. It just seems to take a lot longer to start writing to disk.

[chaley]

Beta release 0.7.905 is now available. See the first post of this thread for details. In particular, that first page has a link to the beta manual. The template tutorial section might be useful to those of you playing with composite fields.

As mentioned above, this release fixes many (we hope all) performance problems related to the backup feature. In particular, it should fix the bulk metadata edit problems.

I anticipate that this will be the last beta release. This series will go to 'production' the end of the week unless a blocking problem surfaces.

Thank you all again for your testing. You do things I never would have thought of.

[kacir]

Another small issue with 904.
I have added a cover image that is lather large jpg file - aproximetely 1800x2800 pixels.
I know. I should have resized it, but
- I am lazy
- I had excuse that I want to see what would happen ;-)

I have the right info panel that displays covers open.

When I put cursor on the book, Calibre quits. Sometimes. I can' reproduce the behavior consistently.

[Manichean]

Originally Posted by chaley

Thank you all again for your testing. You do things I never would have thought of.

So do you developer guys, by the way. And with that, I'm off to test.

... and I mean that in the sense that you constantly amaze us with new features.

[kovidgoyal]

Originally Posted by kacir

When I put cursor on the book, Calibre quits. Sometimes. I can' reproduce the behavior consistently.

You mean you select the book?