Calibre keeps crashing on me

ctop · 02-07-2020, 08:10 PM

Hi there,
I have a large Library of about 13500 ebooks, mostly EPUB. I repeatedly made the checks under Library maintenance and also updated the Metadata with the "Embed Metadata" action. Now I want to send the Library to my Device, but calibre keeps crashing on me. The transfer works OK, but at the end (I assume it is when the metadata.calibre file is generated) it crashes.
To nail down the problem I created a Folder device to speed things up (which did not really work faster), but the problem remains.
I installed the very latest Calibre (4.10.1) and started in Debug mode. With this version, I still observe the crash, but at least now the Debug log contains some useful information, here are the last lines:

Code:

DEBUG:   33.4 USBMS: Fetching list of books from device. Device= FOLDER_DEVICE oncard= None
DEBUG:   33.4 USBMS: dirs are: /home/chris/00scratch/test-device/ 
exception during JSON decode_from_file
Traceback (most recent call last):
  File "site-packages/calibre/ebooks/metadata/book/json_codec.py", line 179, in decode_from_file
  File "json/__init__.py", line 291, in load
  File "json/__init__.py", line 352, in loads
  File "json/decoder.py", line 364, in decode
  File "json/decoder.py", line 382, in raw_decode
ValueError: No JSON object could be decoded
DEBUG:   33.4 USBMS: scan from root False /home/chris/00scratch/test-device/
Job: 0 Get device information finished
No details available.
Job: 1 Set library information finished
No details available.
Traceback (most recent call last):
  File "site-packages/calibre/customize/ui.py", line 428, in get_file_type_metadata
  File "site-packages/calibre/customize/builtins.py", line 194, in get_metadata
  File "site-packages/calibre/ebooks/metadata/epub.py", line 238, in get_quick_metadata
  File "site-packages/calibre/ebooks/metadata/epub.py", line 217, in get_metadata
  File "site-packages/calibre/ebooks/metadata/epub.py", line 165, in get_zip_reader
  File "site-packages/calibre/ebooks/metadata/epub.py", line 148, in __init__
  File "site-packages/calibre/ebooks/metadata/epub.py", line 99, in __init__
  File "site-packages/calibre/ebooks/metadata/epub.py", line 152, in open
  File "site-packages/calibre/utils/localunzip.py", line 271, in open
TypeError: 'NoneType' object is not iterable
WARNING: Invalid mimetype declaration application/epub+zip
Traceback (most recent call last):
  File "site-packages/calibre/customize/ui.py", line 428, in get_file_type_metadata
  File "site-packages/calibre/customize/builtins.py", line 194, in get_metadata
  File "site-packages/calibre/ebooks/metadata/epub.py", line 238, in get_quick_metadata
  File "site-packages/calibre/ebooks/metadata/epub.py", line 217, in get_metadata
  File "site-packages/calibre/ebooks/metadata/epub.py", line 165, in get_zip_reader
  File "site-packages/calibre/ebooks/metadata/epub.py", line 148, in __init__
  File "site-packages/calibre/ebooks/metadata/epub.py", line 99, in __init__
  File "site-packages/calibre/ebooks/metadata/epub.py", line 152, in open
  File "site-packages/calibre/utils/localunzip.py", line 271, in open
TypeError: 'NoneType' object is not iterable
WARNING: Invalid mimetype declaration application/epub+zip
Traceback (most recent call last):
  File "site-packages/calibre/customize/ui.py", line 428, in get_file_type_metadata
  File "site-packages/calibre/customize/builtins.py", line 194, in get_metadata
  File "site-packages/calibre/ebooks/metadata/epub.py", line 238, in get_quick_metadata
  File "site-packages/calibre/ebooks/metadata/epub.py", line 217, in get_metadata
  File "site-packages/calibre/ebooks/metadata/epub.py", line 165, in get_zip_reader
  File "site-packages/calibre/ebooks/metadata/epub.py", line 148, in __init__
  File "site-packages/calibre/ebooks/metadata/epub.py", line 99, in __init__
  File "site-packages/calibre/ebooks/metadata/epub.py", line 152, in open
  File "site-packages/calibre/utils/localunzip.py", line 271, in open
TypeError: 'NoneType' object is not iterable

So I guess there is still a book somewhere which has invalid metadata. Could this have been produced with the process that send the files to the device? That process also crashed, but it was still 4.8.0 and no information was in the debug log.
So basically, I wonder if I am understand the problem correctly, and what to do about it. If there is a problem with a book, how can I find out which one? I added the first book to this Library in 2009, so there is definitely a certain amount of cruft, but I would hate to completely give up the Library.

All the best,

Ctop

kovidgoyal · 02-07-2020, 09:15 PM

This is on what OS?

DNSB · 02-07-2020, 09:17 PM

I'd suggest doing your transfer in smaller blocks, say, 512 books per block. When/if the transfer abends, break that block into 2 blocks and keep doing so until you have your problem book located closely enough to start checking individual books.

So 512, 256, 128, 64, 32, 16 <= at 16 is where I would check the books individually. For the epubs, I would suggest using either Sigil or calibre's editor with the epubcheck plugin for the testing. Though from your posted debug output, it seems that the problem files have issues with the mimetype file which should be uncompressed in the root of the epub's zip container.

ctop · 02-07-2020, 09:17 PM

Quote:

Originally Posted by kovidgoyal

This is on what OS?

I am on a Ubuntu 19.10 machine here.

Ctop

ctop · 02-07-2020, 09:22 PM

Quote:

Originally Posted by DNSB

I'd suggest doing your transfer in smaller blocks, say, 512 books per block. When/if the transfer abends, break that block into 2 blocks and keep doing so until you have your problem book located closely enough to start checking individual books.

Yeah, thought about that and actually made a few tests with various batches of books that looked somehow suspicious but that did not turn up anything. But doing it the way you suggested would be pretty time consuming. I had hoped that there would be a way of getting information about the book Calibre is crashing on from Calibre...

All the best,
Ctop

kovidgoyal · 02-07-2020, 09:23 PM

get a backtrace

echo bt | coredumpctl gdb

And are you using the official binaries or the distro package?

ctop · 02-07-2020, 10:26 PM

Quote:

Originally Posted by kovidgoyal

get a backtrace

echo bt | coredumpctl gdb

And are you using the official binaries or the distro package?

I just issued the above command after another crash, but I got

"No match found"

I am using the latest official binary installed just 2 hours ago.

All the best,

Ctop

kovidgoyal · 02-07-2020, 11:24 PM

Then you need to figure out how to create backtraces on your system, the above command was standard for systemd based linux systems.

ctop · 02-08-2020, 06:21 AM

OK, here is an update on the situation. While I could not create a backtrace, I observed that the crashing always occurred at the end of the transfer, after all books had already been copied and this job is finished.
I think next is the phase, where the metadata are written to .metadata.calibre In all cases I observed, this file had a length of zero bytes. When I batched the transfer in smaller portions, I could see this file grow to more than 200MB, but this is still with about 5000 books. So the file will grow much larger as more books are added. So my hunch was that somehow this file was growing unexpectedly big and caused the routine writing it to crash the whole program.
I then went to investigate the file and discovered a whole lot of columns that I did not use. They had been created by the plugin ZMI (Zotero Metadata Importer) when I tried to import some metadata that I am maintaining in Zotero. In the end, this did not work and so these columns are not necessary. I removed them from the Library database and tried the whole export once again. This time it worked! I could export all the books to the device and can see them there and also in Calibre!
I am not sure if this observation is correct, but at least it solved the problem for me.

https://mobileread.cdn.speedyrails.n...t/thankyou.gif

Thanks for all your suggestions,
Ctop

davidfor · 02-08-2020, 08:16 AM

@ctop: Which device was this happening with? And how big is the metadata.calibre now?

ctop · 02-08-2020, 08:18 PM

Quote:

Originally Posted by davidfor

@ctop: Which device was this happening with? And how big is the metadata.calibre now?

On my Likebook Ares it is now about 130MB with ca. 16500 books.

02-07-2020, 09:17 PM	#3
DNSB Bibliophagist Posts: 45,312 Karma: 168808723 Join Date: Jul 2010 Location: Vancouver Device: Kobo Sage, Libra Colour, Lenovo M8 FHD, Paperwhite 4, Tolino epos	I'd suggest doing your transfer in smaller blocks, say, 512 books per block. When/if the transfer abends, break that block into 2 blocks and keep doing so until you have your problem book located closely enough to start checking individual books. So 512, 256, 128, 64, 32, 16 <= at 16 is where I would check the books individually. For the epubs, I would suggest using either Sigil or calibre's editor with the epubcheck plugin for the testing. Though from your posted debug output, it seems that the problem files have issues with the mimetype file which should be uncompressed in the root of the epub's zip container. Last edited by DNSB; 02-07-2020 at 09:20 PM.

02-08-2020, 06:21 AM	#9
ctop Connoisseur Posts: 63 Karma: 43710 Join Date: Jun 2008 Device: zaurus->palm->iPad->Sony PRS-T1,T2,T3->KoboForma&Likebook Ares->Palma2	Calibre keeps crashing on me [SOLVED] OK, here is an update on the situation. While I could not create a backtrace, I observed that the crashing always occurred at the end of the transfer, after all books had already been copied and this job is finished. I think next is the phase, where the metadata are written to .metadata.calibre In all cases I observed, this file had a length of zero bytes. When I batched the transfer in smaller portions, I could see this file grow to more than 200MB, but this is still with about 5000 books. So the file will grow much larger as more books are added. So my hunch was that somehow this file was growing unexpectedly big and caused the routine writing it to crash the whole program. I then went to investigate the file and discovered a whole lot of columns that I did not use. They had been created by the plugin ZMI (Zotero Metadata Importer) when I tried to import some metadata that I am maintaining in Zotero. In the end, this did not work and so these columns are not necessary. I removed them from the Library database and tried the whole export once again. This time it worked! I could export all the books to the device and can see them there and also in Calibre! I am not sure if this observation is correct, but at least it solved the problem for me. https://mobileread.cdn.speedyrails.n...t/thankyou.gif Thanks for all your suggestions, Ctop

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre keeps crashing! :(	adrenaline	Conversion	2	08-15-2019 12:12 PM
Calibre Keeps Crashing?	QueenFreewood	Calibre	2	05-23-2015 08:57 AM
sigil 7.4 and maverick crashing crashing crashing	Carrigboy	Sigil	103	09-04-2014 11:47 AM
Calibre 8.6.5 crashing	Zeebra	Calibre	9	08-30-2012 11:54 AM
Calibre 0.7.4 crashing	rosehrt	Calibre	4	06-22-2010 12:06 AM

02-07-2020, 09:15 PM	#2
kovidgoyal creator of calibre Posts: 45,251 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	This is on what OS?

02-07-2020, 09:23 PM	#6
kovidgoyal creator of calibre Posts: 45,251 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	get a backtrace echo bt \| coredumpctl gdb And are you using the official binaries or the distro package?

02-07-2020, 11:24 PM	#8
kovidgoyal creator of calibre Posts: 45,251 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Then you need to figure out how to create backtraces on your system, the above command was standard for systemd based linux systems.

02-08-2020, 08:16 AM	#10
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	@ctop: Which device was this happening with? And how big is the metadata.calibre now?

Advert

Advert