[GUI Plugin] Kobo Utilities

[davidfor]

Originally Posted by Grady Cole

I'm on 10.8.5, and I'm running in 64-bit.

Also just got this error when I opened the book in the calibre reader:

calibre, version 2.26.0
ERROR: Unhandled exception: <b>IOError</b>:[Errno 32] Broken pipe

calibre 2.26  isfrozen: True is64bit: True
Darwin-12.6.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '12.6.0', 'Darwin Kernel Version 12.6.0: Wed Dec 17 19:11:40 PST 2014; root:xnu-2050.48.15~1/RELEASE_X86_64')
Python 2.7.9
OSX: ('10.8.5', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: DeDRM && KePub Output && Kobo Utilities && KoboTouchExtended && Modify ePub
Traceback (most recent call last):
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 209, in main
    return run_entry_point()
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
    return getattr(pmod, func)()
  File "site-packages/calibre/utils/ipc/worker.py", line 199, in main
IOError: [Errno 32] Broken pipe

It's all falling apart! hahaha

That definitely isn't a problem caused by the utilities plugin. With the other errors, it makes me think there is something wrong with your calibre installation. I suggest uninstalling and reinstalling. This doesn't affect the library or configuration. If this doesn't fix things, post a more general question with the errors.

[chrisridd]

I also see the failure to store book positions and failure to backup the database, on OS X 10.10.3 running Calibre 2.26 (64-bit) with the current Kobo Extended plugin.

I can read books in the Calibre reader though, so my install isn't as busted as Grady's :-)

[PeterT]

I wonder if (yet again) this is related to OSX being out of file handles? See http://docs.basho.com/riak/latest/op...imit/#Mac-OS-X

[davidfor]

The file handles might be part of it, but there is also a problem with the logging. The device jobs handle this differently to other jobs. I have seen this problem before and the fix was to reduce the logging. In this case I had added an extra statement that more than doubled the logging and forgot to remove it. I have fixed this now with version 2.0.7 which will be available in the next hour or so.

I also fixed a problem with the backup if one of the files was missing. That won't be a common problem, but my Glo HD didn't have an affiliates file when I first connected.

[Grady Cole]

David, you rock. I deleted calibre and downloaded a new copy and the book opens fine (but the Kobo backup didn't work). Downloaded the new plugin, restarted calibre and the Kobo backup worked.

Everything is now good. Thanks for your help, and for developing an awesome plugin.

[chrisridd]

It works for me too, thanks David!

lsof reports the main calibre process using 181 file descriptors, which seems high compared to the process soft limit of 256, but most of that seems down to Qt and python and thus not easily fixed.

I'd be worried if the plugin needed 75 file descriptors!

[davidfor]

Good to hear it worked.

As to the file descripters, I really hope it isn't using 75. The line I removed was writing the log in a different way, so it might have used a couple of extras. It also would have meant that a lot of extra log messages were being written. I have seen a similar looking error on Windows and assumed it was a buffer filling up somewhere. Reducing the messages fixed this, but this time it seems to be something different.

[chrisridd]

I don't know how you test the plugin David, but I noticed when playing in Xcode last night that Xcode has a higher soft limit of file descriptors than default.

That makes sense - Xcode users may need lots of open files - but that also means that anything run directly by Xcode inherits those increased resource settings. If you're testing inside Xcode perhaps that's why the problem doesn't show up?

Read 'man getrlimit' to find out how to read and change the limits.

[davidfor]

No, I do all my calibre work under Windows. From memory, the file handle limits are a lot higher than that under Windows. So, unless Python or the rest of calibre is using a lot, that explains why I am not seeing the problem.

[chrisridd]

OK. I don't remember how Windows works.

If it happens again, I'll see if I can figure out what went wrong.

[kovidgoyal]

I've added code to increase the limit on calibre startup on unix.
https://github.com/kovidgoyal/calibr...82e4f5cbd322d5

I dont usually like to do things like that with OS controlled policy, but the limit is too low on OS X and there have been a fair number of reports of errors caused by this limitation.

[davidfor]

Kovid: I can understand the desire not to fiddle with this sort of thing, but in this case it makes sense. Thanks.

[chrisridd]

I watched calibre 2.26 opening/closing files via a short dtrace script - and noticed some quite strange behaviour. (I wrote the script in case one of David's driver updates has issues again.)

Essentially, every book cover in the library seems to get opened twice (and then closed, so no fd leaks.) Do you know why that might be? It doesn't seem very efficient to keep hitting the filesystem.

Calibre also seems to read the covers (again, double opens and closes) of a number of other books every time you select a book in the main list. It feels like it might be filling a sort of cache (read-ahead), but then when you select one of the other books that got opened, calibre goes and reopens the cover (twice) anyway. Odd.

Here's a bit of the output of my script to show you what I mean about the double opens/closes:

open /Users/cjr/Calibre Library/Alastair Reynolds/Revelation Space (210)/cover.jpg succeeded: fd 36
open /Users/cjr/Calibre Library/Alastair Reynolds/Revelation Space (210)/cover.jpg succeeded: fd 37
close /Users/cjr/Calibre Library/Alastair Reynolds/Revelation Space (210)/cover.jpg fd 36
close /Users/cjr/Calibre Library/Alastair Reynolds/Revelation Space (210)/cover.jpg fd 37
open /Users/cjr/Calibre Library/Arkady Strugatsky/Roadside Picnic (133)/cover.jpg succeeded: fd 36
open /Users/cjr/Calibre Library/Arkady Strugatsky/Roadside Picnic (133)/cover.jpg succeeded: fd 37
close /Users/cjr/Calibre Library/Arkady Strugatsky/Roadside Picnic (133)/cover.jpg fd 36
close /Users/cjr/Calibre Library/Arkady Strugatsky/Roadside Picnic (133)/cover.jpg fd 37

[kovidgoyal]

All books covers should never be opened, only a few books "around" the current book. Different components in the UI open the covers separately, the grid view, the cover browser, the book details panel. The first two also maintain small in memory caches (they have to be separate since the sizes of the displayed covers are different). However, if you hide both the cover browser and the grid view, then you should be seeing only cover opens coming from the book details panel which should be one at a time.

If that does not explain what you are seeing, let me know and I will take a look.

[kovidgoyal]

Never mind, there was indeed a bug causing the double opens, now fixed.