Set Library Information / JSON serializable error

[PeterT]

Connecting my Nexus 7 (2012) with 3.5.9 of CC and 2.13 (64 bit) of calibre I'm getting the following error in the calibre job log for job Set Library Information

Spoiler:

datetime.datetime(2014, 12, 25, 16, 6, 35, tzinfo=tzutc()) is not JSON serializable

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 480, in _set_library_information
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 1560, in set_library_info
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 600, in _call_client
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 520, in _json_encode
File "json\__init__.py", line 243, in dumps
File "json\encoder.py", line 207, in encode
File "json\encoder.py", line 270, in iterencode
File "json\encoder.py", line 184, in default
TypeError: datetime.datetime(2014, 12, 25, 16, 6, 35, tzinfo=tzutc()) is not JSON serializable

and the device disconnects.

Two additional jobs seem to run properly; Get list of books on device and Send metadata to device.

CC log file sent to you.

[chaley]

Something strange here. I don't see why there should be any datetime values there.

Could you post the metadata_db_prefs_backup.json file? It is saving the metadata for all your columns, and I want to see what that is. If you prefer, you can send it to cc_debug_logs at charles dot haleys dot org.

[PeterT]

eMail sent

[chaley]

This is interesting.

Calibre contains a dictionary "field_metadata" that contains descriptions of every field (column) in the library. It is supposed to contain only metadata, not values of the fields. Unfortunately the description of the field #kobolastread has had a value inserted, which is never supposed to happen. You can see this by looking at the field metadata section of metadata_db_prefs_backup.json.

I don't know how one goes about inserting a value into field_metadata. The only way I can find to do it is something like:

  fm = field_metadata.get('#kobolastread')
  fm['#value#'] = something

In this case 'something' is a JSON-encoded datetime value that looks like:

     {
        "__class__": "datetime.datetime", 
        "__value__": "2014-12-22T23:47:57+00:00"
      }

so perhaps the code looks like

  fm = field_metadata.get('kobolastread')
  json_encoded = JsonCodec().encode_book_metadata(book)
  fm['#value'#] = json_encoded['#kobolastread']['#value#']

Does the problem go away if you start calibre then connect CC without having connected your kobo?

[PeterT]

Restarted calibre.
Did a connect as wireless device from cc.
Worked perfectly!
Deleted some books from the tablet and disconnected.
Then connected Kobo Glo.
Transferred some books and disconnected.
Reconnected cc as wireless device.
Same error occurred.

[PeterT]

I sense there is some problem with davidfor's Kobo Utilities and it's function for retrieving and storing last read data from Kobo.

I initially tried disabling that function (I should note I am using a beta version of his driver that supports multiple Kobo devices), then connecting my Glo, followed by cc. All worked properly.

Next I enabled that function, and much to my surprise it continued to work properly. Thinking it over I realized that none of the books on the Kobo had been read since the last connection was made. I flipped the pages in a book and then tried again.

SUCCESS (or do I mean FAILURE). The error reoccurred.

As one final test I restarted calibre, attached the Glo, then CC. This again worked properly since there were no reading stats changed.

[chaley]

Originally Posted by PeterT

I sense there is some problem with davidfor's Kobo Utilities and it's function for retrieving and storing last read data from Kobo
...

I am glad to hear that, because I don't see any reasonable way to protect the rest of calibre (in my case the wireless driver) from bad entries in field_metadata. The only thing I can think of is to loop through the dictionary and remove any values before trying to use it. In the case of the wireless driver the performance hit is minimal because the operation is done once at connect or when the library changes while connecting. However to do something like this to protect all of calibre would require removing the value every time FM is fetched, which is a *lot* of times.

As you have a work-around and as you and davidfor work together frequently, I will put this on a back burner until you tell me that it cannot be fixed in any other way.

BTW: I was the original implementer of the field_metadata stuff. Feel free to ask questions or discuss alternatives.

[PeterT]

Thanks Will let you know what we find. At least it's reproducible at will which is normally the first step in tracking down bugs!

[PeterT]

OK. Have given David a range of logs; both with his beta version of the utilities and the production version. Glad to say the error happens with either version.

[chaley]

I implemented a workaround that is now in source. It will be in the next calibre release. I can't guarantee it really prevents the exception because I can't repeat it, but I think it will.

[PeterT]

I just tried downloading the current source (and yes.. I verified that the change you made https://github.com/cbhaley/calibre/c...8b10cc1a37213d was in the driver) and repeated tests while running from source.

Still receive the same error.

Spoiler:

datetime.datetime(2014, 12, 29, 21, 52, 8, tzinfo=tzutc()) is not JSON serializable

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 87, in run
  File "site-packages\calibre\gui2\device.py", line 480, in _set_library_information
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 1565, in set_library_info
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 605, in _call_client
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 525, in _json_encode
  File "json\__init__.py", line 243, in dumps
  File "json\encoder.py", line 207, in encode
  File "json\encoder.py", line 270, in iterencode
  File "json\encoder.py", line 184, in default
TypeError: datetime.datetime(2014, 12, 29, 21, 52, 8, tzinfo=tzutc()) is not JSON serializable

I also did verify that the version info displayed was [2.14]*[64bit].

[chaley]

Originally Posted by PeterT

I just tried downloading the current source (and yes.. I verified that the change you made https://github.com/cbhaley/calibre/c...8b10cc1a37213d was in the driver) and repeated tests while running from source.

...

Traceback (most recent call last):
...
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 525, in _json_encode
...
TypeError: datetime.datetime(2014, 12, 29, 21, 52, 8, tzinfo=tzutc()) is not JSON serializable[/code][/spoiler]

I also did verify that the version info displayed was [2.14]*[64bit].

Something hasn't gone right with running from source. The above trace says that the exception went through line 525 of the driver. After the patch that should be line 526.

Did you forget to restart calibre after pulling the changes?

[PeterT]

Consider it user error... I had miss set CALIBRE_DEVELOP_FROM; I forgot to add the src directory to the end of the variable's value.

[PeterT]

Also, davidfor has tracked down the problem and fixed the plugin (at least the beta version of it).