weirdness while accessing metadata in plugin

[At_Libitum]

Hi,

I'm a bit stumped as the msgs I am getting aren't making sense.

I have the following bit of code

mi = get_metadata(_stream)
# get a list of all of the metadata fields 
_all_lookup_fields = mi.all_field_keys()
for key in enumerate(_all_lookup_fields):
	self._log(repr(key))
if getattr(mi, 'user_metadata'):
	book.user_metadata = mi.get('user_metadata')

when run it produces the following debug output.

DEBUG:   18.8  (0, 'rating')
DEBUG:   18.8  (1, 'title_sort')
DEBUG:   18.8  (2, 'application_id')
DEBUG:   18.8  (3, 'pubdate')
DEBUG:   18.8  (4, 'series')
DEBUG:   18.8  (5, 'author_sort_map')
DEBUG:   18.8  (6, 'publication_type')
DEBUG:   18.8  (7, 'size')
DEBUG:   18.8  (8, 'author_link_map')
DEBUG:   18.8  (9, 'rights')
DEBUG:   18.8  (10, 'uuid')
DEBUG:   18.8  (11, 'title')
DEBUG:   18.8  (12, 'comments')
DEBUG:   18.8  (13, 'languages')
DEBUG:   18.8  (14, 'cover_data')
DEBUG:   18.8  (15, 'toc')
DEBUG:   18.8  (16, 'thumbnail')
DEBUG:   18.8  (17, 'db_id')
DEBUG:   18.8  (18, 'user_metadata')
DEBUG:   18.8  (19, 'tags')
DEBUG:   18.8  (20, 'timestamp')
DEBUG:   18.8  (21, 'last_modified')
DEBUG:   18.8  (22, 'mime')
DEBUG:   18.8  (23, 'authors')
DEBUG:   18.8  (24, 'publisher')
DEBUG:   18.8  (25, 'series_index')
DEBUG:   18.8  (26, 'lpath')
DEBUG:   18.8  (27, 'user_categories')
DEBUG:   18.8  (28, 'spine')
DEBUG:   18.8  (29, 'author_sort')
DEBUG:   18.8  (30, 'cover')
DEBUG:   18.8  (31, 'book_producer')
DEBUG:   18.8  (32, 'manifest')
DEBUG:   18.8  (33, 'formats')
DEBUG:   18.8  (34, 'guide')
DEBUG:   18.8  (35, 'device_collections')
DEBUG:   18.8  (36, 'identifiers')
DeviceJob: 2 Get list of books on device done, calling callback
Metadata object has no attribute named: 'user_metadata'

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 85, in run
  File "site-packages\calibre\gui2\device.py", line 480, in _books
  File "%tempdir%\reader_overlays.py", line 697, in books
  File "%tempdir%\reader_overlays.py", line 441, in _get_books_from_device
  File "%tempdir%\reader_overlays.py", line 573, in _get_metadata
  File "site-packages\calibre\ebooks\metadata\book\base.py", line 149, in __getattribute__
AttributeError: Metadata object has no attribute named: 'user_metadata'

Now I may know zilch about python, but I am definitely seeing the user_metadata listed when retrieving the present keys, so why does it complain?

the second inconstent response I got when executing this

if hasattr(mi,'identifiers'):
	book.identifiers = mi.get('identifiers')
	_urn = book.identifiers['urn']

u'urn'

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 85, in run
  File "site-packages\calibre\gui2\device.py", line 480, in _books
  File "%tempdir%\reader_overlays.py", line 697, in books
  File "%tempdir%\reader_overlays.py", line 441, in _get_books_from_device
  File "%tempdir%\reader_overlays.py", line 555, in _get_metadata
KeyError: u'urn'

and if I change the hasattr to getattr, lo and behold, no complaint and I even get a value from 'urn' posted to _urn

????

[kovidgoyal]

Metadata objects are rather special, they have overridden attribute access and various other dirty tricks (mainly to maintain legacy compatibility) see book/base.py for the ugly details. What are you trying to do? Maybe I can tell you how to achieve that, easier than explaining the whole thing.

[At_Libitum]

Apologies for the late response. Had downloaded the Calibre python source code and have been burried in the debug window for the better part of the weekend trying to find a solution for the accented characters in filenames. (keep in mind I still know very, very little about python, so am totally not confident using it, let alone adapting existing code). At least that problem is solved, still not much the wiser but did find a way that it didn't complain and that's the main thing.

Ok, back to above issue. What I am trying to do is to lift as much as I can of the metadata out of my books I still got in my Stanza library. Part of this is stored in Stanza's own internal db, parts within the epubs themselves.

Since I have no idea upfront what is stored in the books themselves I am trying to loop through each of the metadata keys I get back from all_field_keys(). And because I have no clue about what content there might be associated with a key (I can assume from the type of key, but some keys can have 0,1 or more entries as in a set. e.g. the User Metadata if I am not mistaken.


All that is fairly straight forward. I would just have expected that if I query which attributes are present in a mi, and then ask for one the I see in the returned list, I get told it ain't there...that just don't make sense to me. Even if it were empty then I would have expected simply a return value of {} or None if it is listed as one of the present attributes.

just as I cannot wrap my head around why a statement with a checking nature would fail where as a statement that actively retrieves (presumably without first checking, why else would there be an hasattr), just goes and pulls what it is supposed to pull. (i.e. the urn attribute)

[kovidgoyal]

If you want to check whether a particular field has a value for a mi object, use mi.is_null(filed_name), with that caveat that this will return True for numeric fields having the value zero as well. Dont use hasattr/getattr and friends.

[At_Libitum]

thanks for the heads-up. Will look at the alternative.

[At_Libitum]

Got another question.

I am doing a

	book.cover = getattr(mi, 'cover', None)

There is also a cover_data and thumbnail attribute. Now I can imagine that cover_data will contain the binary data for the actual cover and thumbnail will contain same in smaller size.

So I am not entirely sure what cover is meant to contain. Most times when I query the value it returns <drive>:\path\to\calibre\cover.xml
the path is likely the current working directory. However, in the OPF 'cover' does not occur as a dc: field but as a member of the manifest and guide section in the OPF. and cover.xml (sometimes it'll be cover.xhtml or cover.html) is then very often the value of the 'cover' entry in these sections.

Is this considered metadata too by Calibre?

[kovidgoyal]

cover is supposed to contain the path to a cover image. cover_data contains cover data and format like this ('jpg', 'xxxxxxxxxxxxxxxxx...')

[At_Libitum]

Thanks, so this one is intended for when the books are already in the Calibre library, in the reader this just points to the file that is used in the epub to show the actual cover.