Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 05-14-2017, 06:28 AM   #1
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 12,227
Karma: 18249247
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo: Touch, Glo, Aura H2O, Glo HD
Problem with Metadata smart_update

@Kovid:

I have had a problem with the collections displayed in the device list for some time. Occasionally, they wouldn't show. Unplugging the device and plugging it back in would fix it. Because of that, I haven't been bothered about it. But, I've had a couple of reports recently about problems with collections, including a report that when Metadata management was set to automatic, the collections weren't showing. So, I've had a better look at it.

I found a problem in the KoboTouch driver that caused of the problems when the collection management was turned off or there were no collection columns. After fixing that, there were still issues.

The problem is most obvious when there is no metadata.calibre file on the device. So, the second time the device is connected, everything is OK.

The problem is that something is clearing the "device_collections" attribute between the "Get list of books on device" and "Send metadata to device" jobs. I tracked it down to where the metadata from the device is merged with that from the library. This is in gui2/device.py in method "update_book". That calls ebooks/metadata/books/base.py, Metadata:smartupdate. This is called with "replace_metadata=True".

The problem is that smartupdate with that options sets some attributes to None if they are not in the source object. The list of attributes to be treated this way includes "device_collections". As the source object is coming from the database, "device_collections" won't be in the source object and hence will be set to None.

The fix I have is to add "device_collections" to the list SC_FIELDS_COPY_NOT_NULL in ebooks/metadata/book/__init__.py. With that, all the cases I have tested, the collections are displayed correctly.

Searching through the code, I don't think that this will cause a problem. I do not have any non-Kobo devices to test with, so, I can't prove it that way. And, of course, I might have missed something in the code.
davidfor is offline   Reply With Quote
Advert
Old 05-14-2017, 07:39 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,862
Karma: 7728435
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I'll take a look.
kovidgoyal is offline   Reply With Quote
Old 05-14-2017, 08:13 AM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,862
Karma: 7728435
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Doesn't putting it in there mean that it will no longer be copied at all by smart_update(). That is likely to have rather far reaching consequences. A more localized fix would probably be to change the update() book method to preserve the contents of that field instead. Something like
Code:
dc = getattr(book, 'device_collections', None)
book.smart_update()
if not getattr(book, 'device_collections', None):
    book.device_collections = dc
kovidgoyal is offline   Reply With Quote
Old 05-16-2017, 08:47 AM   #4
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 12,227
Karma: 18249247
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo: Touch, Glo, Aura H2O, Glo HD
That should work, but it feels clumsy.

The code in smart_update that does the actual copy is:

Code:
        if replace_metadata:
            # SPECIAL_FIELDS = frozenset(['lpath', 'size', 'comments', 'thumbnail'])
            for attr in SC_COPYABLE_FIELDS:
                setattr(self, attr, getattr(other, attr, 1.0 if
                        attr == 'series_index' else None))
            self.tags = other.tags
            self.cover_data = getattr(other, 'cover_data',
                                      NULL_VALUES['cover_data'])
            self.set_all_user_metadata(other.get_all_user_metadata(make_copy=True))
            for x in SC_FIELDS_COPY_NOT_NULL:
                copy_not_none(self, other, x)
            if callable(getattr(other, 'get_identifiers', None)):
                self.set_identifiers(other.get_identifiers())
            # language is handled below
At the moment, "device_collections" ends up being part of SC_COPYABLE_FIELDS. By putting it in "SC_FIELDS_COPY_NOT_NULL", it gets removed from SC_COPYABLE_FIELDS, bu will still be copied if it has a value.

The current value of SC_FIELDS_COPY_NOT_NULL is:

Code:
SC_FIELDS_COPY_NOT_NULL =  frozenset(['lpath', 'size', 'comments', 'thumbnail'])
I think that "device_collections" should be treated the same as "lpath". I can't find anywhere that device_collections is not set by a driver. And the current behaviour means that the device_collections is wiped after matching the books on the device to that in the library.
davidfor is offline   Reply With Quote
Old 05-16-2017, 11:06 AM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,862
Karma: 7728435
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Ah ok, makes sense. I was mis-remembering what smart_update did.

Go ahead and send a PR, I'll merge it.
kovidgoyal is offline   Reply With Quote
Advert
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Metadata problem pookiebr Library Management 13 03-19-2013 10:14 AM
Metadata problem tm3 Library Management 2 11-14-2012 09:23 AM
problem with metadata FranSmith Calibre 17 01-23-2011 11:54 AM
edit metadata problem donschjr Calibre 3 09-30-2010 11:33 AM
Problem with metadata in Connect tiohn Sony Reader 3 11-13-2007 02:03 PM


All times are GMT -4. The time now is 11:41 PM.


MobileRead.com is a privately owned, operated and funded community.