Locate newer books in library than on a device

[snarkophilus]

Is there a way of determining what books are newer in the calibre library than what is on a device, either directly within Calibre or via a plugin? For example, I'd like to know if I've editted an epub in Calibre more recently than when that epub was sent to the device. Even something as basic as "last mod time of library format is newer than last mod time of device format".

I tried searching for a similar question but couldn't find it asked. Also I hope Calibre->Devices is the right (sub)forum.

[kovidgoyal]

Not automatically no. You would need to compare the last modified dates of the files in the calibre library to those on the device. calibre itself has no functionality to do that.

[snarkophilus]

Originally Posted by kovidgoyal

Not automatically no. You would need to compare the last modified dates of the files in the calibre library to those on the device. calibre itself has no functionality to do that.

Thanks for your answer. Is this something that could possibly be done via a plugin? If so, might be time for this old C coder to learn some python!

[kovidgoyal]

Sure, plugins can do pretty much anything. although I dont recall a lot of example plugins that talk to devices, maybe the reading list plugin is your best bet to start from.

[jackie_w]

Originally Posted by snarkophilus

Thanks for your answer. Is this something that could possibly be done via a plugin? If so, might be time for this old C coder to learn some python!

I have a quick and (very) dirty calibre plugin which can do what you want. It is nowhere near polished enough to be an official community share but it may be of some help as a quick-start project if you are planning to dip your toes in calibre plugin/python waters. PM me if you decide to go ahead.

Just a caveat, though, in case you aren't already aware ... Kobo make heavy use of the TOC when importing new books. If your epub edits have included changes to the TOC then simply overwriting the device's old epub/kepub with the calibre library's newly edited version will probably result in an unreadable book on the Kobo and you'll need to delete the old before re-sending the new. If you've only corrected things like typos, broken paragraphs, CSS, ... then a simple overwrite should be fine. I'm only guessing, but I suspect that if this inconvenient TOC business didn't exist then davidfor would have already included the feature you asked for in the calibre KoboTouch driver and/or the Kobo Utilities plugin.

[snarkophilus]

Thanks Kovid, Jackie. I've sent Jackie a PM, lets see where that gets me to...

[theducks]

Snark
Why not just send the suspect books again? I have my Kobo set to Not treat the same title as New, therefore, it replaces what is there. Only if there are major (cleanup) edits, does the current reading position vary by much.
FWIW I frequently despan those stupid- unnecessary spans that set "font-size to 1em and line-height: 1.2" That is the default for the body), while in fixing typos or other weirdness

[davidfor]

Originally Posted by jackie_w

I have a quick and (very) dirty calibre plugin which can do what you want. It is nowhere near polished enough to be an official community share but it may be of some help as a quick-start project if you are planning to dip your toes in calibre plugin/python waters. PM me if you decide to go ahead.

You have me intrigued. Can I see it as well?

Just a caveat, though, in case you aren't already aware ... Kobo make heavy use of the TOC when importing new books. If your epub edits have included changes to the TOC then simply overwriting the device's old epub/kepub with the calibre library's newly edited version will probably result in an unreadable book on the Kobo and you'll need to delete the old before re-sending the new.

I have to get back to finishing my ToC updater. What I have works fine with ePubs, but, I have to get to kepubs. And integrate it with the Kobo Utilities plugin. And remove some horrible code.

If you've only corrected things like typos, broken paragraphs, CSS, ... then a simple overwrite should be fine. I'm only guessing, but I suspect that if this inconvenient TOC business didn't exist then davidfor would have already included the feature you asked for in the calibre KoboTouch driver and/or the Kobo Utilities plugin.

It isn't one of the things I've really thought about. I tend to use the Reading List plugin for this. A list for each device and when I add or change a book, I put it on the lists for the devices I want it on. Later I connect the device and it goes there automatically. That plus sorting by either the date or modified date and I will usually find anything I need to send to the device.

What I would see as useful is to be able to find all books added or modified since the previous time the device was connected. This is stored in the "driveinfo.calibre" file on the device. It gets read when the device is connected, but gets updated to the current time before the driver really gets to see it. It's buried just deep enough to make it hard to change, and there isn't really anything that currently uses it.

[jackie_w]

Originally Posted by davidfor

You have me intrigued. Can I see it as well?

Prepare to be underwhelmed. Intrigue-worthy it definitely isn't I'll PM you a link.

Originally Posted by davidfor

It isn't one of the things I've really thought about.

We had a very brief conversation about it a couple of years ago and you reminded me about the updated TOC issue. I realised at that point that the support issues would be endless (and repetitive), no matter how good the help notes and warnings were. I can't say I blame you for wiping it from your memory banks.

Originally Posted by davidfor

What I would see as useful is to be able to find all books added or modified since the previous time the device was connected. This is stored in the "driveinfo.calibre" file on the device. It gets read when the device is connected, but gets updated to the current time before the driver really gets to see it. It's buried just deep enough to make it hard to change, and there isn't really anything that currently uses it.

Just so you know, this plugin only runs on demand rather than automatically at connection time. It's also only currently aimed at identifying books already on the device (using metadata.calibre) which have changed in the calibre library (by comparing file system timestamps) rather than also looking for brand new books. Still, I'm sure you'll be able to add all sorts of bells & whistles.

[DNSB]

Originally Posted by jackie_w

Just a caveat, though, in case you aren't already aware ... Kobo make heavy use of the TOC when importing new books. If your epub edits have included changes to the TOC then simply overwriting the device's old epub/kepub with the calibre library's newly edited version will probably result in an unreadable book on the Kobo and you'll need to delete the old before re-sending the new. If you've only corrected things like typos, broken paragraphs, CSS, ... then a simple overwrite should be fine. I'm only guessing, but I suspect that if this inconvenient TOC business didn't exist then davidfor would have already included the feature you asked for in the calibre KoboTouch driver and/or the Kobo Utilities plugin.

Ummm... that's why davidfor has the "Do not treat replacements as new books" checkbox on the Collections, covers and uploads" tab of the KoboTouch driver. I've only used that option once when I updated the covers on a mass of epubs but made no changes to the text and ToC. With a few hundred books, the time saved by not losing my annotations of errors on a dozen or so was worth the risk.

[chaley]

For the adventurous: the base calibre device interface provides a method, synchronize_with_db(), that is called during device connection. It was added to support syncing by the wireless device driver (WD). This method supports a device:

• telling calibre to send a format to the device. The WD sets this if the format on the device is older than the format in calibre.
• updating the calibre database with information coming back from the device. The WD uses this if the device hands back a new is_read boolean or date.
• telling calibre to update the GUI for a book. In the WD this happens if the device driver updates the calibre database.

The method in the device driver (synchronize_with_db) is called on the GUI thread and is given a handle to the calibre database. As mentioned above, it is called while calibre is generating the list of books on the device.

For an example, see devices/interface.py, gui/devices.py, and devices/smart_device_app/driver.py. I admit that the code is complicated (sigh), and I am very willing to help if someone is interested.

[snarkophilus]

Originally Posted by theducks

Snark

Thanks for the informal address, Ducks

Why not just send the suspect books again? I have my Kobo set to Not treat the same title as New, therefore, it replaces what is there. Only if there are major (cleanup) edits, does the current reading position vary by much.

The problem was working out which are the suspect books! I can see that I've got 65 books that are newer in the library than are on my Kobo. I've been on a bit of a "let's do some simple cleanups on stuff on my reading list" spree.

I could have just re-sent everything that was on the Kobo, but that just didn't seem like the right solution...

And let's not start on that my reading list is growing rather than shrinking

FWIW I frequently despan those stupid- unnecessary spans that set "font-size to 1em and line-height: 1.2" That is the default for the body), while in fixing typos or other weirdness

I hear ya

Originally Posted by davidfor

That plus sorting by either the date or modified date and I will usually find anything I need to send to the device.

Wow, didn't know that that was a displayable field. However it appears to be if anything including metadata has been modified (eg, changing reading list order). So maybe not quite as handy as it seems at first glance?

Originally Posted by chaley

For the adventurous: the base calibre device interface provides a method, synchronize_with_db(), that is called during device connection. It was added to support syncing by the wireless device driver (WD).

[ .... ]

I admit that the code is complicated (sigh), and I am very willing to help if someone is interested.

Sounds useful from what you've described. I wonder if I've jumped in at rather the deep end of the plugin world. I hope you don't live to regret that offer

[chaley]

Originally Posted by snarkophilus

Wow, didn't know that that was a displayable field. However it appears to be if anything including metadata has been modified (eg, changing reading list order). So maybe not quite as handy as it seems at first glance?

If you want the actual modified date of the epub in the calibre library, create a custom column "built from other columns" using the template

{:'select(formats_modtimes('yyyy/MM/dd hh:mm:ss'), 'EPUB')'}

Note that the formats_modtimes() function must actually read the disk to get the times, which can have an effect on calibre performance.

Sounds useful from what you've described. I wonder if I've jumped in at rather the deep end of the plugin world. I hope you don't live to regret that offer

NB:The method is only available in device drivers, which can be plugins.

[theducks]

Originally Posted by snarkophilus

I could have just re-sent everything that was on the Kobo, but that just didn't seem like the right solution...

I just sort my Library on {Last_Modified} and Filter by On_device:true.

I do wish Calibre had a way of displaying (when connected) WHAT THE the timestamp of the metadata.calibre was when first connected that session.
Device was previously updated: yyyy/mm/dd:HH:mm

Originally Posted by snarkophilus

And let's not start on that my reading list is growing rather than shrinking

If it wasn't, you would probably have to turn in your MR card