Beta test of major new features

[Scott Nielsen]

Originally Posted by chaley

A 'tweak' has been added to control the format of the original Publication Date column in the GUI, so you can change it to show the year only. See this post for details.

That's just brilliant, Chaley! I'm on the road this week but as soon as I'm back I'll be sure to d/l a beta and try it out.

[chaley]

Using code originally written by user_none (thank you!), we have added columns to both the device and library displays that indicate whether a book is on the device or is in the library. When you plug in a device, the column appears and then is filled in appropriately. Unplug the device and the column goes away.

This feature is available in releases 0.6.92 and higher. Download links are in post #1.

Some caveats:
1) the matching algorithm is approximate. Calibre makes its best guess about whether a given book is the same on the device and library. There are two matching schemes, one using title and author, and one using title and a short ID.

The scheme using the short ID is much more accurate, but it requires that the file on the device end in _NNN where NNN is the library key. In the past, only Sony's included this information but now all USB-based devices do it. To get more accurate matching, you will probably need to copy the books to the device again. What worked best for me was to empty my device (I formatted it), then fill it again from Calibre.

Title + author matching checks all authors, which means it won't match if the book has multiple authors but the device saves only one of them (which my Sony Pocket seems to do). It also won't match if the metadata for the format does not match the metadata in the library database.

Neither will match if the title in the book metadata does not match the title in the library metadata.

Bottom line: there is no guarantee that the OnDevice/InLibrary indications are 100% accurate. However, when using the short ID, testing has shown that they are very close.

2) The current beta has some problems with column geometry. When the ondevice column comes and goes, columns to the right change to the size of their neighbor. I am looking for ways to fix this. Work-around: move the column to the end.

3) Deleting books from either the library or the device does not (yet) update the other display. Adding books from the library to the device does the right thing.

4) We have tested it on devices we have: Sony PRS-300, Sony 505, and a Kindle. Other devices *should* work, but the binary goblins are still out there.

As always, have fun.

PS: There might be some comments in this post

[rollercoaster]

Interesting... this will be very handy

Calibre contributers keep breaking our expectations and then some

[Starson17]

Originally Posted by chaley

1) the matching algorithm is approximate. ...There are two matching schemes, one using title and author, and one using title and a short ID.

FYI, the importing code that operates when the Add/Save option "If books with similar authors and titles found, merge the new files automatically" is selected, makes a similar attempt to match existing records in the db with something else. In that case, the "something else" is books being added, instead of books already on a device.

IIRC, that code requires an exact match of {authors}, but uses a fuzzy match of title (convert to lower case, remove punctuation, eliminate excess whitespace, remove leading indefinite articles "a", "an", "the", etc.).

I don't know if you see any reason the matching algorithms should be coordinated, but in case you aren't aware of the other algorithm's existence, I thought I'd comment. I don't have any ebook devices (or none that Calibre considers to be a "Device"), so I'm not really up to speed on the benefits or disadvantages of one matching algorithm over another.

[chaley]

Originally Posted by Starson17

IIRC, that code requires an exact match of {authors}, but uses a fuzzy match of title (convert to lower case, remove punctuation, eliminate excess whitespace, remove leading indefinite articles "a", "an", "the", etc.).

I haven't looked at the other match algorithms. I should.

I am using user_none's code, which first lower_cases the title and then uses a regexp to prepare the title for matching. The regexp,

re.sub('(?u)\W|[_]', '' ...)

removes all non-alphanumeric and '_' characters.

I don't have any ebook devices (or none that Calibre considers to be a "Device")

I also have a device that isn't a calibre device -- an HTC phone. I have been wondering whether it makes sense to build a general driver that matches any USB storage that the other drivers do not, and simply uses a folder on the device to store books. One way to do it is to have a USB plugin that accepts lists of device id:folder pairs. I don't think it would be outrageously hard, but I could be surprised, especially when cross-platform issues are factored in.

What I don't know is if it would be useful. Sometimes I want it, and sometimes I think that saving to disk is enough. However, with the new OnDevice feature, I think that I will want my phone to be considered a valid device. Reflection continues.

[kovidgoyal]

Originally Posted by chaley

I also have a device that isn't a calibre device -- an HTC phone. I have been wondering whether it makes sense to build a general driver that matches any USB storage that the other drivers do not, and simply uses a folder on the device to store books. One way to do it is to have a USB plugin that accepts lists of device id:folder pairs. I don't think it would be outrageously hard, but I could be surprised, especially when cross-platform issues are factored in.

Why don't you just add a driver for the phone? I think Starson is referring to devices that aren't mountable as disks.

[chaley]

Originally Posted by kovidgoyal

Why don't you just add a driver for the phone? I think Starson is referring to devices that aren't mountable as disks.

Ahhh, that is the easy way. I was thinking of something more general, but perhaps I am being silly.

[kovidgoyal]

Originally Posted by chaley

Ahhh, that is the easy way. I was thinking of something more general, but perhaps I am being silly.

A general framework would be nice to have, but to make it worthwhile you have to implement the following (all cross platform)

1) Allow the user to pick a removable disk
2) Find the usb device it belongs to
3) Find any other disks that belong to that device
4) Allow the user to specify what formats and base dorectory to use when sending to the device

I agree that it would be great to have this, I've just never been motivated enough to climb this hill.

[Starson17]

Originally Posted by kovidgoyal

Why don't you just add a driver for the phone? I think Starson is referring to devices that aren't mountable as disks.

I've got some that aren't mountable (ActiveSync) and some that are USB disk mountable, but don't properly report all data that Calibre wants reported (the HTC HD2) and some that I haven't looked at closely, but might have sufficient info (ordinary USB card reader). I completely agree with Charles, which is that "Sometimes I want it, and sometimes I think that saving to disk is enough. "

[chaley]

I could do all that for windows and perhaps for linux (would need to set up a VM), but not for Mac. So I will ignore it.

I have a driver for my phone, an HTC Touch Diamond2 running WM 6.5. It took me a little while to understand what to do because I didn't fully understand card matching until I traced it through the code. My phone won't show its main mem as a drive and it presents its card without any significant ID details.

Although the driver works, the lack of extra ID details makes me nervous.

The USB ID info is u'usb\\vid_0bb4&pid_0c30&rev_0000']
The storage ID is

\\?\STORAGE#VOLUME#_??_USBSTOR#DISK&VEN_&PROD_&REV_#3FBF5000-7351-0801-3561-580209983530&0#{53F56307-B6BF-11D0-94F2-0A0C91EFB8B}#{53F5630D-B6BF-11D0-94F2-00A0C91EFB8B}

All of VEN, PROD, and REV are empty.

I have no idea how this will present on a Mac or Linux, so my driver doesn't permit them.

FYI - the driver code is:

class HTC_TD2(USBMS):

    name           = 'HTC TD2 Phone driver'
    gui_name       = 'HTC TD2'
    description    = _('Communicate with HTC TD2 phones.')
    author         = 'Charles Haley'
    supported_platforms = ['windows']

    # Ordered list of supported formats
    FORMATS     = ['epub', 'pdf']

    VENDOR_ID   = {
            # HTC
            0x0bb4 : { 0x0c30 : [0x000]},
            }
    EBOOK_DIR_MAIN = ['EBooks']
    EXTRA_CUSTOMIZATION_MESSAGE = _('Comma separated list of directories to '
            'send e-books to on the device. The first one that exists will '
            'be used')
    EXTRA_CUSTOMIZATION_DEFAULT = ', '.join(EBOOK_DIR_MAIN)

    VENDOR_NAME      = ['']
    WINDOWS_MAIN_MEM = ['']

#    OSX_MAIN_MEM = 'HTC TD2 Phone Media'
#    MAIN_MEMORY_VOLUME_LABEL  = 'HTC Phone Internal Memory'

    SUPPORTS_SUB_DIRS = True

    def post_open_callback(self):
        opts = self.settings()
        dirs = opts.extra_customization
        if not dirs:
            dirs = self.EBOOK_DIR_MAIN
        else:
            dirs = [x.strip() for x in dirs.split(',')]
        self.EBOOK_DIR_MAIN = dirs

[chaley]

Having my phone's storage card treated as a calibre device is very nice. I get easy copying and management of books and the OnDevice stuff works a treat. The only downside found so far is the time to 'mount' the device, while calibre scans metadata.

[kovidgoyal]

Unfortunately, you need the storage id to have non null values for VEN and PROD, otherwise you cannot have more than one such driver in the calibre code base.

Unless you can figure out how to get the UUIds associated with the usb vendor and product ids.

On linux and osx that driver will work automatically (they have much saner USB subsystems than windows).

[Starson17]

Originally Posted by chaley

I have a driver for my phone, an HTC Touch Diamond2 running WM 6.5. All of VEN, PROD, and REV are empty.

Same issue with the HTC HD2.

[user_none]

Originally Posted by chaley

The scheme using the short ID is much more accurate, but it requires that the file on the device end in _NNN where NNN is the library key.

Just curious, are you not using the metadata caching code that I was using with the on device column? It included id's associated with the library entry so the file names didn't have to be modified?

[chaley]

Originally Posted by user_none

Just curious, are you not using the metadata caching code that I was using with the on device column? It included id's associated with the library entry so the file names didn't have to be modified?

No, not yet. Two reasons: 1) I am still looking at it to understand how it would work across all devices that support their own metadata, such as Sonys (as you know, non-trivial, but as I have one, important to me), and 2) am looking at a more general solution where the device isn't a 'device' but is instead an instance of a library. Option 2 is my preference, but Kovid says that if we go that way, he wants to redesign libraries from the ground up to have a plug-in architecture and support more-or-less arbitrary data sources - aka do a *lot* of work. He is willing to do the architecture work, and I am willing to try to implement some instances of it, so off we go. We will see...

Given the above, and because ondevice is such a useful thing to have, it seemed to make sense to slightly extend work you had already done to provide a more reliable indication. Adding the id to the name, which is what the Sonys already do, was the most straightforward and carried a minimal cost. Of course, it does not provide the performance enhancement that metadata caching gives, and it junks up the names a bit.