Searching and downloading ebooks directly from Calibre server over Wi-Fi

[TheWizard]

(totally manual kindle hacking here, will fill with formal .diff/.patch/.update files as I have time, just wanna get some of this out there for those who need it)

Hi MR Kindle devs! thanks a lot for all the various software and tweaks you have assembled over the years. My Kindles would be useless for my primary use case (reading big scanned technical-reference books and docs) without amazing work like koreader.

In this thread I'm gonna share some tweaks I've made to my K3 and KPW to work more efficiently with Calibre. I have hundreds of huge PDFs so sideloading them all in advance via USB is a big waste of time and not totally do-able in the 2GB of flash. Its much better for me to download books from the Kindle on-demand via Wi-Fi, but the OS needs a few tweaks.

I have Calibre set to run its "Content Server" on boot. This will provide an HTTP portal from which you browse your book catalog from any web browser, including the Kindle!

By tweaking the search bars on the kindle, you can pull up books in your Kindle browser nearly as fast as you could pull them up from the Calibre's GUI itself.

On the KPW, /usr/share/webkit-1.0/pillow/javascripts/search_bar.js implements the little search menu (magnifying glass) on the home screen.
Back this up and make a working copy on your PC. You'll need to run the file through a "prettifier" as it is machine-condensed javascript. If you search for "wikipedia", you'll find the function that builds a search URL for searching Wikipedia. You can copy the function, and edit it to run against your Calibre server.

    const i = {
        id: "wikipedia",
        launcher: function(aL) {
            return aa + "?view=" + encodeURIComponent(SearchBarStringTable.wikipedia + encodeURIComponent(aL));
        }
    };
    const cb = {
        id: "calibre",
        launcher: function(aL) {
            return aa + "?view=" + encodeURIComponent(SearchBarStringTable.calibre + encodeURIComponent(aL));
        }
    };

    this.updateDefaultSearchDomains = function(aN) {
        var aM = nativeBridge.getDynamicConfigValue(DYNCONFIG_OBFUSCATED_MARKETPLACE_KEY);
        var aO = nativeBridge.getDynamicConfigValue(DYNCONFIG_URL_BAIDU_BAIKE);
        S = aO ? aO : SearchBarStringTable.baiduBaike;
        var aL = nativeBridge.getDynamicConfigValue(DYNCONFIG_URL_BAIDU_SEARCH);
        f = aL ? aL : SearchBarStringTable.baiduSearch;
        if (!aM) {
            Pillow.logWarn("getMarketplaceFailed");
            return;
        }
        if (aM === L) {
            am = false;
            g = true;
        } else {
            am = true;
            g = SearchBarStringTable.addBaiduBaike;
        } if (am) {
            aA(i);
            aA(cb); //calibre
        } else {
            az(i);
            az(cb); //calibre
        } if (g) {
            aA(ax);
        } else {
            az(ax);
        } if (aN) {
            this.setCustomSearchDomains();
        }
    };

There are corresponding strings that you need to add well.

/usr/share/webkit-1.0/pillow/strings/search_bar_strings.js

var SearchBarStringTable = {
    myItems: "My Items",
    enterText: "Search",
    go: "Go",
    google: "http://www.google.com/search?q=",
    wikipedia: "http://en.wikipedia.org/w/index.php?title=Special%3ASearch&search=",
    baiduBaike: "http://baike.baidu.com/search/word?&pic=1&enc=utf8&word=",
    baiduSearch: "http://www.baidu.com/s?tn=baiduhome_pg&ie=utf-8&rn=4&wd=",
    addBaiduBaike: false,
    calibre: "http://mycalibreserver.example.com:8080/mobile?num=25&sort=date&order=descending&search=",
};

/usr/share/webkit-1.0/pillow/strings/search_domain_strings.js

var SearchDomainStringTable = {
    domainLabels: {
        catalog: 'My Items',
        store: 'Kindle Store',
        google: 'Google',
        wikipedia: 'Wikipedia',
        dictionary: 'Dictionary',
        website: 'Web Address',
        baiduBaike: 'Baidu Baike',
        baiduSearch: 'Baidu',
        calibre: 'Calibre',
    },
};

After installing these javascript mods, restart the kindle, and you will have a new Calibre option in your search menu.

I haven't yet implemented a Calibre search on the K3, but it should be possible using JBPatch. For now, I use a browser bookmark and search from the page itself, but a having home screen search shortcut would shave off a bunch of keystrokes.

The last tweak is enabling the kindle browser to download the ebooks you have. Amazon has restricted the browser to only download marked-up ebooks based on MobiPocket (.MOBI, .AZW). PDFs, DJVUs, and EPUBs are what I mostly have and they are rejected by the browser. however, the block is very trivial to circumvent with a hex editor.

on the K3, you want to grab /usr/bin/browserd. On the KPW/KT, it is /usr/bin/mesquite.

Make backups of those files (important!) and load up your favorite hex editor and run a string search for "mobipocket". You will find a cluster of strings detailing the MIME types and file extensions that the browsers will allow for download. You can change any of the EXISTING mimes and extensions IN-PLACE to better fit your needs. Just be sure to start the edit from same starting position as the old mime/filetype. You cannot alter the starting positions or append new entries as those array locations (address offsets) are hardcoded. I changed the .azw* entries to .pdf and .djvu as I'll never need to download an .azw file via the browser, the Kindle store takes care of that using the background downloader.

A skillful reverse engineer could probably neutralize the entire mime-checker function to allow any file to be dowloaded, but that is not me. Perhaps if I a little more, I'll be able to in the future. Another way I tried was creating a custom mesquite app based on WebLaunch that used DBUS/LIPC to call cURL after a download link was "rejected" by the WebKit engine, but that was more work and not as nice as just making the engine do its job .

Like I said, this could be automated to a large extent, depending on available time and interest. But I now have more books to read! thanks again kindle devs from 111F in Arizona