Replacing Kobo Bookstore

[kido.resuri]

Nice! Anyway, I think you are violating rules with this large image, could you make it smaller and make a link for the large version?

Is it the same location? Guess not. Or at least the code to change is the same?

The Gutenberg Project "bookstore" is a very nice idea, I support it. It could be done i think, needs a bit of trans-coding the contents.
I'll look into this later if I have some spare time for it.

[miyoyo]

Unfortunately, the libnickel binary is probably the n°1 file subject to change over time, so
the entire structure moves at every update.

Still, I can't seem to find the authorization token in the storeapi requests anymore, do you still have them or am I just missing something?

[miyoyo]

Just an additional detail, but it appears Kobo downloads the four first related reads to the e-reader silently in the background, and not the previews, but the complete, 1 MB+ Adobe DRM versions.

They are stored on Akamai servers.

Just a heads up if you use mobile data.

EDIT: Nope, these are previews

[davidfor]

Originally Posted by miyoyo

Just an additional detail, but it appears Kobo downloads the four first related reads to the e-reader silently in the background, and not the previews, but the complete, 1 MB+ Adobe DRM versions.

They are stored on Akamai servers.

Just a heads up if you use mobile data.

What makes you think that? I've looked at this before, and have checked now. It is definitely not happening here. The only books downloaded from the Kobo servers to my devices are those I have purchased, or those I have a preview of. They do download two versions of the cover, but nothing else.

And if they did download a book to your device, it wouldn't be an Adobe DRM protected version. It would be a kepub.

[miyoyo]

Originally Posted by davidfor

What makes you think that? I've looked at this before, and have checked now. It is definitely not happening here. The only books downloaded from the Kobo servers to my devices are those I have purchased, or those I have a preview of. They do download two versions of the cover, but nothing else.

And if they did download a book to your device, it wouldn't be an Adobe DRM protected version. It would be a kepub.

I captured traffic from a freshly reset kobo, and it had a request to the kobo download servers (non https obviously) redirecting to an akamai server with epubs with text that can't be read and a whole lot of chapters.

The books can also be previewed offline (and I doubt a preview will weigh over 1.5 Megabytes)

As soon as I can,i'll send you links/epubs over PM

EDIT:
I stand corrected, the epubs aren't protected, it was just a 7zip fail

But still, why the hell do epubs have such heavy fonts?!

(Doesn't change the fact download tokens are public)

[davidfor]

Sorry, I wasn't questioning where the files are on the net. I've never bothered looking before, but, I'm not surprised the books come from an Akamai server. That is the service they provide.

You seemed to be saying that Kobo was putting books on my device that I hadn't asked for. From experience, I knew this wasn't the case. But, I see you were really questioning the content of the previews.

For the font file size, it is whatever is need. They can be stripped to just the needed elements, but I most book designers don't seem to do that. So, they end up being the biggest part of an ebook.

The links used are protected on Kobo's side. And you need a key to get them. Plus they check you have permission before sending the actual download link. If someone was sniffing the traffic, they could work out the URL for the books. But, based on the URL used, trying to find other books will be long process. It could be more secure, but, whether it is needed, I don't know.

[miyoyo]

I've sent you a PM with a de-gzipped traffic analysis of the android app, every entry contains a token, so I don't think they're user-unique, and they can be downloaded from any browser that supports 302 redirects

To avoid double-posting i'll just update this post.

I've managed to setup a apache2 reverse proxy to transform HTTP requests into HTTPS
Then I've entered kobo's FQDNs into my own DNS server to point them to my web server
And I've captured everything I could using wireshark

I've realized this badly-made graph to symbolize how the sync process works in general.

The process goes from top to bottom, and the requests should be in that order.

The automatic sync, that happens when the device connects to wifi, only happens if the sync queue isn't empty, and consists of downloading the sync queue, downloading covers&metadata and finally uploading events.

I'm slightly concerned about the events, even if it's just "First time/Last time/Amount of times you've done it", sometimes I just don't like people peeking into what I've done, but whatever.

The only URL that is critical for automatic sync is the storeapi one, as it passes the sync queue, the download targets (could logically be alterred) and the metadata.

So, if a sync patch needs to be made, only this URL needs modifications, so that's one patch in libnickel, one modification in Kobo eReader.conf and the server itself.

I'd probably also go for a services patch to be able to switch between home screens and the odd other internal setting, an api patch for manual update distribution and anything else involves emulating the store api, so that's probably one for later.

This is becoming REAL guise.

EDIT2: Sync queue data here, stripped as well as I could
https://pastebin.com/Dh1p5Cit

[kido.resuri]

Nice. Could you capture sync while downloading a "purchased" book from Kobo store? A free book would do the job aswell.

[miyoyo]

I'm having a bit of a hard time unanonymizing the sync trafic, as it seems to be either linked to an IP or using some kind of black magic, but I'm setting up a reverse proxy to try to filter all of the traffic.

[kido.resuri]

Just look for the user id in the packet headers

[miyoyo]

Sorry for the time it took to fix, apparently old versions of wireshark don't show failed dns requests... or I broke the settings.

Not to worry, it was just me not correctly resolving the domain name, I've fixed it and here is what I can tell you about the book download process.


So, it firsts downloads the usual sync data (including the book's reading status), then it downloads the book's metadata (using https://storeapi.kobo.com/v1/library/{BookUUID}/metadata), approx. 9kb, then it uses the unique ANDROID download token over HTTP (not HTTPS) to storedownloads.kobo.com, that redirects to the actual book's download link, that is a static url.

So technically, to siphon books, you just have to listen to any storedownloads.kobo.com requests that passes around.

And to tell it to download books from anywhere, you just have to change the storeapi redirect URL.

This could logically allow for a modular sync system, with plugins for ex. Google drive or dropbox.

There are also different book variations, such as versions for iPhone, android, iPad, "generic" and a jpub HTML5 version

EDIT:
I'm wondering what language to write the sync server in
I've thought about GOlang, but the fact that it's compiled and that it has terrible regex knocks it out of the competition
Then, there's JavaScript, it's got average regex performance, it's memory heavy but with node.js it's probably the fastest to develop for, and one of the few good async solutions.

Then again, it depends if it's made for single-user operation or we actually give a shit about the user id, because if we don't then there are a whole lot of languages we could use that are bad at async.

[miyoyo]

New reply, to separate answers

I'll post random info I find here

Book Metadata regex:

({"(?:.{8}-.{4}-.{4}-.{4}-.{12}|Book)":)?\[?{"(?:Recommendation":{"Reasons":\[\]},")?Contributors":"([^"]*)?","WorkId":"(.{8}-.{4}-.{4}-.{4}-.{12})","(?:Subtitle":"([^"]*)?",")?(?:SeriesNumber":"(\d*)","SeriesName":"([^"]*)?","SeriesId":"(.{8}-.{4}-.{4}-.{4}-.{12})","SeriesNumberFloat":(\d*),")?(?:Subtitle":"([^"]*)?",")?IsFree":((?:fals|tru)e),"ISBN":"(\d*?)","PublicationDate":"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{7}Z?)","ContributorRoles":\[.*?\],"(?:IsInternetArchive":((?:fals|tru)e),")?CrossRevisionId":"(.{8}-.{4}-.{4}-.{4}-.{12})","Title":"([^"]*)?","Description":".*?","Language":"(..)","ImageId":"(.{8}-.{4}-.{4}-.{4}-.{12})","PublisherName":"([^"]*)?","Rating":([0-9]*\.[0-9]+|[0-9]+),"TotalRating":(\d*?),"(?:RatingHistogram":({"1":\d*?,"2":\d*?,"3":\d*?,"4":\d*?,"5":\d*?}),")?Slug":"([^"]*)?","IsContentSharingEnabled":((?:fals|tru)e),"RedirectPreviewUrls":\[(?:{"DrmType":"([^"]*)?","Format":"([^"]*)?","Url":"([^"]*)?","Platform":"([^"]*)?","Size":(\d*?)},?)*\],"HasPreview":((?:fals|tru)e),"Price":{"Currency":"(...)","Price":(\d*?(?:\.\d*?)?)},"(?:PromoCodeAllowed":((?:fals|tru)e),")?EligibleForKoboLoveDiscount":((?:fals|tru)e),"IsPreOrder":((?:fals|tru)e),"(?:RelatedGroupId":"(.{8}-.{4}-.{4}-.{4}-.{12})",")?Id":"(.{8}-.{4}-.{4}-.{4}-.{12})"}

Kobo image distribution:

https://kbimages1-a.akamaihd.net/{IMAGE-UUID}/167/256/100/false/image.jpg
                                             /     |   \      \
                                     Horizontal    |    \     Monochrome
                                                   |  JPEG quality
                                                Vertical

As far as I can tell there are no restrictions to the image resizing, except that it has to keep the aspect ratio.

[miyoyo]

Might be a bit of an unearthing, but since this is the project that quite literally got me into advanced programming, and reverse engineering, I ain't about to stop (yet )

I have found a solution that should
- Persist across reboots
- Presist across updates
- Not require any modification to libnickel.so.1.0.0
- Be installable with a single KoboRoot.tgz
- Be uninstallable with a single KoboRoot.tgz

I'll keep you posted, probably next week.

PS: Fun fact, tracking has gone off the charts, basically everytime you touch something they know it.

[sherman]

Originally Posted by miyoyo

Might be a bit of an unearthing, but since this is the project that quite literally got me into advanced programming, and reverse engineering, I ain't about to stop (yet )

I have found a solution that should
- Persist across reboots
- Presist across updates
- Not require any modification to libnickel.so.1.0.0
- Be installable with a single KoboRoot.tgz
- Be uninstallable with a single KoboRoot.tgz

I'll keep you posted, probably next week.

PS: Fun fact, tracking has gone off the charts, basically everytime you touch something they know it.

Oooh...

Looking forward to this

Any hints?

[miyoyo]

This is the result of a good while of reverse engineering the binary, so for now I'd like to keep my edge till I release stuff, but it's simpler than you might think.