Replacing Kobo Bookstore

[kido.resuri]

The authorization header is x-kobo-userkey and the header value is your ClientId, which can be found in .kobo/Kobo/Analytics.cong
If you add these custom headers, the kobo servers will start responding. The Kobo eReader requests all data in json format, by using the headers "Accept" - "application/json" and "Accept-Encoding" - "gzip" and "Content-Type" - "application/json". Some data uses "text/json", so every packet must be analyzed one by one.
I also changed the "User-Agent" header to

"Mozilla/5.0 (Linux; U; Android 2.0; en-us;) AppleWebKit/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1 (Kobo Touch)"

as this is what Kobo eReader use.

[davidfor]

Originally Posted by kido.resuri

The authorization header is x-kobo-userkey and the header value is your ClientId, which can be found in .kobo/Kobo/Analytics.cong

Are you sure of that? It doesn't match the value used here. And the database would be a better source.

If you add these custom headers, the kobo servers will start responding. The Kobo eReader requests all data in json format, by using the headers "Accept" - "application/json" and "Accept-Encoding" - "gzip" and "Content-Type" - "application/json". Some data uses "text/json", so every packet must be analyzed one by one.
I also changed the "User-Agent" header to

"Mozilla/5.0 (Linux; U; Android 2.0; en-us;) AppleWebKit/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1 (Kobo Touch)"

as this is what Kobo eReader use.

From an experiment I did ages ago, the server doesn't check the user agent.

[kido.resuri]

Well, I'm not really sure as you ask now. To be honest, I used the value from the captured packet, not from any config file or database, but the format of the ID is the same.

[davidfor]

Interesting. We obviously think differently. My first thought would be to find out where it came from.

The two ids look the same because the are both UUIDs. But, you said the ID you used was the "ClientId" in the ".kobo/Kobo/Analytics.conf". And based on what I know about this, and looking at my "Analytics.conf", that is not the case. The id I see there is not used anywhere else. The id needed for the requests to the store comes from the database.

[miyoyo]

Just came in: Kobo's Authorization tokens last about 48 hours, then, the page returns a 401 saying that the token is expired.

[kido.resuri]

@davidfor ok, sorry, I was wrong. The x-kobo-userkey header is the exact same as in the KoboReader.sqlite database, user table, UserKey.

@miyoyo, I don't know where are you trying these, but I just fired up the above mentioned extension in firefox, and loaded this url:
https://storeapi.kobo.com/v1/user/re...=0&page_size=1
(captured from the sync process of Kobo device, slightly modified, as it uses page_size=50), and got this response:

Spoiler:

{"Items":[{"Book":{"Recommendation":{"Reasons":[]},"Contributors":"Erin Kellison","WorkId":"<some workid was here, removed>","Subtitle":"Shadow Series 1","IsFree":false,"ISBN":"9780996398060","PublicationDate":"2015-10-22T04:00:00.0000000Z","CrossRevisionId":"<some revision id was here, removed>","Title":"Shadow Bound","Description":"<p><span>DEATH</span></p> <p>Some people will do anything to avoid it. Even trade their immortal souls for endless existence.</p> <p><span>WRAITHS</span></p> <p>Secretly, inexorably, they are infiltrating our world, sucking the essence out of unsuspecting victims with their hideous parody of a kiss.</p> <p><span>SEGUE</span></p> <p>Adam Thorne founded the Institute to study and destroy his monster of a brother, but the key to its success is held in the pale, slender hand of a woman on th...","Language":"en","ImageId":"<some image id was here, removed>","PublisherName":"Fire Flower Publishing, LLC","Rating":3.8,"TotalRating":5,"RatingHistogram":{"1":1,"2":0,"3":1,"4":0,"5":3},"Slug":"shadow-bound-5","IsContentSharingEnabled":true,"RedirectPreviewUrls":[{"DrmType":"None","Format":"EPUB3_SAMPLE","Url":"http://storedownloads.kobo.com/download?downloadToken=<download token was here, removed>","Platform":"Generic","Size":798142}],"HasPreview":true,"Price":{"Currency":"USD","Price":3.99},"EligibleForKoboLoveDiscount":false,"IsPreOrder":false,"Id":"<some id removed>"}}],"ItemCount":1,"TotalPageCount":43,"TotalItemCount":43,"CurrentPageIndex":0,"ItemsPerPage":1,"Filters":{}}

Note, I removed the IDs and download tokens from the response, as I don't want to violate any law, perhaps I wouldn't.

I didn't change any auth or userkey headers since weeks ago, so i don't think any of these auth tokens ever expire. At least not 48 hours. May I know how and where did you try?

[kido.resuri]

On the other hand, weeks ago I was playing with the idea of changing the certificates in nickel to be able to set up a https server locally that kobo accepts, but never made any success, with creating the required certificates. Surely i should create some root certification, then sign some other certifications and then sign the one to be used with the webserver, and then hack this certificate into nickel. Seemd to be a lot of work and mess that i had no time to do or dig myself into it's depths.

So if you maybe have the time and talent to do such, it would be great. But to do this for every user, who would use this to sync their own book collection... it would be a real pain in the ass to do the certifications for everyone i think.

[kido.resuri]

To leak some other info, here is an example data of the response from https://services.kobobooks.com/Appli...es.svc/shelves

Spoiler:

[{"Id":"<id removed>","Type":"Custom","Name":"Rejtþ Jenþ","InternalName":"Rejtþ Jenþ","LastModified":"\/Date(1487534079000-0500)\/","CreationDate":"\/Date(1487534079000-0500)\/","ItemCount":0},{"Id":"<id removed>","Type":"Custom","Name":"Rejtő kÜnyvek","InternalName":"Rejtő kÜnyvek","LastModified":"\/Date(1485096701000-0500)\/","CreationDate":"\/Date(1484965247000-0500)\/","ItemCount":0},{"Id":"<id removed>","Type":"Custom","Name":"Harry Potter","InternalName":"Harry Potter","LastModified":"\/Date(1487534079000-0500)\/","CreationDate":"\/Date(1483326374000-0500)\/","ItemCount":0},{"Id":"<id removed>","Type":"Custom","Name":"Kßlső programok","InternalName":"Kßlső programok","LastModified":"\/Date(1483326374000-0500)\/","CreationDate":"\/Date(1483326374000-0500)\/","ItemCount":0},{"Id":"<id removed>","Type":"Custom","Name":"Receptek","InternalName":"Receptek","LastModified":"\/Date(1487534080000-0500)\/","CreationDate":"\/Date(1482899651000-0500)\/","ItemCount":0}]

This is also part of the sync process, and it is about the collections ("shelves") on the eReader.
There's also a nice "initialization" and "sync" request-respond pair on storeapi, I may post it if anyone is interested.

[miyoyo]

After battling with the automatic wipe of the services in Kobo eReader.conf, I just modified the rcS to mount /dev/mmcblk0p3 as read only, and removed HTTPS from the file.

Unfortunately nickel doesn't care at all, and just contacts mobile.kobobooks.com with HTTPS.
Also, interestingly, the kobo setup process resets if it doesn't have read/write access to the files on the sdcard.

EDIT:
Kobo devs have no Idea what they're doing either.


Has anybody got any idea where the mobile.kobobooks.com string is stored?

[kido.resuri]

As far as i remember, it's in libnickel.

[kido.resuri]

Anyone else has a /usr/openssl.cnf on Kobo? I can't remember if I put it there or not, but according to the file date, it wasn't me.

[miyoyo]

Yes, I do have it.

Here's the file's stat:

[root@(none) usr]# stat openssl.cnf
  File: openssl.cnf
  Size: 9374            Blocks: 20         IO Block: 1024   regular file
Device: b301h/45825d    Inode: 32833       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2010-06-24 22:28:31.000000000
Modify: 2010-06-24 22:28:31.000000000
Change: 2017-03-26 13:35:51.000000000

[root@(none) usr]#

[miyoyo]

Little update:

Managed to turn HTTPS off in the mobile.kobobooks.com request by modifying the binary

see thumbnail - editor

This is a "decompiled" version of the "MakeMobileRequest" function, and it uses "QUrl::setScheme", according to the official documentation, it is used to add the prefix "HTTPS" to the url string, therefore if the string is outsourced (From the QT lib for example) HTTPS stays

The request contains... the elusive Kobo eReader.conf's services, which is overwritten at every Sync!

It also contains store categories and "Tops", with their UUIDs, maximum amount of downloaded books and how often they should be updated.

I put it all in a pastebin, as it contains nothing confidential or personal (except for my DeviceID, which I redacted) over at Pastebin

If anyone wants to patch their libnickel themselves, replace 2E F7 F0 EB at 0x005BD094 as 00 BF 00 BF.

[kido.resuri]

Wow, nice! Deffinately a patch could be. The finished replaced bookstore could do the auto-patching if finished once. I may continue work on it with the help of this nice work of yours!
Also, which version did you use? A new firmware just came out.

[miyoyo]

I used 4.3.8966, i'm checking for this in the newest one

EDIT:
Yep, still there
http://i.imgur.com/762AFJI.png


Also, just a random idea, but a store that sources books from the gutenberg project would be amazing.