Replacing Kobo Bookstore

[kido.resuri]

Hello dear developers and reverse-engineers!

I was wondering if it was possible to replace the Kobo Bookstore with something like calibre's content server, as Kobo Bookstore is not really available in my language (hungarian), so chances for me buying from that store is very unlikely. (I may would, if there was real hungarian content in the bookstore).
Replacing it would make a convinient way to transfer books to the eReader. Currently the built-in browser flickers too much when browsing and downloading from the content server, which unfortunately makes me sick (I'm sensitive for this kind of stuff).

I noticed the service links in the config file, but had no luck changing them at first try, always got some error on the eReader (Kobo Aura H2O).

Did anyone try to replace the Bookstore links/pages in Kobo? Is it possible?

Or is there any other, convinient way to transfer books wirelessly to the Kobo?



UPDATE:
You can try the first experimental version of synchack following the instructions in post #15.

[davidfor]

Sorry, it's just not going to happen. The URLs listed in the config file are the least of you worries. There is a lot of code in nickel for parsing the results and displaying them. And you would probably need to do it at both ends. Change nickel so that it would treat the calibre server as it's shop, and change the pages sent back from the server to more closely match what nickel is expecting.

For the occasions when I need to download from calibre over WiFi, I use the browser. It handles the pages fairly well and you can download epubs or kepubs with no problems.

[kido.resuri]

Sorry to hear. Still i don't see it impossible, just it would require too much digging and hacking.
I tried the browser-based download, it works well, but the screen flickers too much, and this is something my stomach doesn't tolerate (I have some sort of sickness of this kind).

[Ken Maltby]

You can use KOReader's OPDS support to view site catalogs for the ebooks and download them wirelessly, including your local calibre media server catalog. KOReader also can be recognized as a wireless device in calibre (like calibre companion on Android).

There is also new Cloud Storage & FTP support. As well as the zsync support, for transferring between devices running KOReader.

The OPDS sites that come with KOReader include; Project Gutenberg, Feedbooks, ManyBooks, Internet Archive, as well as the Local calibre catalog. You can also add your own, with the "Add new OPDS catalog" selection.

Luck;
Ken

PS; Once you have downloaded a book using KOReader you can read it using the stock reading software (assuming it is in a format that Nickel can use).
Also: the latest KOReader download site is here.

[kido.resuri]

Okay, I started playing with the webstore api of Kobo, I'm especially interested in hijacking the Kobo Sync. Did anyone else made some reverse-engineering on this? I hoped the Sync also uses http or https on port 80 or anything set in the config, but this is not the case.

I need to find out the port for Sync to make any progress, so this is the next task for now.

This is what I'm expecting from this experiment: writing a desktop app for the PC to hijack the Sync in Kobo to upload books and maybe "updates", like patches and anything that one would put in a KoboRoot.

[markyisri]

sounds interesting! any updates?

[Ken Maltby]

Originally Posted by kido.resuri

Okay, I started playing with the webstore api of Kobo, I'm especially interested in hijacking the Kobo Sync. Did anyone else made some reverse-engineering on this? I hoped the Sync also uses http or https on port 80 or anything set in the config, but this is not the case.

I need to find out the port for Sync to make any progress, so this is the next task for now.

This is what I'm expecting from this experiment: writing a desktop app for the PC to hijack the Sync in Kobo to upload books and maybe "updates", like patches and anything that one would put in a KoboRoot.

I wouldn't recommend it, but you have seen the Kobo Desktop PC program that Kobo provides, right? It's not something I would use but, as I understand it, it pretty much does what you describe.

Luck;
Ken

[kido.resuri]

Kobo Desktop does not sync sideloaded books, does it?

[davidfor]

Originally Posted by kido.resuri

Kobo Desktop does not sync sideloaded books, does it?

None of the Kobo software syncs non-Kobo sourced books.

[kido.resuri]

Now that's what I knew too, and that is why I thought "hijacking" Sync.

[kido.resuri]

I made some early progress with this, I was able to get data from kobostore, the respond for initialization request is the following (formatted to be more readable, it is really one line):

{
"Resources":
{
"user_profile":"https://storeapi.kobo.com/v1/user/profile",
"checkout_borrowed_book":"https://storeapi.kobo.com/v1/library/borrow",
"library_sync":"https://storeapi.kobo.com/v1/library/sync",
"library_items":"https://storeapi.kobo.com/v1/user/library",
"add_entitlement":"https://storeapi.kobo.com/v1/library/{RevisionIds}",
"delete_entitlement":"https://storeapi.kobo.com/v1/library/{Ids}",
"tags":"https://storeapi.kobo.com/v1/library/tags",
"autocomplete":"https://storeapi.kobo.com/v1/products/autocomplete",
"user_reviews":"https://storeapi.kobo.com/v1/user/reviews",
"user_ratings":"https://storeapi.kobo.com/v1/user/ratings",
"user_recommendations":"https://storeapi.kobo.com/v1/user/recommendations",
"taste_profile":"https://storeapi.kobo.com/v1/products/tasteprofile",
"fte_feedback":"https://storeapi.kobo.com/v1/products/ftefeedback",
"featured_lists":"https://storeapi.kobo.com/v1/products/featured",
"category":"https://storeapi.kobo.com/v1/categories/{CategoryId}",
"exchange_auth":"https://storeapi.kobo.com/v1/auth/exchange",
"device_auth":"https://storeapi.kobo.com/v1/auth/device",
"device_refresh":"https://storeapi.kobo.com/v1/auth/refresh",
"post_analytics_event":"https://storeapi.kobo.com/v1/analytics/event",
"user_loyalty_benefits":"https://storeapi.kobo.com/v1/user/loyalty/benefits",
"reading_state":"https://storeapi.kobo.com/v1/library/{Ids}/state",
"library_metadata":"https://storeapi.kobo.com/v1/library/{Ids}/metadata",
"update_accessibility_to_preview":"https://storeapi.kobo.com/v1/library/{EntitlementIds}/preview",
"rename_tag":"https://storeapi.kobo.com/v1/library/tags/{TagId}",
"delete_tag":"https://storeapi.kobo.com/v1/library/tags/{TagId}",
"quickbuy_create":"https://storeapi.kobo.com/v1/store/quickbuy/purchase",
"product_reviews":"https://storeapi.kobo.com/v1/products/{ProductIds}/reviews",
"review":"https://storeapi.kobo.com/v1/products/reviews/{ReviewId}",
"product_recommendations":"https://storeapi.kobo.com/v1/products/{ProductId}/recommendations",
"product_nextread":"https://storeapi.kobo.com/v1/products/{ProductIds}/nextread",
"product_prices":"https://storeapi.kobo.com/v1/products/{ProductIds}/prices",
"book":"https://storeapi.kobo.com/v1/products/books/{ProductId}",
"book_subscription":"https://storeapi.kobo.com/v1/products/books/subscriptions",
"featured_list":"https://storeapi.kobo.com/v1/products/featured/{FeaturedListId}",
"category_featured_lists":"https://storeapi.kobo.com/v1/categories/{CategoryId}/featured",
"category_products":"https://storeapi.kobo.com/v1/categories/{CategoryId}/products",
"library_prices":"https://storeapi.kobo.com/v1/user/library/previews/prices",
"library_book":"https://storeapi.kobo.com/v1/user/library/books/{LibraryItemId}",
"library_stack":"https://storeapi.kobo.com/v1/user/library/stacks/{LibraryItemId}",
"tag_items":"https://storeapi.kobo.com/v1/library/tags/{TagId}/Items",
"quickbuy_checkout":"https://storeapi.kobo.com/v1/store/quickbuy/{PurchaseId}/checkout",
"rating":"https://storeapi.kobo.com/v1/products/{ProductId}/rating/{Rating}",
"authorproduct_recommendations":"https://storeapi.kobo.com/v1/products/books/authors/recommendations",
"external_book":"https://storeapi.kobo.com/v1/products/books/external/{Ids}",
"remaining_book_series":"https://storeapi.kobo.com/v1/products/books/series/{SeriesId}",
"content_access_book":"https://storeapi.kobo.com/v1/products/books/{ProductId}/access",
"delete_tag_items":"https://storeapi.kobo.com/v1/library/tags/{TagId}/items/delete",
"review_sentiment":"https://storeapi.kobo.com/v1/products/reviews/{ReviewId}/sentiment/{Sentiment}",
"products":"https://storeapi.kobo.com/v1/products",
"categories":"https://storeapi.kobo.com/v1/categories",
"image_host":"//kbimages1-a.akamaihd.net/",
"store_host":"store.kobobooks.com",
"store_home":"store.kobobooks.com/{language}-{region}",
"social_authorization_host":"https://social.kobobooks.com:8443",
"social_host":"https://social.kobobooks.com",
"discovery_host":"https://discovery.kobobooks.com",
"eula_page":"https://www.kobo.com/termsofuse?style=onestore",
"password_retrieval_page":"https://www.kobobooks.com/passwordretrieval.html",
"sign_in_page":"https://secure.kobobooks.com/auth/login?redirectUrl=https://store.kobobooks.com/",
"facebook_sso_page":"https://secure.kobobooks.com/auth/Facebook/sso",
"store_search":"https://store.kobobooks.com/{culture}/Search?Query={query}",
"store_top50":"https://store.kobobooks.com/{culture}/ebooks/Top",
"store_newreleases":"https://store.kobobooks.com/{culture}/List/new-releases/961XUjtsU0qxkFItWOutGA",
"registration_page":"https://secure.kobobooks.com/auth/register?redirectUrl=https://store.kobobooks.com/",
"privacy_page":"https://www.kobo.com/privacypolicy?style=onestore",
"stacks_host_productId":"https://store.kobobooks.com/collections/byproductid/",
"book_detail_page":"https://store.kobobooks.com/{culture}/ebook/{slug}",
"book_detail_page_rakuten":"http://books.rakuten.co.jp/rk/{crossrevisionid}",
"book_landing_page":"https://store.kobobooks.com/ebooks",
"magazine_landing_page":"https://store.kobobooks.com/emagazines",
"purchase_buy":"https://store.kobobooks.com/purchase/buy/",
"purchase_buy_templated":"https://store.kobobooks.com/{culture}/purchase/buy/{ProductId}",
"love_points_redemption_page":"https://store.kobobooks.com/{culture}/KoboSuperPointsRedemption?productId={ProductId}",
"categories_page":"https://store.kobobooks.com/ebooks/categories",
"redeem_interstitial_page":"https://store.kobobooks.com",
"love_dashboard_page":"https://store.kobobooks.com/{culture}/kobosuperpoints",
"help_page":"http://www.kobo.com/help",
"image_url_template":"https://kbimages1-a.akamaihd.net/{ImageId}/{Width}/{Height}/false/image.jpg",
"image_url_quality_template":"https://kbimages1-a.akamaihd.net/{ImageId}/{Width}/{Height}/{Quality}/{IsGreyscale}/image.jpg",
"customer_care_live_chat":"https://kobo.frontlinesvc.com/app/chat/chat_landing/email/{UserEmail}",
"free_books_page":
{
"EN":"https://store.kobobooks.com/{culture}/p/free-ebooks",
"FR":"https://store.kobobooks.com/{culture}/p/livres-gratuits",
"IT":"https://store.kobobooks.com/{culture}/p/libri-gratuiti",
"NL":"https://store.kobobooks.com/{culture}/p/gratis-boeken",
"PT":"https://store.kobobooks.com/{culture}/p/livros-gratis",
"ES":"https://store.kobobooks.com/{culture}/p/libros-gratis",
"DE":"https://store.kobobooks.com/{culture}/p/kostenlose-ebooks",
"TR":"https://store.kobobooks.com/{culture}/p/ucretsiz-kitap"
},
"use_one_store":"False",
"kobo_superpoints_enabled":"False",
"kobo_onestorelibrary_enabled":"False"
}
}

Looking at the above, it is very similar to the links in the Kobo config file, but a bit more.
The response for the library_sync is (again, formatted):

[
{
"NewEntitlement":
{
"BookEntitlement":
{
"ActivePeriod":
{"From":"2016-12-27T00:43:44.0000000Z"},
"IsRemoved":false,
"Status":"Active",
"Accessibility":"Full",
"CrossRevisionId":"<there was some id here>",
"RevisionId":"<there was some id here>",
"IsHiddenFromArchive":false,
"Id":"<there was some id here>",
"Created":"2016-12-27T00:43:44.0000000Z",
"LastModified":"2016-12-27T00:43:44.0000000Z",
"IsLocked":false,"OriginCategory":"Free"},
"ReadingState":{"EntitlementId":"<there was some id here>",
"Created":"2016-12-27T00:43:44.0000000Z",
"LastModified":"2016-12-27T00:43:44.0000000Z",
"StatusInfo":{"LastModified":"2016-12-27T00:43:44.0000000Z",
"Status":"Undefined",
"TimesStartedReading":0},
"Statistics":
{"LastModified":"2016-12-27T00:43:44.0000000Z"},
"CurrentBookmark":{"LastModified":"2016-12-27T00:43:44.0000000Z"},
"PriorityTimestamp":"2016-12-27T00:43:44.0000000Z"},
"BookMetadata":
{"CrossRevisionId":"<there was some id here>",
"RevisionId":"<there was some id here>",
"Publisher":{"Name":"","Imprint":""},
"Language":"en",
"Isbn":"1990001706406",
"Slug":"english-hungarian-dictionary-angol-magyar-szotar",
"CoverImageId":"c0ffee00-0000-0000-0000-000000000000",
"IsSocialEnabled":true,
"WorkId":"<there was some id here>",
"ExternalIds":[],"IsPreOrder":false,
"EntitlementId":"<there was some id here>",
"Title":"English-Hungarian Dictionary: Angol-magyar szĂłtĂĄr",
"Description":"",
"Categories":[],
"DownloadUrls":
[
{
"DrmType":"None",
"Format":"PDF",
"Url":"http://storedownloads.kobo.com/download?downloadToken=eyJ0eXAiOjIsInZlciI6IjEiLCJwdHlwIjoiQ2xpZW50QVBJRG93bmxvYWRUb2tlbiJ9.WpgGHsQlcKMKpr6j3cd1xw.nXBd4aYCRCrzO2yqbXOB6fckaJN8WMGEUyVH_G9-WGLIv7qUy3jOyASHthHfHtzYg1lLxnaY3WZ18PEU9tf2M0CWts7XIFuEK_U5WRiJPyL2r0epJKnQHXiYMw0eL9cGue8j5j4KPosOQF19pSLbYHSBESNRDna34WaPr1M40g-Hdxom2B4wIpdmEi5gGNVpApU6My1TZPqWZ9yQs_jhtaiMVLCl-N0lRZNKOhoEal5diHbRPdbt7CfOiuVeGnircKL_CSyrd18bHQcUQkPH1GglQQ54LW7b7jl50wjVZGLNUkup988YFo6ToTbCznz74zroFEkby3uehAJgkqe2csdKsJLw-ndKfvBE-o2cnVjlCKBBRn63fWTmMLhfTaRkVB7G3VYlhZFxbdmgG0n8iMbzFK-8aOkj_w5dEIgYEg57GvR1euuyhF4VpSyGUDE9NH9hdoWVk2e42A1Flbe0l-vkSQyjTKxpMYM5kPyBCIMirxQ3fuTjU5evMrj35mTFhUwdJTI7vv64pEcwA2bC8A",
"Platform":"Generic","Size":0},
{
"DrmType":"None",
"Format":"EPUB",
"Url":"http://storedownloads.kobo.com/download?downloadToken=eyJ0eXAiOjIsInZlciI6IjEiLCJwdHlwIjoiQ2xpZW50QVBJRG93bmxvYWRUb2tlbiJ9.tvqUq_oRftclStLFqMWODg.Xfl-s8oUIns9kKTmKnYbXHwtEv9Dq5PLbKQo2XMLx3ruCMgHwwZ-PKo8p4JAnK5U8q6n0OY6YKdIfoiFKg_HWkuOa3n0Y7Slk884_LPDtI2PqDtfuFxES8mrc_JNDNxL3ornuxBj0FOHBeTQ3ItV3x5lFhlFfFQW1oQkqLAKXJOg8-lwn5Atc9iZQ4q2XwRhYG0QnHBCdDT9hQTt76kmRc-tfAaE3crfIxNsc3eJSkTURPEALC6UQv0BQMLDJ9YEqfM5XqkLMqPqWQhuXNjicB_j9QwmeTR4kmmAARfQAt8ZRsPlGmBirNBSRhFNTeDs06u0fNCn1he5qeOSWt4Ln3nmmTwJ8HUg39MMx02OI7Pbg0QomsB7jKxNUFzAjmghqdriQEnYyUbYNp6V4ldfUfyfr7ySMgB1ZA2N5L9_oawldJepQD5SnkzeEo98aqCG9xM0V3rLW_CYLMNQ0FK2mPhSnGJLTqebjV3fUNbNg8DrEO_gkbb-IL7EzCjDSELNwCJQkqZUtB4reil3CWedUQ",
"Platform":"Generic",
"Size":0
}
],
"Contributors":["Ferencz Bizonfy"],
"Series":{},
"CurrentDisplayPrice":
{
"TotalAmount":0,
"CurrencyCode":""
},
"CurrentLoveDisplayPrice":
{"TotalAmount":0},
"IsEligibleForKoboLove":false,
"PhoneticPronunciations":{}}}}
]

As you can see from the above, there is the actual download link for the book (don't worry, this is a free, DRM-free e-book from the kobo bookstore). I'm expecting this to be hijacked by a custom webserver, pretty easily maybe.

Now, the Kobo starts the sync with some https request to some url not shown in the config, but is somewhere in the kobobooks.com domain, this is what i'm reverse engineering now, without this, it won't ask for the init.

More updates to come later...

[kido.resuri]

Some new findings, still too early stage. The Sync process at first does a https request like this:

https://api.kobobooks.com/1.0/UpgradeCheck/Device/<long devicecode with many zeros in .kobo/version file>/Kobo/<version number>/<Kobo serial number, found in .kobo/version file>/

The answer is - if there is an update available for the device - (pasted only the formatted xml):

<UpgradeCheckResult>
  <Data i:nil="true"/>
  <ReleaseNoteURL>http://api.kobobooks.com/1.0/ReleaseNotes/52</ReleaseNoteURL>
  <UpgradeType>UpgradeRequired</UpgradeType>
  <UpgradeURL>http://download.kobobooks.com/firmwares/kobo5/january2016/kobo-update-3.19.5761.zip</UpgradeURL>
</UpgradeCheckResult>

I was - almost - able to hijack this, well, I need to patch some urls in the libnickel file to make it work, and do some hosts file hacking also. Now the biggest problem was that it wants to request https urls, but to hijack that, i would need a real signed https server, which is a bit difficult. I think I found a way to bypass this, but I have to play with it a little more.

More to come later, if anyone else - other than me - is interested.

[kido.resuri]

Okay, news: successfully hijacked the wifi update, I could send update files via the built-in Sync!
It was basicly a test KoboRoot.tgz zipped, containing only one file /mnt/onboard/list.txt to see if it works.
And it does. Kobo downloaded the file from my webserver, unzipped and did the update. The list.txt appeared on the USB mount. Also, if I select "Update Later", the KoboRoot.tgz waits for the next restart in /mnt/onboard/.kobo, as it should.



This also needs a libnickel patch to work, by the way.

Expect more updates on topic in the near future!

[patrik]

Nice!

[kido.resuri]

Okay, so for those of you who wants to try this, I post the required modifications and tools for a custom update server.
This is highly in developement stage, so use everything at your own risk.

Later I will make a pack that installs as much as possible automatically, but now you are on your own to set it up.
EDIT: This is for firmware version 3.19.5671!
EDIT2: I just tried with 4.2.8432, and it works!


First, you will need a fix IP for your PC. This can be done at best by a fix DHCP lease in your router. You have to figure it out, how to set this for your router, but look under DHCP section, respectively.

You will need to install an apache webserver on your PC. If you are using windows, you can use http://www.easyphp.org/ for example, it has more than you need, but will be a good start for now. I'm using it too, by the way, an older version actually, so I might not be able to help with the most recent version to set up. But easyphp should have everything you need, so a simple install should do.

Anyhow, if you have your webserver up and running, you can test it by opening your browser and try to navigate to http://localhost/. If you see a directory list or anything other than an error, then it's ok.
Now you have to find the www root of your webserver. The latest version of EasyPHP uses <installdir>\eds-www\

Now extract the attached zip file to your www-root folder.
There is only one config file now: 1.0\config.inc
You can edit it and set the filename for the update.
You can put your KoboRoot.tzg files in a zip in the downloads folder, but make sure you have the same filename set in config.inc
And you can also edit the release-notes.html in the downloads folder if you like, but I haven't seen it showing up in Kobo after the update, so I guess it doesn't matter.
EDIT: Release Notes can be viewed prior to downloading the update, when the UpdateType is set to 1 in config.inc

After this, open your browser and try this link:
http://localhost/1.0/UpgradeCheck/KoboDummyUpdateCheck/

You should see something like this:
{"UpgradeType":2,"UpgradeURL":"http://<yourhostname>/1.0/firmwares/kobo-update-3.19.5761.zip","ReleaseNoteURL":"http://<yourhostname>/downloads/release-notes.html","Data":null}

Now copy and browse to the UpgradeURL and try to download the zip file.
Make sure you can unzip it and can see the contents of it. If not, don't go to the next step, but make sure you have the right zip file in the downloads folder.

If the downloaded zip file is good, then try to refresh the first URL. You should now see this:
{"UpgradeType":0,"UpgradeURL":null,"ReleaseNoteURL ":null,"Data":null}

And there is a new file created in downloads: count_kobo-update-3.19.5761.zip.txt
This makes sure that Kobo doesn't download the update file again after the update, thus geting into an update loop. To enable the update again, simply delete this count_kobo-update-3.19.5761.zip.txt file.
You can download the update as many times you want from the link tho.

So, if you have the above nice and running, let's move on to the Kobo.


You will need to apply this patch to libnickel:

<Patch>
patch_name = `Remove https`
patch_enable = `yes`
## api.kobobooks.com UpgradeCheck
find_base_address = `h\0t\0t\0p\0s\0:\0/\0/\0a\0p\0i\0.\0k\0o\0b\0o\0b\0o\0o\0k\0s\0.\0c\0o\0m\0/\01\0.\00\0/\0U\0p\0g\0r\0a\0d\0e\0C\0h\0e\0c\0k\0/\0%\01\0/\0%\02\0/\0%\03\0/\0%\04\0/\0%\05`
replace_string = 0000, `h\0t\0t\0p\0s\0:\0/\0/\0a\0p\0i\0.\0k\0o\0b\0o\0b\0o\0o\0k\0s\0.\0c\0o\0m\0/\01\0.\00\0/\0U\0p\0g\0r\0a\0d\0e\0C\0h\0e\0c\0k\0/\0%\01\0/\0%\02\0/\0%\03\0/\0%\04\0/\0%\05`, `h\0t\0t\0p\0:\0/\0/\0s\0a\0p\0i\0.\0k\0o\0b\0o\0b\0o\0o\0k\0s\0.\0c\0o\0m\0/\01\0.\00\0/\0U\0p\0g\0r\0a\0d\0e\0C\0h\0e\0c\0k\0/\0%\01\0/\0%\02\0/\0%\03\0/\0%\04\0/\0%\05`
</Patch>

And add this line to Kobo's /etc/hosts file:

<your PC's IP> sapi.kobobooks.com

After this (and if you made a firewall exception for apache), Kobo will try to update from your webserver. Everything else will work from kobobooks.com and kobo.com, so it does not affect the Bookstore and anything else.



P.S.: I might stop reverse-engineering at this point, as one could add content to the Kobo by an automated script (I will do this later), even adding to Kobo's database.
However, my main goal is to take over the Bookstore completly, I have some untold progress with that also, it just requires some more time. I could make Kobo get my custom list of Categories in the store, for example. More updates on this later. Untill then, try the above if you dare, and don't forget to post anything related.