Replacing Kobo Bookstore

[tshering]

Originally Posted by kido.resuri

The problem really is that I currently can't create a zip that Kobo would accept, all the methods I tried failed, Kobo downloads the zip, but can't extract it's contents... If I create a zip with for example Total Commander's built in zip, then, and only then it works.

Did you compare the archives created by Total Commander and those by 7zip to get an idea what is different. It might suffice to have a look at the archives with 7zip File Manager, or if this does not help, inspect the archives with a hex editor.

[kido.resuri]

I did compare, but only with Total Commander's Viewer. The difference can be seen with this. But I don't know how to resolve this issue. I tried with 7zip, pkzip (the old, freeware 2.5 version, as i didn't find cli in the new, trial version), even winrar. I also tried with the windows built-in zip, with VB script, all produce different zip archives (can be seen in the TC Viewer), but all fail to work with Kobo. My "last" idea is to use TC's built in zip, but I need to do some research on it (it's probably in WCMZIP32.dll).

[PeterT]

Remember you need to make a tgz, which is a gzipped tar file. Does https://www.techwalla.com/articles/h...ile-in-windows help?

[davidfor]

Originally Posted by kido.resuri

I did compare, but only with Total Commander's Viewer. The difference can be seen with this. But I don't know how to resolve this issue. I tried with 7zip, pkzip (the old, freeware 2.5 version, as i didn't find cli in the new, trial version), even winrar. I also tried with the windows built-in zip, with VB script, all produce different zip archives (can be seen in the TC Viewer), but all fail to work with Kobo. My "last" idea is to use TC's built in zip, but I need to do some research on it (it's probably in WCMZIP32.dll).

What do you mean "fail"? The firmware checks for an updated using the API and if there is fetches the release notes, downloads the zip file, unpacks that and then prompts you to install. Which bit is failing? And are you doing all bits?

The above has always been the case ("always" is since I first looked). Maybe it also handled other types of compressed file, but I don't believe Kobo every distributed the firmware as anything except a zip file (and an SD card in the dim distant past).

If you think the issue is zip file compatibility, the device is running Linux. It is probably using gzip. Try that. And of coarse, make sure you can send the file that Kobo is pushing.

And of course, it is completely possible that Kobo have changed something. It is even possible that they are making changes to prevent exactly what you are trying to do. It won't be easy to tell until the next update happens and someone tracks what happens. Maybe you need to go back a firmware version or two and test again.

[kido.resuri]

I have no idea which bit is failing. The device downloads the zip, I can see the top bar showing the percent, but then nothing happens. No KoboRoot.tgz is extracted to /mnt/onboard/.kobo . If I have a zip created with Total Commander's built in zip, then it works perfectly. Now, with the latest FW, If I use KoboRoot.tgz as the update archive directly, the device downloads it, puts it in /mnt/onboard/.kobo, but does not reboot, nor it displays a message if the update type is "Available" or "Required". ("Forced" also does not reboot instatly.) If then I manually do a reboot, then it does the update, but not automaticaly. It worked with KoboRoot.tgz in fw 3.19.

I'll try gzip, maybe that works, thank You!

[PeterT]

Are you ensuring the files have ThE CoRReCt casing in their names?

[kido.resuri]

I'm releasing the current, developement version of this custom Kobo Sync Server.
Just unzip it to c:\, but unzipping it to any folder may work also.
There are two config files, in KoboSync\config:

send2kobo.conf

This is the config file of the send2kobo utility.
The contents are self-explaining, I'll later write some more on this, if needed.

KoboSync.conf

This is the configuration file for the Sync Server. Change $KoboUpdateFile to 'kobo-update.zip' to experiment with the zip method. You can also set the $UpdateType, as described in the file.

What is this and how to use it

The 'core' is basicly EasyPHP-DevServer-13.1VC11 (www.easyphp.org), with everything unneccessary removed, to keep the file size as low as possible. This webserver hosts the custom Kobo Sync Server, which is currently 2 php files, found in \localweb\1.0
This can be easily started by executing "EasyPHP-KoboSyncServer.exe". On first run, if you put the whole thing to anything other than c:\KoboSync, it will ask you if you want to re-generate the config files. Make sure you click "Yes". The indicator next to "Apache" should turn green and the label next to it "Started". If not, you can try manually starting by clicking on "Apache", or try restarting the application. If you minimize this window, it will hide in the tray.
Also make sure you allow Apache in the firewall, if it pops up.
Important: the webserver uses port 88 instead of 80!
You can test if it works by opening
http://localhost:88/1.0/UpgradeCheck/test in your browser. It should show you some text between { and }, containing the update url, etc. If the page fails to load

Next, you should install synchack to Kobo by executing:
c:\KoboSync\send2kobo\install-synchack.bat
This will patch your Kobo's firmware file if possible, by changing the UpdateCheck URL to your PC's hostname. As the warning say, this is a potentially dangerous task. So be sure to make a backup of your Kobo device before executing the install script!
It will ask you to plug Kobo in, then unplug (it will safely unmount it, but you will have to unplug it manually), plug it again after it reboots, then unplug it again.
This will also install "Send to Kobo" into the Send To right click menu.

This right-click menu plug-in will prepare a KoboRoot.tgz for the sync server with the neccessary scripts to add epubs to Kobo by the virtua SD card method. It works with 3.19.5761, maybe with 4.2.8432, but needs more testing as I killed my time with the zip issue.

If everything was OK, then if you tap "Sync" on the Kobo device, it will search for updates on your PC.

Can be used this now to try which zip method works...


I used code from GeoffR's firmware patch, and some found online mostly on stackoverflow. I hope it does not violate any license.

EDIT1+EDIT2: I just realised that I left some code which now prevents the update to work, during the bug-hunting. The download link is presented without the port number, and therefor Kobo can't download the update file, as the webserver is not running on port 80.
Also modified the files as some failed to work.
So if you downloaded before, please download it again.
The following files have changed:
KoboSync\localweb\1.0\UpgradeCheck
KoboSync\config\KoboSync.conf
KoboSync\localweb\1.0\firmwares
Thank you and sorry for the inconvenience.

[kido.resuri]

I had to re-upload the above zip, please read the edit comments.

[tshering]

Nice idea!
It might be reasonable to replace in install-synchack.bat

tools\get-libnickel.tgz

by

tools\data\get-libnickel.tgz

[kido.resuri]

Hmm.. It was late last night and I changed a few things after testing to be a bit more clear, and I might forgot to change that. Thanks!
I also found some other mistakes in the send2kobo.bat, so I'll soon update the attachment.
Sorry, my bad. That pesky zip issue realy took my time from the important things...

On the zip issue:
I examined the files with HEX editor, read about ZIP headers, found the differences, but no luck yet on finding the solution...

[kido.resuri]

I uploaded the modifications, please re-download if you downloaded it before. This should finally work. For the zip issue, I think I'm close to the solution. WinZip, and only WinZip might be able to compress with the settings that Kobo needs. It requires Deflate method with maximum compression. (general purpose bit flag (address 0x6) is set to 0x02, and compressing method (addr. 0x8) is set to 0x08)

[kido.resuri]

I can create zip with winzip, but winzip needs to be installed to work on the user's pc. I don't want that.
I found pkzip25, which also can create the required type of zip file, but not with files already compressed - like KoboRoot.tgz is. If I try to compress a text file for example, it will create a Kobo-compatible zip. Kobo will extract the contents to .kobo. But if I try to compress KoboRoot.tgz with the same settings, it creates a zip with "store only" method. That is very frustrating at the moment.

[kido.resuri]

I don't have the Kobo with me now, but I may found the solution with some workaround.
Could anyone of you please try this zip (make sure you don't re-zip it) with the KoboSync above?
This will add a placeholder.txt to the .kobo folder, and after the successful update, a test.txt to the USB root.
Don't forget to post here if it succeed!

It was created with 7za (the tar and tgz) and with PKZIP25.

[miyoyo]

I also did my little investigation, and here's what I found:

• There are only four HTTPS domains (excluding logins and oauths) in libnickel.so, these being:
  • api.kobobooks.com - As far as I can tell, it's just for system upgrades
  • storeapi.kobo.com - HTTPS Enforced, Pretty self-explainatory, user data, subscriptions, etc, but no dice on books. It does have a nice API tho.
  • services.kobobooks.com - This one is interesting, it fails on HTTP but if the request is done in HTTPS by something like WGET, we get values from the sqlite database. Maybe OTA settings?
  • partner.kobobooks.com - Not used, as far as I can tell.
• The current order of website calls is: api.kobobooks.com, then mobile.kobobooks.com (Still can't get this one as HTTP, the request is HTTPS even when every trace of HTTPS is wiped from libnickel.so), then storeapi.kobo.com and finally services.kobobooks.com
• StoreAPI and Services make use of the nx-kobo-userkey header, it's a UUIDv4, and is not linked to the hardware, maybe linked to the initial login?
• They also need a special Authorization header. I couldn't get one from the eReader, but I pulled one from a capture of the android app and it seems to work.
• The CA certificates are written at the end of nickel, and, except if you can forge one of the exact same size as another one, you won't go far. They are in the .der format

Also, I think another thing to look at should be the Desktop/Mobile apps, it's way easier to capture traffic from that.

Fun fact: their server also isn't W3C compliant, as it can return 404 when the only problem is that the request type isn't valid.

[kido.resuri]

I captured data by doing a man-in-the-middle: added a wifi AP to my PC's ethernet, and shared the real wifi connection by Windows. So all traffic goes through my PC. The I started Wireshark sessions on the ethernet interface, then replaced every https with http in the Kobo conf file. I also used Firefox extensions HttpRequester and "Modify Header Value (HTTP Headers)", and added the headers from the captured packets. This way I can get all data from the real Kobo servers, even if they are HTTPS. The only data I can't get with this method are from the URLs that I can't change to HTTP in Kobo, like mobile.kobobooks.com, this one seems to be responsible to download the books to the reader when it does a sync.
Then, when I have the captured real data, I - or someone else - can *easily* make a local webserver with the correct data format and responses. I haven't played with it since the last posts, as I didn't have much time (and my fiancée started using the eReader, didn't want to mess it up for her), but I could add my custom Bookstore entries to the Kobo device, like custom book categories, and it worked fine as far as I got.

If anyone else is interested in the reverse engineering, I would gladly share my experiences other than the above i described. The best would be to integrate it with Calibre, or maybe create a whole new "store" database software.

Edit: I also set up an apache webserver on my capture PC, and later changed all the URLs in Kobo conf to access this, and mimiced the answers of the real kobostore to get further in the data-layers.