Calibre not seeing Kobo in Windows 10 Preview builds

[Dave R]

Originally Posted by JDesat

I actually ended up using a bit of both solutions found on this thread.

I used the partition tool pointed to in message 46 but then I basically followed DNSB's steps. The only difference was that, after formatting, I unplugged the kobo and went through the steps to set it up again (choosing english and connecting it with kobo desktop to reload the dictionaries, etc.). After that I was able to go to the non-Anniversary machine and upload the backup to the kobo again without a problem. The only issue was that almost anytime it asked me if I wanted to over-right a file from the backup onto the kobo I chose no, and kept the files that kobo desktop had just uploaded from setup.

I hope that helps!

Thank you !
1. Using the partition tool to set a FAT32 MBR partition
2. Switching off then on to allow the Kobo to go through its set up procedure
3. Re-loading only the backup files that were missing after start up

..worked superbly on my H2O.

It even passed the ultimate test and loaded public library books via ADE.






Dave

[frostschutz]

I installed Windows 10 in a VM today. Using Qemu-KVM with USB Passthrough for the Kobo H2O.

Windows wants to format it. Says Drive E:\, invalid filesystem, yadda yadda.

So I shutdown Windows, do a fsck.fat.

# fsck.vfat /dev/sdj 
fsck.fat 4.0 (2016-05-06)
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  489:8d/00, 490:ef/00, 491:02/00, 492:d2/00, 494:ff/00, 495:ff/00, 496:ff/00
  , 497:ff/00, 498:ff/00, 499:ff/00, 500:ff/00, 501:ff/00, 502:ff/00
  , 503:ff/00, 504:ff/00, 505:ff/00, 506:ff/00, 507:ff/00, 508:ff/00
  , 509:ff/00
1) Copy original to backup
2) Copy backup to original
3) No action
? 2
Perform changes ? (y/n) y
/dev/sdj: 8176 files, 308555/829223 clusters

Choosing 2) Copy backup to original, Windows is happy to mount it again. No formatting required.

Even if this was the problem, this is something Windows should be able to do by itself.

This is also a fix you should be able to do in Windows directly if there is a fsck.fat (program that repairs FAT filesystems) for Windows? Or just a dd, copy 6th sector to 1st sector. (Haven't tried that myself use with care).

Unfortunately this fix isn't persistent. Once the Kobo reboots, it's back to square one.

No clue how letting Windows format it makes the fix permanent. I'll have to do that later. If somehow Kobo's factory reset counter stops modifying the first sector after Windows formatted it - then that would probably be it.

Note that VM and USB Passthrough is not quite the same as a real Windows install. Behaviour might differ.

Edit: Okay, never mind rebooting, even unplugging/replugging the H2O makes the problem reappear so this doesn't help much...

[frostschutz]

So what happens after unplugging the Kobo from USB?

Nickel spawns two processes, one is `dd if=/dev/mmcblk0p3 bs=512 count=1` (read the first sector of the data partition, not sure why nickel needs dd for that but okay), then it does something with that data and spawns `dd of=/dev/mmcblk0p3 bs=512 count=1` so the first sector is overwritten. (perhaps it's forcibly resetting the factory reset counter to 0?)

Replugging the Kobo to USB and Windows no longer mounts and fsck.fat complains about differing original/backup sectors again.

I think Kobo definitely needs to stop messing with this sector. There happens to be a free sector right next to it (last sector of the recovery partition is unused because that happens to have an odd number of sectors)... or just use a file in the rootfs already, what's the point of messing with these sectors in the first place.

Still unclear how this behaves any different when Windows formats it.

Maybe I should try copying original to backup instead and hope that the first sector is always messed with in a way that stays identical to the backup.

[frostschutz]

Originally Posted by frostschutz

Maybe I should try copying original to backup instead and hope that the first sector is always messed with in a way that stays identical to the backup.

Nope, not good at all. Original and backup identical, fsck.fat is happy with it, Linux is happy with it, Kobo is happy with it, Win10 hates it. I had a good backup sector, bad original sector, and copying original to backup ruined it.

List of differences (after reboot and all) is now considerably smaller than before.

# fsck.fat /dev/sdj 
fsck.fat 4.0 (2016-05-06)
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  489:8d/00, 490:ef/00, 491:02/00, 492:d2/00
1) Copy original to backup
2) Copy backup to original
3) No action
?

Bytes 489-492 are different now, earlier it was 489-509. So it's something in 493-509 that Win10 doesn't like. And choosing 2) no longer helps.

I have to read up what those bytes are supposed to be in a FAT32 header. Also I'm completely unclear where change of those other bytes originally came from.

Edit: Apparently byte 90-509 is supposed to be 420 bytes of bootloader code. Which shouldn't be used by anything or for anything as no one is booting from this. Which is probably why Kobo thought it would be okay to mess around with. Why does bootloader code matter to Win10... and where does it come from... a mystery.

[frostschutz]

Sorry for monologueing... it's for science!

So I decided to experiment and zero out those 420 bytes entirely.

dd bs=1 count=420 seek=90 if=/dev/zero of=/dev/sdj

and same for the backup sector. As a result, fsck.fat was happy, Linux was happy, Win10 was happy, Kobo was happy - or so I thought, after rebooting Kobo it's back to square one:

# fsck.fat /dev/sdj 
fsck.fat 4.0 (2016-05-06)
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  494:ff/00, 495:ff/00, 496:ff/00, 497:ff/00, 498:ff/00, 499:ff/00, 500:ff/00
  , 501:ff/00, 502:ff/00, 503:ff/00, 504:ff/00, 505:ff/00, 506:ff/00
  , 507:ff/00, 508:ff/00, 509:ff/00
1) Copy original to backup
2) Copy backup to original
3) No action
?

Kobo set bytes 494-509 to 0xff at some point, and Win10 doesn't like it for some reason. Primadonnas...

Choosing 2) (which zeroes out those bytes since it's still zero in the backup) to make Win10 happy, that works again.

So I was wondering, what's the smallest change that makes it work (or makes it not work)? I zeroed out a single byte 509... didn't work, 508-500 still nope, ... seriously what am I even doing... 499-495 oh hey it works. So if 494 alone is 0xff that's fine for some reason. What the heck.

And for some reason, that sticks until reboot. Unplugging the Kobo, replugging it, 494 alone stayed 0xff and everything else stayed 0x00. After reboot it's bytes 489-494 that differ.

# fsck.fat /dev/sdj 
fsck.fat 4.0 (2016-05-06)
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  489:8d/00, 490:ef/00, 491:02/00, 492:d2/00, 494:ff/00
1) Copy original to backup
2) Copy backup to original
3) No action

Yet, Win10 is still happy with this. So I've finally reached a state that all OS like and survives reboots of Kobo.

All in all this is incredibly frustrating.

Why does Win10 like, and not like, something that no one else cares about (bootloader code). Why are some byte combinations there problematic and others aren't. It all seems very random.

Why does Kobo sometimes overwrite bytes 495-509 with 0xff and sometimes leave them alone. How is this decision made. It seems very random, too.

A couple of unplugs, reboots later, Win10 is still happy with this Kobo partition. Equilibrium of randomness.

The main problem is reaching this state in the first place. If you just write zeroes to all 420 bytes, Kobo just 0xff everything afterwards. This happens even if you just punch the Reset button instead of doing a regular disconnect. (I tried the reset in case it was just some cache somewhere but nope.)

So you have to zero everything, reboot, and then zero bytes 495-509 again.

dd bs=1 seek=90 count=420 if=/dev/zero of=/dev/sdj
sync
# unplug, reboot kobo
dd bs=1 seek=495 count=15 if=/dev/zero of=/dev/sdj
sync

Or at least that's what I ended up with. Of course you could probably dump the working final state onto it directly and hope it sticks.

If someone can reproduce this, it could be put in a mod.

And if this is anywhere close to the mark, this should be in an official firmware update. Stop messing with this sector. And fix whatever may be problematic in there. Zeroing out entirely works but only if Kobo leaves it alone afterwards instead of spamming it with 0xff.

[rtiangha]

Keep it coming frostschutz. I LOVE science!!

If I had to guess, the new Windows 10 is trying to guard against boot sector viruses, not mounting anything whose MBR seems weird or not what is to be expected (just what is to be expected beats me; hopefully there's a FAT32 MBR standard out there). But that's just a guess.

Why Kobo chooses the FAT32 partition to do its dd weirdness is beyond me. According to my Glo HD image, there's about 16MB of unallocated space at the beginning of the card, which is way more than is needed to hold the master MBR and partition table. Why don't they do their dd weirdness there?

[JDesat]

While this thread seems to have gotten too technical for me, it doesn't really matter since the formatting thing fixed my issue. That said I just wanted to point out that Calibre now seems to take a long time to "Send metadata to device". The actual books get transferred in a few seconds, but then the metadata sync is taking over 3 minutes. It never took that long before this issue came up in the first place. It's not a huge problem or anything, I just wanted to point it out and see if others noticed the same thing. Oh, it happens both when sending books to the kobo and when deleting books from the kobo (if I do that in Calibre).

[rtiangha]

Also, if we know which byte to write that makes all OSes happy and can survive a reboot (although probably not a factory reset), that could be put in a script and flashed to rootfs to be executed once (or even at regular intervals), and that could be one way for those with existing FAT32 partitions but who didn't back up their contents to recover without having to be forced to wipe everything (it would still require them to have access to a second working PC to copy the KoboRoot.tgz file over, though, unless this were to be the fix that Kobo ends up using and sends it to devices via OTA).

[davidfor]

Originally Posted by frostschutz

Sorry for monologueing... it's for science!

Monologueing is fine. It gives the hero a chance to escape your trap and foil your dasterdly plot.

I'd looked at the bootsector changes from the output of filesystem check. Fiddling with them was beyond what I'm normally willing to do. It would be interesting to know why Kobo are doing it this way. The file systems should be mounted at this point, so writing to a file should be OK. It's probably a carry-over from the original Kobo devices that has been working and hence not changed.

[z270]

I spoke with Kobo customer care, they're on this and are in the process of producing a fix. If you contact them they ask for information on your system and scenario to get a better idea of the problem and what it affects.

[frostschutz]

I've created a mod that performs the changes I described above on the Kobo. If successful it should make Kobo <-> Win10 compatible without formatting/losing any files.

Take it with a grain of salt since I've only tested with my H2O, and only with a Win10 virtual machine, not a real Win10 PC.

Just because it works for me doesn't mean it will work for you. Use at your own risk.

Backup your files first.

To install, copy KoboRoot-fatpatch.tgz » .kobo/KoboRoot.tgz

[yes you have to do that on a PC that still sees the KOBOeReader drive, or with a Linux LiveCD - someone else will have to find/write a Windows program that does this directly in Win10]

After installing the mod, the reader will reboot, display a fat clown, and reboot again. At that point the mod performed the changes and removed itself (you have to reinstall the mod to re-try), and things should just work.

If you use this, please report back whether it worked or not.

Edit v2: Updated to have a longer delay before the operation starts, maybe that works better for older/slower devices

Edit v3: There was a bug that affected devices with 2GB storage which has FAT32 backup sector in a different place. Unbelievable.

Bug Notice: This may cause power/screensaver settings to vanish after applying this mod. I'm not sure what's causing this yet. Workaround: rename the .kobo/Kobo/Kobo eReader.conf file to make Kobo return to default settings.

Edit v4: There was a syntax error that could cause the mod to get stuck and never reboot.

Experimental windows version here: https://www.mobileread.com/forums/sho...44#post3368644

~~~ Removed the fatpatch hack since Windows has released a patch that resolves this issue. ~~~

[tshering]

Originally Posted by frostschutz

I have to read up what those bytes are supposed to be in a FAT32 header. Also I'm completely unclear where change of those other bytes originally came from.

This is where nickel tells pickel whether it has failed several times in a row to run successfully.

[beachboybogart]

That wasn't the question. The question was, does this fix work for actually getting books onto your Kobo.

Originally Posted by JSWolf

If Windows sees the Kobo properly and Calibre does not, you just follow the directions to report what Windows does see so Kovid can fix it in Calibre. Problem solved.

[davidfor]

Originally Posted by beachboybogart

That wasn't the question. The question was, does this fix work for actually getting books onto your Kobo.

No, the question is can the Windows machine see the device, mount it as a drive and write to it. If it can, you can copy books to the device. Once that is working, you can worry about calibre. And I can almost guarantee that calibre will work.

[beachboybogart]

Originally Posted by davidfor

No, the question is can the Windows machine see the device, mount it as a drive and write to it. If it can, you can copy books to the device. Once that is working, you can worry about calibre. And I can almost guarantee that calibre will work.

I would simply like to know if anyone has had success doing this. I agree that if the device can be written to in Windows, it should work. Trying it out with the Kobo app or Calibre is one way to validate the full recovery of functionality. As of now, no one has stated whether everything works the way it used to.