calibre+kindle+linux = fsync disconnect problem still

[yorikk]

Linux 5.4.74-1-lts
calibre 5.4.2-1
kindle fw 5.5.0

USB connection

Kindle disonnected when books synced.
VFAT errors in logs etc, nothing synced.

echo "write through" > /sys/block/<dev>/queue/write_cache

Solved the problem but need to do it every time.

Maybe there is something other to fix it permanently?

Another Linux kernel?
Another Calibre version?
Another Kindle firmware?

Please help

[kovidgoyal]

I highly doubt that's the fsync problem. Try resetting your kindle.

[yorikk]

Kovidgoyal,

Tried from Kindle menu:

reboot
full reset

Tried few good USB cables.

Just copy files is ok even on 1Gb file.

Then I search for bugs and found few links: kernel bug (no fix), calibre bug (you asked which version and unable to repeat on your version).

Only 1 solution work:

echo "write through" > /sys/block/<dev>/queue/write_cache

Then run Calibre and successfully uploaded books.

[DNSB]

Originally Posted by yorikk

Kovidgoyal,

Tried from Kindle menu:

reboot
full reset

Tried few good USB cables.

Just copy files is ok even on 1Gb file.

Then I search for bugs and found few links: kernel bug (no fix), calibre bug (you asked which version and unable to repeat on your version).

Only 1 solution work:

echo "write through" > /sys/block/<dev>/queue/write_cache

Then run Calibre and successfully uploaded books.

Could you add that line to a shell script to start calibre to automate the process?

[yorikk]

No way to do it automatically from non-root user.
Maybe have to place fix to udev rules?

But this is not normal

Seems there is conflict between Linux USB+vfat subsystem VS buggy Kindle firmware.

When Calibre is syncing books, Kindle just turn off USB mode, load main Home screen and disappear from Linux (but Linux show it is still mounted).

I'm sure if Linux kernel developers refuse to fix it for Kindle, must be another way from Calibre side to fix the problem.

Kovidgoyal, if you need any help and/or research for the problem, I can do it.

[yorikk]

It is very easy to repeat Kindle vs Linux bug.

Even without run Calibre:

1. Connect Kindle

2. It is automounted to /mnt/sdb1

3. Run as non-root: "sync /mnt/sdb1"

4. Kindle disconnect and vfat errors coming:

sd 3:0:0:0: [sdb] tag#0 access beyond end of device
blk_update_request: I/O error, dev sdb, sector 6385536 op 0x0: ( READ) flags 0x80700 phys_seg 1 prio class 0

5. sync + "read something" is NOT FIX because it make magic only first time, next time "read something" comes from cache and Kindle disconnect anyway

[yorikk]

Updated my Kindle Voyage firmware:

Previous was 5.5.0
Now is latest 5.13.3

sync => disconnect problem still exist

[yorikk]

Tried fresh kernel:

Linux 5.9.4

On very clean computer without modern bullshit like systemd, udevd, udisks, etc.

Kindle is still disconnect just in few seconds after mount + run "sync /mnt/sdb1"

Seems this is Linux bug in chain Linux + Kindle

I have Windows 10 and MacOS X computers but do not know how to check this issue there.

[yorikk]

Tried to mount Kindle with "msdos" filesystem type:

# mount -t msdos /dev/sdb1 /mnt/sdb1 # success

# ls -1 /mnt/sdb1/

active~1
amazon~1
docume~1
drivei~1.cal
fonts
fseven~1
metada~1.cal
spotli~1
system
trashe~1

(so it's definitely not vfat)

# sync /mnt/sdb1

... few seconds ... KINDLE DISCONNECTED !

[yorikk]

Kovid

I read topic "Bug 203973 - VFAT filesystem causes errors when opened through Calibre and disconnects device" (CLOSED WILL_NOT_FIX)

In real life seems this issue is affecting Kindle only when it is used with Calibre software.

As you see in my research, problem is still active for today. Even on latest Calibre, Linux Kernel and Kindle firmware.

As I read in that topic, problem comes with 4.19.50+ kernel.

Maybe you can detect if (Kernel version >4.19.50 and device is Kindle) => do not use any sync/flush calls ?

Or just disable all fsync() for Kindle under any Linux (because 4.19.50 is old kernel for today) ?

[kovidgoyal]

I cannot reproduce with kernel 5.9.2 + calibre 5.4.2 and any of the various kindles I have lying around. As I said, it is highly unlikely to be this bug, since, after doing an fsync calibre always does a read from the kindle. See the fsync() function in src/calibre/__init__.py

[yorikk]

1. Without fsync changes:

- Update cached metadata: FAILED (disconnected)
- Upload any book: FAILED (disconnected)

2. Commented first os.fsync:

/usr/lib/calibre/calibre/__init__.py (fsync)

Not enough, problem still exist.

Tested few times:

- Update cached metadata: OK
- Upload any book: FAILED (disconnected)

3. Commented another os.fsync:

/usr/lib/calibre/utils/filenames.py (case_preserving_open_file)

Tested few times:

- Update cached metadata: OK
- Upload any book(s): OK

Problem SOLVED by disable both os.fsync calls

PS. seems utime() is not enough and also second os.fsync call make problem when uploading books

[kovidgoyal]

Again, I cannot replicate, and I am not removing fsync() for an issue I cannot replicate. utime works just fine to prevent the kindle from ejecting.

Feel free to comment them out in your local calibre.

[yorikk]

Another research:

1) fsync + utime = FAILED, DISCONNECTED

sync /mnt/sdb1
touch /mnt/sdb1/driveinfo.calibre
# FAILED

2) fsync + utime + sync = OK

sync /mnt/sdb1
touch /mnt/sdb1/driveinfo.calibre
sync
# OK

Kindle disconnects in ~2 seconds after fsync. Calling touch (like utime) making FS change but OS cache not write it in 2 seconds. If utime() is synced immediately (by Linux "sync" command), everthing is OK.

[yorikk]

Finally, found way to keep fsync and solve the problem:

1) /usr/lib/calibre/calibre/__init__.py

just added after os.utime:

os.system("sync")

2) /usr/lib/calibre/calibre/utils/filenames.py (yeah, there is os.fsync but no utime...)

added after os.fsync (but need more pretty code):

os.utime(os.path.join(cpath, fname), None)
os.system("sync")

My Kindle + Calibre is alive again !