calibre+kindle+linux = fsync disconnect problem still

[kovidgoyal]

Calling sync like that is a potentially *very* slow operation, especially if network filesystems are mounted. Why is your kernel taking two seconds to do writes? Fixing that seems like a much better way to go.

[yorikk]

Please add utime() for other os.fsync (caused problem when books uploaded)

About slow sync..... Yes this may be a problem but:

os.system("sync &") will do it in background w/o delays

I'll try to find where is cache write delay is configured.

But I'm sure there is default settings...

[kovidgoyal]

The other fsync's only code path is from put_file() in usbms/cli.py immeditely after doing it, the file is written to, so there is no need for an extra utime()

[yorikk]

Found another way to fix the problem:

Mount vfat have "flush" option (but it's not enabled by default):

flush If set, the filesystem will try to flush to disk more early than normal. Not set by default.

When Kindle is mounted with this option, nothing is need to be changed in Calibre code. Cache update, books uploading is OK. But "sync /mnt/sdb1" still disconnect Kindle even with "flush" mount option.

Anyway, on my Linux with default settings, there is 100% require both os.fsync() to be followed by utime and "sync" system command in Calibre.

Kovid, please check your vfat mount options when Kindle is connected, for "flush" option (grep fat /proc/mounts).

Maybe without this option you can easy repeat the problem.

[kovidgoyal]

Yes, there is flush, and that is a default mount setting chosen by Udisks2 for removable drives. How are you mounting your kindle?

[yorikk]

My Kindle and other usb-flash devices is mounted by udev rule which is calling mount with default options (no flush).

Flush option making most of old usb-flash drives is extermly slow, so I prefer faster defaults.

Kovid, if you can do something to fix Calibre with default mount, please please please do it.

Both os.fsync() make "Linux-vs-Kindle-problems".

I hope now you can repeat it on your system.

[PeterT]

Why not have a custom udev entry for your Kindle that has the optins you need just for that device?

[yorikk]

Originally Posted by PeterT

Why not have a custom udev entry for your Kindle that has the optins you need just for that device?

It's possible (already done) but who know how many Kindle users mount it with default options, leading to this very unusual bug ?

Linux && Kindle && os.system("sync &")

or Linux && Kindle && !/flush/ in mount options && os.system("sync &")

[PeterT]

But you're the only Kindle user reporting this issue......

[yorikk]

Linux, Kindle, Calibre: I/O error vfat filesystem disconnect problem.

Any fresh Linux system without udisks2 may be affected by this bug.

Solutions (any of 5 is ok):

1. Disable fsync for Kindle block device (sdb, etc):

echo "write through" > /sys/block/DEVICE/queue/write_cache

2. Remount Kindle with "flush" option:

mount -o remount,flush DEVICE MOUNTPOINT

3. Use udisks2 as automatic external device mount service (it use "flush" option by default)

4. Patch Calibre source files, comment 2 calls to os.fsync:

/usr/lib/calibre/calibre/__init__.py
/usr/lib/calibre/calibre/utils/filenames.py

5. Patch Calibre source files, add os.system("sync &") :

/usr/lib/calibre/calibre/__init__.py:

after:
os.utime(fileobj.name, None)

add:
os.system("sync &")

/usr/lib/calibre/calibre/utils/filenames.py:

after:
os.fsync(ans.fileno())

add:
os.utime(os.path.join(cpath, fname), None)
os.system("sync &")

PS:

Links to other topics related:

https://bugzilla.kernel.org/show_bug.cgi?id=203973

https://bugs.launchpad.net/calibre/+bug/1834641

[kovidgoyal]

Not using flush for removable storage devices is an *extremely* bad idea. It means it becomes much for likely there is data corruption if the device is accidentally removed, connection interrupted. I'm afraid I am not going to add special support for this to calibre. This bug is anyway a bug in the Kindle firmware/linux kernel, not calibre. Adding a utime() call after fsync is the extent of the workaround I am willing to implement for itin calibre.

[Debuk]

Originally Posted by PeterT

But you're the only Kindle user reporting this issue......

Joined here to say I am affected by this, too. Or by something somewhat similar, at least.

I was lucky to find this thread easily, since having to disconnect and reconnect the device for every book copied is not a proper experience. Thsnks yorikk for your analysis. At least I know there is something quite broken with Kindle firmware.

I may be facing something different, though.

udisk2 was already being used, but it isn't adding a flush option, it gets mounted with:

rw,nosuid,nodev,noexec,relatime,sync,uid=<uid>,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro,uhelper=udisks2

The calibre version (3.39.1, from Debian) is prior to the utime fix, but the kernel version is also prior to 4.19.50, so this system [em]shouldn't[/em] be affected. And in fact, the one-liner

python -c "p = '/run/media/kovid/Kindle/driveinfo.calibre'; f = open(p, 'r+b'); import os; os.fsync(f.fileno());"

doesn't disconnect it (NB: "import os" is missing in the code), yet something is disconnecting it after every calibre book transfer. Maybe not fsync related, after all, since commenting the two fsync() calls (plus restarting calibre, obviously) didn't help.