Calibre get stuck syncing

loviedovie · 06-19-2016, 11:39 AM

Hi

I am having a major issue with Calibre and CC and I am not sure where the probvlem comes from exactly.

If my wireless device looses connection in the middle of a sync, Calibre gets stuck syncing mode (I can see in the jobs). And it never times out or disconnecs or cancels the job even though no device is connected since the device has lot its connection.

The main problem with this issue is that I cant get my CC to reconnect again becauuse Calibre thinks a device has already been connected for a while.

I am using the 2.58 version on Linux 64 Debian testung.

thanks

chaley · 06-19-2016, 11:59 AM

Strange. The calibre wireless device driver does time out if the connection really disappears. It sends a "HELLO" every 10 seconds that something must receive or the connection shuts down. My guess is that the connection is still open -- that something between CC and calibre is keeping the network connection alive. It also could be that the network timeout defaults on your system are *very long*. How long have you waited for calibre to detect that CC has disappeared?

In any event, you can force calibre to disconnect. Click on the Device button and choose "Eject this device".

You can also reboot your device which should remove any trace of the connection, should the version of Android on your device be doing something strange with networking. If this doesn't cause calibre to disconnect then something is very broken on your calibre machine. A calibre debug log might give some clues.

There is also the question about why CC is losing the connection. That implies that the WiFi you are using isn't reliable.

loviedovie · 06-19-2016, 12:55 PM

@chaley

The disconnect trick does not work because Calibre complains that there are on going jobs so it refuses to disconnect. The only solution I found is to restart Calibre.

As far as the timing out goes, I do not think that Calibre figured out that the device disconnected after sometime. Maybe I did not wait long enough. I am sure that I waited at least 10 mins when it happened.

This happened to at least 6 times inm last 2 days when I was trying to sync large number of books ( which were alredy in the folder).

I do not think thatr the issue is CC, it must be Calibre getting fooled here. Because if the wireless connection is not available on the device there is not much CC can do which is what generally happened to me.

Basically the device looses the connection and Calibre never ejects the device after loosing it. It sounds like Calibre does not think that the device has been disconnected even though the device lost its wifi.

chaley · 06-19-2016, 02:53 PM

Quote:

Originally Posted by loviedovie

@chaley

The disconnect trick does not work because Calibre complains that there are on going jobs so it refuses to disconnect. The only solution I found is to restart Calibre.

From what you said in your thread in the CC forum I suspect that the job is calibre doing the analysis of the book files CC discovered in its library directory.

loviedovie · 06-26-2016, 12:10 AM

After doing couple more tests, I believe that Calibre never terminates the device if the device connection drops abrubtly which results in refused connections from any device.

I think that Calibre should terminate jobs and connections if it cant engage in meaningful andsuccessful handshakes with the device after certain time periods, say half an hour to one hour. If within this time frame none of the connected devices responds back properly, Calibre should just terminate the device connection (and jobs related) because it is hard to restart Calibre on the go on a cell phone or an eink device. Or maybe CC can have an option to for a reconnect regardless of the connected device.

Basically my connection was disconnected while syching when I was out. 6 hours later I wanted to connect back with CC and guess what Calibre still was thinking that there was already a device connected and was refusing my wireless device connection attempt. So I had to restart Calibre. Thankfully I was close to my desktop this time.

chaley · 06-26-2016, 05:32 AM

I have run the following experiments to check if CC and calibre disconnect properly.

In order to avoid any "normal" shutdown processing, in the second through fourth experiments I "disconnected" by removing my WiFi hub from the network. This process ensures that neither CC nor calibre are aware that the network has been cut until something times out.

I include the relevant portions of the calibre debug log for each experiment.

Experiment 1: the baseline -- connect and disconnect normally. As expected, it works; calibre disconnects normally.

Spoiler:

Experiment 2: disconnect during the startup handshake. Calibre disconnects normally. In some cases CC can wait for some hours (the general network keepalive timeout) showing a dialog box waiting for you to tap "Cancel", but this is not the behavior you are describing.

Spoiler:

Code:

Job: 17 Get device information started
SMART_DEV ( 762.65:  0.002) get_device_information
SMART_DEV ( 762.68:  0.024) _call_client GET_DEVICE_INFORMATION wfr True
SMART_DEV ( 781.60: 18.904) _write_metadata_cache
SMART_DEV ( 781.68:  0.061) _write_metadata_cache wrote 6 entries, purged 0 entries
SMART_DEV ( 781.70:  0.002) _receive_from_client device went away
SMART_DEV ( 781.71:  0.000) _call_client other exception
Traceback (most recent call last):
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 613, in _call_client
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 649, in _receive_from_client
ControlError: Device closed the network connection
DeviceJob: 17 Get device information done, calling callback
Device closed the network connection

Traceback (most recent call last):
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\gui2\device.py", line 87, in run
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\gui2\device.py", line 463, in _get_device_information
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 1193, in get_device_information
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 613, in _call_client
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 649, in _receive_from_client
ControlError: Device closed the network connection

DeviceJob: 17 Get device information callback returned
SMART_DEV ( 781.78:  0.059) post_yank_cleanup
READING LIST: Device disconnected
Job: 17 Get device information finished
Device closed the network connection

        Traceback (most recent call last):
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\gui2\device.py", line 87, in run
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\gui2\device.py", line 463, in _get_device_information
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 1193, in get_device_information
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 613, in _call_client
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 649, in _receive_from_client
        ControlError: Device closed the network connection

Experiment 3: disconnect after CC and calibre finish all startup processing. CC is waiting for commands from calibre. Calibre notices the disconnect after some 20 seconds and disconnects. CC may or may not notice at approximately the same time. If CC does notice then it shows a dialog. If it does not notice then it waits until you manually disconnect (CC's menu). As mentioned above CC should time out in a few hours but I didn't wait that long.

Spoiler:

Experiment 4: disconnect while calibre and CC are negotiating about syncing books. Calibre noticed and disconnected. Usually the disconnect happened in seconds, but during two runs of the experiment it took all of the 300 seconds (5 minutes) allowed by the calibre wireless driver.

However, there is a problem on the CC side. CC does not notice for hours (the network "keepalive timeout") and there is no way to cancel the process other than to force-stop CC. A similar case was reported in the past but I didn't find the cause. Now I have it and the fix will be in the next release. In any event, this isn't the problem you are reporting because it is obvious that CC thinks it is still connected and waiting for books to be transferred.

Spoiler:

Code:

SMART_DEV (2627.01:  0.216) _create_upload_path lengths mobi 189 21 87
SMART_DEV (2627.03:  0.000) upload_books lpath xxx.mobi
SMART_DEV (2627.06:  0.011) _json_encode series sort =  1632
SMART_DEV (2646.33: 19.256) _send_byte_string socket error [Errno 10054] An existing connection was forcibly closed by t
he remote host 10054
SMART_DEV (2646.36:  0.000) _write_metadata_cache
SMART_DEV (2646.43:  0.046) _write_metadata_cache wrote 6 entries, purged 0 entries
DeviceJob: 38 Upload 4 books to device:1633, xxx done,
 calling callback
[Errno 10054] An existing connection was forcibly closed by the remote host

Traceback (most recent call last):
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\gui2\device.py", line 87, in run
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\gui2\device.py", line 584, in _upload_books
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 1452, in upload_books
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 689, in _put_file
  File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 575, in _send_byte_string
error: [Errno 10054] An existing connection was forcibly closed by the remote host

DeviceJob: 38 Upload 4 books to device:1633, Book with lower case author, xxx callback returned
SMART_DEV (2646.50:  0.050) post_yank_cleanup
READING LIST: Device disconnected
Job: 38 Upload 4 books to device:1633, xxx finished
[Errno 10054] An existing connection was forcibly closed by the remote host
        Traceback (most recent call last):
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\gui2\device.py", line 87, in run
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\gui2\device.py", line 584, in _upload_books
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 1452, in upload_books
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 689, in _put_file
          File "D:\CBH_Data\calibre.git\calibre_dev\src\calibre\devices\smart_device_app\driver.py", line 575, in _send_byte_string
        error: [Errno 10054] An existing connection was forcibly closed by the remote host

Experiment 5: Force-close CC while leaving the WiFi connected. This checks that Android correctly closes the network sockets. Calibre noticed the disconnect very quickly.

Spoiler:

As you can see, none of these experiments repeat the behavior you report. I don't deny that you are seeing the behavior you describe but I can't fix anything until I can reliably repeat it.

You might try producing a calibre debug log, which might give some clue(s) about what it happening.

loviedovie · 06-27-2016, 12:32 AM

Chaley

Thanks for extensive debugging.

I wonder if I am having a platform related improper disconnects here.

I am using Calibre on Debian 64bit Testing.

chaley · 06-27-2016, 02:55 AM

Quote:

Originally Posted by loviedovie

Chaley

Thanks for extensive debugging.

I wonder if I am having a platform related improper disconnects here.

I am using Calibre on Debian 64bit Testing.

Are you using the standard calibre install or a distro package? If it is the distro package, try using the standard one. Distro versions of calibre are often broken in strange ways.

Run calibre in debug mode until you capture one of these events. As I said earlier, we might be able to get some clues from a log.

loviedovie · 06-29-2016, 10:19 PM

@chaley

Can you try this which involves ssh-ssh terminal(connectbot)-ssh server to be able to forward ports.

Basically I forward 9798 to localhost on my tablet so that I can connect to Calibre as a wireless device from anywwhere in the world securely.

Lets say Calibre runs on 10.6.0.150

I forward my port as -L9798:10.6.0.150:9798 and I pit localhost:9798 in CC. Once the connection is established on ssh I then connect as wireless device. This works great for me. But I am wondering if Calibre device being stuck has anything to do with this set up, for instance when ssh co0nnection drops in the middle due to wireless issues.

thanks

chaley · 06-30-2016, 06:01 AM

Quote:

Originally Posted by loviedovie

@chaley
Can you try this which involves ssh-ssh terminal(connectbot)-ssh server to be able to forward ports.

Sorry, but I don't have the time or interest to debug exotic network setups.

It is probable that the tunnel isn't being torn down when CC is disconnected. Calibre is sending packets to CC when CC isn't there. Some SW managing the tunnel should notice that the packets calibre is sending are going down a rat hole and shut down the tunnel. I don't know how you can fix that in your setup. Manually shutting the tunnel down on the device might do the job.

06-19-2016, 11:39 AM	#1
loviedovie Addict Posts: 295 Karma: 2139988 Join Date: Nov 2014 Device: bookeen	Calibre gets stuck syncing Hi I am having a major issue with Calibre and CC and I am not sure where the probvlem comes from exactly. If my wireless device looses connection in the middle of a sync, Calibre gets stuck syncing mode (I can see in the jobs). And it never times out or disconnecs or cancels the job even though no device is connected since the device has lot its connection. The main problem with this issue is that I cant get my CC to reconnect again becauuse Calibre thinks a device has already been connected for a while. I am using the 2.58 version on Linux 64 Debian testung. thanks Last edited by loviedovie; 06-19-2016 at 11:51 AM.

06-26-2016, 12:10 AM	#5
loviedovie Addict Posts: 295 Karma: 2139988 Join Date: Nov 2014 Device: bookeen	After doing couple more tests, I believe that Calibre never terminates the device if the device connection drops abrubtly which results in refused connections from any device. I think that Calibre should terminate jobs and connections if it cant engage in meaningful andsuccessful handshakes with the device after certain time periods, say half an hour to one hour. If within this time frame none of the connected devices responds back properly, Calibre should just terminate the device connection (and jobs related) because it is hard to restart Calibre on the go on a cell phone or an eink device. Or maybe CC can have an option to for a reconnect regardless of the connected device. Basically my connection was disconnected while syching when I was out. 6 hours later I wanted to connect back with CC and guess what Calibre still was thinking that there was already a device connected and was refusing my wireless device connection attempt. So I had to restart Calibre. Thankfully I was close to my desktop this time. Last edited by loviedovie; 06-26-2016 at 12:14 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Synching between Calibre 1.19 and Kindle Touch	booke02	Amazon Kindle	0	01-12-2014 04:48 PM
Synching bookmarks between Calibre and Kindle Paperwhite	lucitedragon	Library Management	1	05-28-2013 11:40 PM
Calibre, iPad & itunes not synching correctly	Oscar Berke	Devices	1	05-29-2011 05:56 PM
Stuck with Calibre 0.7.34 from now on?	Lupa	Calibre	18	01-08-2011 08:42 AM
Synching wordplayer with Calibre using content server	djstorman	Calibre	0	11-11-2009 07:18 PM

06-19-2016, 11:59 AM	#2
chaley Grand Sorcerer Posts: 12,339 Karma: 8012652 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	Strange. The calibre wireless device driver does time out if the connection really disappears. It sends a "HELLO" every 10 seconds that something must receive or the connection shuts down. My guess is that the connection is still open -- that something between CC and calibre is keeping the network connection alive. It also could be that the network timeout defaults on your system are very long. How long have you waited for calibre to detect that CC has disappeared? In any event, you can force calibre to disconnect. Click on the Device button and choose "Eject this device". You can also reboot your device which should remove any trace of the connection, should the version of Android on your device be doing something strange with networking. If this doesn't cause calibre to disconnect then something is very broken on your calibre machine. A calibre debug log might give some clues. There is also the question about why CC is losing the connection. That implies that the WiFi you are using isn't reliable.

06-19-2016, 12:55 PM	#3
loviedovie Addict Posts: 295 Karma: 2139988 Join Date: Nov 2014 Device: bookeen	@chaley The disconnect trick does not work because Calibre complains that there are on going jobs so it refuses to disconnect. The only solution I found is to restart Calibre. As far as the timing out goes, I do not think that Calibre figured out that the device disconnected after sometime. Maybe I did not wait long enough. I am sure that I waited at least 10 mins when it happened. This happened to at least 6 times inm last 2 days when I was trying to sync large number of books ( which were alredy in the folder). I do not think thatr the issue is CC, it must be Calibre getting fooled here. Because if the wireless connection is not available on the device there is not much CC can do which is what generally happened to me. Basically the device looses the connection and Calibre never ejects the device after loosing it. It sounds like Calibre does not think that the device has been disconnected even though the device lost its wifi.

06-27-2016, 12:32 AM	#7
loviedovie Addict Posts: 295 Karma: 2139988 Join Date: Nov 2014 Device: bookeen	Chaley Thanks for extensive debugging. I wonder if I am having a platform related improper disconnects here. I am using Calibre on Debian 64bit Testing.

06-29-2016, 10:19 PM	#9
loviedovie Addict Posts: 295 Karma: 2139988 Join Date: Nov 2014 Device: bookeen	@chaley Can you try this which involves ssh-ssh terminal(connectbot)-ssh server to be able to forward ports. Basically I forward 9798 to localhost on my tablet so that I can connect to Calibre as a wireless device from anywwhere in the world securely. Lets say Calibre runs on 10.6.0.150 I forward my port as -L9798:10.6.0.150:9798 and I pit localhost:9798 in CC. Once the connection is established on ssh I then connect as wireless device. This works great for me. But I am wondering if Calibre device being stuck has anything to do with this set up, for instance when ssh co0nnection drops in the middle due to wireless issues. thanks