Clara HD Syncing Problems

[roguexuk]

Hi all,

I'm hoping to pick your brains, and sorry if this is in the wrong forum, it was a toss up between here & the Calibre one.

I treated myself to a Kobo Clara HD after Christmas. It seems to work great straight out of the box.

But once I started to sideload my books on through Calibre, the wifi sync just seems to stop - either hangs on 'Starting Sync' or 'Checking for Updates', and the connection keeps dropping.

I tried plugging it in & syncing with Kobo desktop, but the same thing happens. I've tried re-starting the router.

As I said, I have Calibre, I also have Kobo Utilities & the Kobo Touch Extended Plug-in. (I have it set to create 2 collection columns: tags & #myshelves)

I've been trawling through threads here & on Reddit, but haven't had much luck in fixing it.

Repairing Account doesn't make any difference. The Signing out of Account never seems to work - the screen just says signing out for a long time, in the end I give up & re-start it. The same goes with Factory Reset.

I've tried the trick of starting the sync & then tapping on the Wifi button, to force the connection to stay open, but that only worked once (took a couple of hours), but after that, it seemed to make the reader re-start.

The only thing that seems to work is the Manual Reset, which of course, means that I can only have the books I bought from kobo on the device. But the wifi sync is like lightning this way.
I can't tell you how many times I've manually reset this thing, and I only bought it 3 weeks ago!

Strangely enough, I also synced up my Aura (original), and at first it had the same problem after using Calibre, so I manually re-set it, connected it up to Calibre & it works fine!

I'm trying to figure out if there's some setting that I've checked for the Clara that's screwing things up, or if my database is screwed up (although according to Kobo Utilities the database is ok).

What do you think?

Thanks

EDIT: Ooh I forgot to say that although the sync hangs, it does seem to get as far as syncing book progress, l could still browse the store, but it wouldn't download any books I'd bought.

[mathil]

I've always had the same problem with all of my Kobos, but syncing through Kobo Desktop always kept working, thankfully. Shortly after I reset it and reload all of my sideloaded books, syncing takes forever and then fails, the connection keeps dropping etc.

Hope somebody can help you.

[PeterT]

How many books are you sending at a time? Remember that Kobo Touch Extended does a conversion to kepub on the fly; I'm not sure but it might well convert all the books before sending any.

[roguexuk]

Originally Posted by PeterT

How many books are you sending at a time? Remember that Kobo Touch Extended does a conversion to kepub on the fly; I'm not sure but it might well convert all the books before sending any.

Hmm, with new ereaders, I usually send over the contents of a couple of Calibre tags at a time, some have hundreds of epubs in them.

Should I move the contents of a tag then disconnect, letting the Kobo do it a bit a time?
Or make sure I already have a kepub version of all my files? Which shouldn't be hard to do as I always have AZW & epub version of all my books so I can use either my Kobo or Kindle.

[compurandom]

I've also had syncing problems. I recently did a soft factory reset from the menu and I've only copied a fraction of my books back onto the device, and now sync is working and fast.

I've been adding handfuls of books a day (weeding out ones I don't really need on the device all the time), and so far the problem hasn't come back.

I'm wondering if it is just dumping a huge number of books on it all at once that makes sync fail, or if there's one specific book (which I haven't hit yet) that causes it, or if it is the total number of books (or shelves) that causes it.

In this message @davidfor suggested conflicting case in shelf names causes it, but his solution didn't completely work for me, but I haven't been able to verify why.

[roguexuk]

Originally Posted by compurandom

I've also had syncing problems. I recently did a soft factory reset from the menu and I've only copied a fraction of my books back onto the device, but sync is working and fast.

I've been adding handfuls of books a day (weeding out ones I don't really need on the device all the time), and so far the problem hasn't come back.

I'm wondering if it is just dumping a huge number of books on it all at once that makes sync fail, or if there's one specific book (which I haven't hit yet) that causes it, or if it is the number of books that causes it.

In this message @davidfor suggested mixed case in shelf names causes it, but his solution didn't completely work for me, but I haven't been able to verify why.

It's worth a try, I admit, the most recent time I synced with Calibre, I did only a couple at a time & the sync worked. So I thought 'Great, I'll load 'em all back on then'. Then the sync didn't work.

I did try @davidfor's solution, and it did turn out that I had #myshelves on my main PC calibre & #MyShelves on my portable calibre on the old laptop, so I was hopeful it might work, but no joy.

I think after my Clara is done repairing its account (it's been doing it for 3+ hours :-( & it still doesn't work, I'll try the Manual re-set again & load the books on a bit at a time see if it makes a difference.

Thank goodness. 4 posts in & you guys are already coming up with things I hadn't thought about. Thank goodness for mobileread :-)

[davidfor]

Originally Posted by roguexuk

It's worth a try, I admit, the most recent time I synced with Calibre, I did only a couple at a time & the sync worked. So I thought 'Great, I'll load 'em all back on then'. Then the sync didn't work.

I did try @davidfor's solution, and it did turn out that I had #myshelves on my main PC calibre & #MyShelves on my portable calibre on the old laptop, so I was hopeful it might work, but no joy.

That isn't the problem I described, but, could cause issues.

The issue was that I changed the case of the collections. I use a column called #status for collections. This had values like "toRead" and "reading". I changed these to "ToRead" and "Reading". When I connected the device after this change, calibre removed the collections "toRead" and "reading" and added collections "ToRead" and "Reading". What it did was case sensitive. But, when the device synced, it did it case insensitive. As far as it was concerned "toRead" and "ToRead" where the same and there was an error when it tried to do the update. It's possible that some actions in the sync were case sensitive and some case insensitive and that cause the issue. Add in that I have multiple devices connecting to calibre at different times and syncing at different times and that I had missed one place where the case had not been changed, and it was a mess.

There is probably an error in how calibre handles this. And there might be another based on a comment that someone else has made. The latter might mean that Kobo changed something and I hadn't noticed it.

I am still having issues syncing the Clara HD. It generally works, but, seems to take to long and does stop sometimes. The other devices don't seem to have a problem. The main difference I can think of is that the Clara HD has a lot of sideloaded books. I tend to read sideloaded epubs on the Clara HD and purchased or sideloaded kepubs on the Libra H2O. That came about accidentally because while testing something on the Libra H2O, I removed all the sideloaded books from it and have never gotten around to put the epubs back on.

I haven't had time to debug the issue with the Clara HD properly. I may just bite the bullet and do a factory reset and not load the read books back on it.

[compurandom]

Originally Posted by davidfor

That isn't the problem I described, but, could cause issues.

My apologies for describing the problem sloppily.

In my case, the shelves comes from three columns in calibre with both intentional and unintentional overlaps, and the source case could be different on different books.

Digging into my configuration, I think some is being merged from plugboard and some is News items that seem to get it from tags. (Do those not use plugboard??)

I'm still thinking if I keep transfering a handful of books at a time, eventually I'll hit the trigger.

[davidfor]

Firstly, the plugboards are not involved for collection management. They change the titles and other metadata, but, the collection management uses the data from the columns unchanged.

If you the values used for collections have different case version of the same value, then you will be causing problems. As stated, calibre does it case insensitive. At the end of the connection, you will have whichever was last of the values. And the syncing will then have issues dependent on what was the case the last time you synced.

[compurandom]

Originally Posted by davidfor

Firstly, the plugboards are not involved for collection management.

Right. I remember now, it's a field in the driver config, and I'm using a composite column. I suppose I should delete it from the plugboard so I'm not confused in the future.

I'm also tempted to modify the composite column to squash case to fix this problem permanently. Unfortunately, that might make some tags ugly.

Or add a tool to search for mixed case so it can be fixed. I've fixed it in tags manager a few times, but when there's multiple columns involved, it's messier. (Edit: I was fixing case within a hierarchical tag, not sure if tags manager would see it on the same tag?)

Another possible solution would be to trick the kobo into not syncing side loaded shelves at all. (Driver option?) I tried doing this by changing the sync column in the database before disconnecting, but it only sometimes helped. Maybe it was too late by then and I already had the mixed case on the device half synced.

[davidfor]

Originally Posted by compurandom

Right. I remember now, it's a field in the driver config, and I'm using a composite column. I suppose I should delete it from the plugboard so I'm not confused in the future.

I'm also tempted to modify the composite column to squash case to fix this problem permanently. Unfortunately, that might make some tags ugly.

You could use another column to do the case squashing. One is used in calibre for display and the other for collections. There is a performance hit, but, if you don't display the column anywhere, it should only happen when the collection management is being done.

Or add a tool to search for mixed case so it can be fixed. I've fixed it in tags manager a few times, but when there's multiple columns involved, it's messier. (Edit: I was fixing case within a hierarchical tag, not sure if tags manager would see it on the same tag?)

It's one of the things I don't like about the tag management in calibre. You can accidentally change the tag case while adding a tag to a book and the books with that tag are shown as modified. It is to easy to do.

Another possible solution would be to trick the kobo into not syncing side loaded shelves at all. (Driver option?) I tried doing this by changing the sync column in the database before disconnecting, but it only sometimes helped. Maybe it was too late by then and I already had the mixed case on the device half synced.

That won't work. That basically means that the collection hasn't been synced yet. It should only be false for a new collection until the next sync. The _SyncTime should get updated each time a sync happens. Though it might only happen when something is changed. The trick might be to set the sync time into the future. Depending on how the checks are done, it might skip any that are in the future. But, there is also risk in that if Kobo were to store that timestamp on their server. It might mean no syncing at all of collections until that time.

[compurandom]

Originally Posted by davidfor

You could use another column to do the case squashing.

No point in that. The composite column is only there to merge tags between the three columns. I only display it in calibre to verify what is going into the kobo.

I was actually concerned about the tags being ugly on the kobo.
Or worse, changing case mid stream when half the books are already loaded.

(...modifying the database to not sync shelves...)

That won't work. That basically means that the collection hasn't been synced yet. It should only be false for a new collection until the next sync. The _SyncTime should get updated each time a sync happens.
[...]
But, there is also risk in that if Kobo were to store that timestamp on their server. It might mean no syncing at all of collections until that time.

Since 90% of my books are side loaded, and even non-sideloaded books get at least a stub book with no format in calibre, I don't rely on kobo's shelf sync. I'd be ok with it being disabled if other sync features (like automatic initial downloading of DRM books) were not broken.

I actually don't particularly care for the idea of the shelves for all my side loaded books being uploaded to kobo's servers, and they probably don't need the load either.

[roguexuk]

Well, I've had a go at following @compurandom's idea of only moving a few books/files at a time.

Well... so far so good.
I'm only moving over about 40 epubs at a time, and have been wi-fi syncing after each to make sure it's still working

Of course, I moved a lot of epubs (in stages) over a couple of hours , so I ended up with a corrupted database
But logging in and out of my Kobo account sorted that.

for all your help.