Syncing takes forever

[thepossum]

I have a Kobo Glo that I have to sync every day, because I have a newspaper subscription. This has become an aggravating daily experience, because it takes much too long.

Once the Glo connects to the wifi, the newspaper downloads in mere seconds, and then sits there spinning away on the "Loading Content" screen for approximately five or six minutes. That is a very long and very unnecessary amount of time, forcing me to recharge the device more often then otherwise necessary. Eventually this process finishes and I get a new tile on my homescreen showing me today's "Recommended" books.

Now, I'm more technically inclined than most folk, so in order to get to the bottom of this I set up my laptop to be my Kobo's wifi access point, with squid running as a web proxy. This would let me see all http requests my Glo makes when it tries to sync, and this is what I found:

On initial connection to the wifi, it downloaded http://www.msftncsi.com/ncsi.txt

Then it did this ***360*** times!

POST http://mobile.kobobooks.com/#####/mobileRequest.ashx

where the '#####' was a five-digit number, possibly linked to my account. The 'POST' means that it is *uploading* something to the Kobo servers, not downloading information.

Once that was done, it pulled something from this page 7 times:

http://ecimages.kobobooks.com/Image.ashx?

and finally a 'HEAD' request to http://download.kobobooks.com/ereade...s/dicthtml.zip

There may have been more data being passed, as the proxy server only handles requests on http port 80, not https port 443, or other internet protocols.

What I really want to know is what the heck my Kobo thinks is so important that it needs to make 360 separate uploads during a sync that only needs to download one single file each day.

I have a lot of epub books on my Kobo's internal storage, mostly free downloads I got from Baen. I have all my marketing stuff disabled, my Kobo is not linked to my Facebook account, I have not activated my Adobe, I told my Kobo not to use any of the reading stats.

So I find that having my Glo sending Megabytes of data to Kobo to be nothing short of creepy. And wasteful.

[GeoffR]

The loading content part of sync used to take a few minutes on my Glo, having gradually increased over time (I havent done a factory reset for a long time).

It turned out there were more than 1000 old recommendations in the Glo's database, and after using Calibre to delete them the loading content sync time dropped to a few seconds.

[Willa]

Originally Posted by GeoffR

It turned out there were more than 1000 old recommendations in the Glo's database, and after using Calibre to delete them the loading content sync time dropped to a few seconds.

How did you do it with calibre . . would be really interested to know

[GeoffR]

Originally Posted by Willa

How did you do it with calibre . . would be really interested to know

In Calibre, go to: Preferences --> Plugins --> Device interface plugins --> KoboTouch --> Customise plugin

and tick the "Show Recommendations" box. Then connect the device and click the "Device" button to get the view of books on the device.

If you scroll down you will see the ones marked as recommendation in the "Collections" column. You can select and delete them.

There is a quirk in Calibre: if a recommendation has the same title/author as a book on the device then it doesn't show it as a recommendation, but instead shows as a second copy of the book. You can tell which copy is the recommendation by looking in the size column, the recommendation will have size zero (or close to it). You can delete those ones too.

[Willa]

Originally Posted by GeoffR

In Calibre, go to: Preferences --> Plugins --> Device interface plugins --> KoboTouch --> Customise plugin

and tick the "Show Recommendations" box. Then connect the device and click the "Device" button to get the view of books on the device.

If you scroll down you will see the ones marked as recommendation in the "Collections" column. You can select and delete them.

There is a quirk in Calibre: if a recommendation has the same title/author as a book on the device then it doesn't show it as a recommendation, but instead shows as a second copy of the book. You can tell which copy is the recommendation by looking in the size column, the recommendation will have size zero (or close to it). You can delete those ones too.

#


Oooh thank you so much Ratty . . er Geoff

[Willa]

Good grief - 521 of the little buggers ! many thanks !

[davidfor]

Originally Posted by GeoffR

I
There is a quirk in Calibre: if a recommendation has the same title/author as a book on the device then it doesn't show it as a recommendation, but instead shows as a second copy of the book. You can tell which copy is the recommendation by looking in the size column, the recommendation will have size zero (or close to it). You can delete those ones too.

I don't think that should happen. I'll have a look at it. What type of book is the real one? Sideloaded or from Kobo?

[GeoffR]

Originally Posted by davidfor

I don't think that should happen. I'll have a look at it. What type of book is the real one? Sideloaded or from Kobo?

It happens with both. In some cases I have a sideloaded epub bought elsewhere and Kobo recommends the same book. In other cases I have downloaded a kepub from Kobo, and Kobo recommends a different edition of the book, but with same title and author of course.

The device view shows two copies of the book, one with 0.0 in the "Size (MB)" column, and nothing in the "Collections" column. The library view shows "Main (2 books)" in the "On Device" column.

[davidfor]

Originally Posted by GeoffR

It happens with both. In some cases I have a sideloaded epub bought elsewhere and Kobo recommends the same book. In other cases I have downloaded a kepub from Kobo, and Kobo recommends a different edition of the book, but with same title and author of course.

The device view shows two copies of the book, one with 0.0 in the "Size (MB)" column, and nothing in the "Collections" column. The library view shows "Main (2 books)" in the "On Device" column.

Just did a quick test here, and I'm not getting the same results. The collections column has "Recommendations" as expected. Can I get a copy of your database to have a look at?

[Uschiekid]

Originally Posted by Willa

Good grief - 521 of the little buggers ! many thanks !

You think that's bad, I had 1500!!!


@GeoffR--My hero!

@davidfor, I think I found the location of all those phantom images....sorry, didn't realise the basic plugin had that option. nothing to do with the KoboUtilities erase images not working.
Down to a reasonable and manageable 500 images (from 5000). Takes 2 seconds to load when connected to the computer and easy to look through.
If someone else comes to you with false claims that not all unneeded images are deleted, you'll know where to point them



and now to try synching and see if the insane content loading time goes down!

UPDATE: It seems to speed up the process, although I still seem to have the issue that it will occasionally get stuck in the sync process and I have to cancel and restart to get it to really sync.

Off-topic: I hate how synching unpredictably will sometimes replace my custom dictionaries. If it were predictable it wouldn't bug me so much, but I never know if after a sync, I'll have to go back and add the custom ones back or not. I think it used to only do it when there was a firmware upgrade, but now it just seems to do it when it feels like it

[thepossum]

Re: Off-topic: unpredictable dictionary replacements

The 'HEAD' request I'd seen above just tells me that during the sync, the Kobo is probably just looking at datestamps to see if your current file is older than the one being provided by Kobo. If you were somehow able to fudge the datestamp on your sideloaded dictionary, perhaps it will never be replaced?

I know that with Linux you can use a command like:

touch --date="Jan 1 2100" filename

to set the file's datestamp well into the future. There are probably multiple ways of doing the same thing under Windows, Google is your friend ;-)

I think this calls for either a bit of experimentation or a forums search here for people who've already run into this problem here on mobileread?

[davidfor]

Originally Posted by thepossum

Re: Off-topic: unpredictable dictionary replacements

The 'HEAD' request I'd seen above just tells me that during the sync, the Kobo is probably just looking at datestamps to see if your current file is older than the one being provided by Kobo. If you were somehow able to fudge the datestamp on your sideloaded dictionary, perhaps it will never be replaced?

I know that with Linux you can use a command like:

touch --date="Jan 1 2100" filename

to set the file's datestamp well into the future. There are probably multiple ways of doing the same thing under Windows, Google is your friend ;-)

I think this calls for either a bit of experimentation or a forums search here for people who've already run into this problem here on mobileread?

That might work, but based on other things the firmware does, it is probably checking the date in the database. Changing the "LastModified" timestamp in the "Dictionary" table is probably the way to go.

[Uschiekid]

Ok, thanks thepossum and davidfor, I'll give that a try when I have time. I use ubuntu.

Other people have commented a little, but I don't remember any suggestions, so I'll try yours

[thepossum]

Bad news for me -- I used the current version of Calibre, and removed 500+ "Recommended" entries from my database. Upon my next sync, my Glo locked up and needed a pin reset. After trying to sync again, same thing. And again.

So dangit, it's time for a factory reset. This sucks, because I have to re-sideload all of my books, and I lose all my bookmarks, and so on.

After going through all that, my syncs are indeed muuuuch faster again. You'd be surprised how you just don't notice how slow it gets when it just slowly creeps up on you a little bit at a time. So this time I figured I'd try to be proactive: if the cause of the incredibly long sync times is the amount of Recommended entries in the database, and removing that many at a time creates some kind of internal database problem, maybe I should remove them a little bit at a time.

I deleted the (smaller) crapload of Recommended entries from just a day or two, but again upon my next daily sync, my Glo still won't sync without locking up.

I need a REAL solution here. Kobo? Are you listening?

[taming]

I wonder if the Kobo Utilities plugin could have a setting to remove recommendations, much like it can be set to remove unwanted tiles.