Kobo Wifi/Calibre issue

[SandManSlim]

Hey there,

I have been using Calibre for about a year now ever since I got a Kobo Wifi for Christmas last year. I have also been using this forum for some of the problems that I experienced in the past, but have not registered until now. I just want to say that I have been enjoying calibre and my Kobo thus far(even with some of the pesky problems I have had before).

I recently created a new library on my main PC(used to have my calibre/kobo library on my laptop), but since I am getting rid of my laptop I figured I would create a new library. I didn't transfer anything between libraries, I just installed calibre on my PC, and transferred the books from the device to library. I had to do a lot of resorting/tagging again, but that wasn't a problem.

The problem I have now is that whenever I connect my Kobo to calibre it takes about 40 -50 minutes to send metadata from calibre to the device(including when I first connect, it takes only mere seconds to get device info and get list of books, but still takes about 45 minutes to send metadata during the beginning of a connect). It doesn't matter if I send one book to the device, or even if I am just connecting to calibre. It usually takes that long every time now. Also, the device is always processing info after I disconnect from calibre, even if I have done absolutely nothing in calibre.

I am using Auto management in the metadata preferences, but not sure that should cause this sort of problem. I have about 1300 books in my Ereader.

I was also getting a scan and fix message from windows when I connected my device to my computer(I eventually gave in and tried that, it said it fixed it but I am having issues with the metadata and processing after disconnect).

My goal is to find a solution without having to do a hard reset. Can anyone help me?

PS, If I have to do a hard reset, will the books in my library disappear if I connect my empty kobo? Or will I be able to go to library and load all of my books back onto my device?

[davidfor]

That doesn't sound right for a Kobo WiFi. I could think of a reason for a Kobo Touch, but over 40 minutes would still be to long.

Some questions:

- What version of calibre?

- When the device is connected, what driver is being used? Right click on the device button in the toolbar, select "Configure this device" and tell me what the titlebar says.

- On the configuration for the driver, what do you have for the collections field? This is the one after the text "The Kobo supports several...". What are the other options set to.

- When the device is connected, could you have a look in the ".kobo" directory and tell me the size of the file "KoboReader.sqlite"?

- Is Windows mounting the device as USB2? If it is mounting it as USB1.2, it would be a lot slower.

- You say you created a new library but didn't

For your other comments:

- If the device is processing each time you disconnect from the PC, it means there are books on the device it hasn't processed. This might mean there is a book that it tries to process each time but fails. Then next time you disconnect it the device will try again. If this is the case you might be missing other books from the library on the device. To check this, you would have to compare a file listing on the device against the device library list to see what was missing.

- "Auto management" does add more time. But, it shouldn't add that much. With auto management and a column specified in the collections field, it will be changing the reading status and shortlist each time you connect. This will account for some of the time. 1300 is a reasonable amount of books, but I was talking to someone who had 5000 on a Kobo Touch, and the sync time was about 15 minutes.

- Letting Windows fix the filesystem on the device was the correct thing to do. This could have contributed to the slowness.

- I suspect there might be an issue that will need a factory reset. If you do this, if all the books are in calibre, you will able to send the books from calibre to the device. If you have any kepubs downloaded from Kobo, they will be put on when you do a sync with Kobo. That should happen automatically after the latest firmware is loaded.


I'm not sure how much this will help. But the info will help we think about what is going on in the driver.

[SandManSlim]

- What version of calibre?

0.9.2, I think there is a new update which I have not downloaded yet.

- When the device is connected, what driver is being used? Right click on the device button in the toolbar, select "Configure this device" and tell me what the titlebar says.

The titlebar? Not sure what you mean by that, epub, pdf, txt, cbz, cbr are the only ones checked in the above format box. There is a save template with "{author_sort}/{title} - {authors}".

- On the configuration for the driver, what do you have for the collections field? This is the one after the text "The Kobo supports several...". What are the other options set to.

Read, Closed, Im_Reading. Everything is checked except for Show recommendations, show previews, and attempt to support newer firmware.

- When the device is connected, could you have a look in the ".kobo" directory and tell me the size of the file "KoboReader.sqlite"?

45.9 mb

- Is Windows mounting the device as USB2? If it is mounting it as USB1.2, it would be a lot slower.

I believe it's USB2, I would have no clue as to why it would be mounting it as USB1.2.

- You say you created a new library but didn't

My apologies, I should have worded that better, technically no I did not create a new library. I just installed calibre on a new device, and uploaded the books on the device into the calibre library.

For your other comments:

- If the device is processing each time you disconnect from the PC, it means there are books on the device it hasn't processed. This might mean there is a book that it tries to process each time but fails. Then next time you disconnect it the device will try again. If this is the case you might be missing other books from the library on the device. To check this, you would have to compare a file listing on the device against the device library list to see what was missing.

This is the case for every time I disconnect from calibre. It is always trying to process something, and apparently it fails every time.

- "Auto management" does add more time. But, it shouldn't add that much. With auto management and a column specified in the collections field, it will be changing the reading status and shortlist each time you connect. This will account for some of the time. 1300 is a reasonable amount of books, but I was talking to someone who had 5000 on a Kobo Touch, and the sync time was about 15 minutes.

Agreed, there is no way it should take this long, especially if it used to only taking 5 minutes.

- Letting Windows fix the filesystem on the device was the correct thing to do. This could have contributed to the slowness.

I let it fix it, but I am still having the "metadata slowness" problem.

- I suspect there might be an issue that will need a factory reset. If you do this, if all the books are in calibre, you will able to send the books from calibre to the device. If you have any kepubs downloaded from Kobo, they will be put on when you do a sync with Kobo. That should happen automatically after the latest firmware is loaded.

The only books I have from Kobo, were the 100 free books it came with. The rest is all on calibre. I was only worried that if I connected my empty Kobo to calibre, it would somehow delete all the books in the library, silly question but I needed to ask it regardless. I figured it might come to this though, but if I can add my books from the calibre library again, then it will not be so bad if a hard reset is needed.

PS, I want to thank you for taking the time to try and help me, I realize there is only so much you can do from your end, but I still appreciate it.

[davidfor]

Originally Posted by SandManSlim

0.9.2, I think there is a new update which I have not downloaded yet.

0.9.3 is out and has a performance improvement in the driver but that shouldn't get used by the WiFi.

The titlebar? Not sure what you mean by that, epub, pdf, txt, cbz, cbr are the only ones checked in the above format box. There is a save template with "{author_sort}/{title} - {authors}".

The titlebar of a window is the bit across the top of the window with the title of the window and the minimize, maximise and close buttons in it.

Read, Closed, Im_Reading. Everything is checked except for Show recommendations, show previews, and attempt to support newer firmware.

The field has "Read, Closed, Im_Reading" in it? That's not how that field works. It should have a column name in it which is used to set the status of books that have those values in the column. As those columns don't exist in your library, this shouldn't be affecting it.

45.9 mb

Big, but in proportion to my database and the number of books in ot.

My apologies, I should have worded that better, technically no I did not create a new library. I just installed calibre on a new device, and uploaded the books on the device into the calibre library.

Actually, I just reread what I wrote and I stopped in the middle of a thought. I think I started writing something went back and checked what you had written and then forgot to remove my comment. But, what you did should be OK.

This is the case for every time I disconnect from calibre. It is always trying to process something, and apparently it fails every time.

Calibre shouldn't be causing this unless you send a book over. Can you connect without running and then disconnect and see what happens.

The only books I have from Kobo, were the 100 free books it came with. The rest is all on calibre. I was only worried that if I connected my empty Kobo to calibre, it would somehow delete all the books in the library, silly question but I needed to ask it regardless. I figured it might come to this though, but if I can add my books from the calibre library again, then it will not be so bad if a hard reset is needed.

The Kobo books should be put back on by the reset. And you can use calibre to put the others back on. I've done this plenty of times with my Touch.

Something you can do that might give me some info, is to run calibre in debug mode. To do this:

- Start calibre without your reader connected
- Right-click on the Preferences button and select "Restart in debug mode"
- After calibre restarts, connect your reader
- When the sync with the reader finishes, close calibre
- A log file will be displayed. Send me the log. Hopefully it will give me a clue.

[SandManSlim]

Sorry for the late response, my 3d card gave out a day or two after you sent your last message. I decided to just overhaul my whole computer. I got all of the hardware installed, as well as a fresh install of windows about a week ago, then my monitor decided to fry in the middle of the night. So I finally got my new monitor today, but I still need to re upload my calibre library from my device.

I will probably still have the same problem, as I suspect it is my kobo that is having the problem, but we will see. If I do have the same problem, I will try and do as you have suggested. I might need to check to see if all of those settings i had before are the same(ie titlebar info).

Again, my apologies for the late reply, everything seemed to hit the fan right after starting this thread.

[davidfor]

Originally Posted by SandManSlim

Sorry for the late response, my 3d card gave out a day or two after you sent your last message. I decided to just overhaul my whole computer. I got all of the hardware installed, as well as a fresh install of windows about a week ago, then my monitor decided to fry in the middle of the night. So I finally got my new monitor today, but I still need to re upload my calibre library from my device.

Ouch! After that I'm not at all surprised at the delay.

I will probably still have the same problem, as I suspect it is my kobo that is having the problem, but we will see. If I do have the same problem, I will try and do as you have suggested. I might need to check to see if all of those settings i had before are the same(ie titlebar info).

I have made some changes since the last time you tried that will speed things up. Plus some more going into the next calibre build.

Edit: Sorry, I forgot this was a WiFi rather than Touch or later. I haven't made any changes in the WiFi part of the driver.

[SandManSlim]

ok, so I re-uploaded my calibre library from my wifi device to the calibre library on my pc. I tried automatic management, and it is still taking around 40 - 50 minutes to send metadata.

It might be because I changed the tags(Im_reading) for every one of my books. The reason I am doing that is because the "im currently reading list" on my device doesn't show up correctly if I don't have the tags closed, read, or Im_currently reading tagged to all of my books). I was actually going off an earlier post that you made in another thread.

Originally Posted by davidfor

What do you have "Metadata management" set to on the "Sending books to devices" page of the calibre preferences? It sounds like you have it set to automatic. Using automatic, the current driver will reset the reading status of all books except the ones with the tags "Im_Reading", "Read" or "Closed". And it will reset the shortlist unless the book has the tag "Shortlist". If you don't want this to happen, change this option to "manual".

https://www.mobileread.com/forums/sho...d.php?t=192688

You may be wondering why I just don't switch to manual. Well I like being able to change the metadata on the fly without having to send another copy of the book to the device, while deleting the older copy. The other reason is that my "currently reading" section also gets screwed up(just like if I don't have the correct tags in auto management, all my "read" books become 99%, or my currently reading is not saving) when I use manual management. Is there something I need to change in manual management to correct this issue?

I have not tried transferring something with manual management yet, so I do not know if it takes 40 minutes for the metadata as well, but I will check that out.

Last but not least, my Kobo is still trying to process something regardless of whether or not I change anything in it's library. I am under the assumption that I might just need to hard reset the device to get rid of this issue at least.

I will be trying calibre in debug mode tonight, I will upload the log when it is finished.

[SandManSlim]

calibre Debug log
calibre 0.9.3
Windows-7-6.1.7601-SP1 Windows
('Windows', '7', '6.1.7601')
Python 2.7.3
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Starting up...
Started up in 36.60 seconds with 1138 books
Job: 1 Get device information started
DeviceJob: 1 Get device information done, calling callback
DeviceJob: 1 Get device information callback returned
Job: 2 Get list of books on device started
DEBUG: 0.0 Version of driver: (2, 0, 2) Has kepubs: True
DEBUG: 0.0 Version of firmware: 1.9 Has kepubs: True
Job: 1 Get device information finished
No details available.
DEBUG: 4.2 Database Version: 61
DeviceJob: 2 Get list of books on device done, calling callback
DeviceJob: 2 Get list of books on device callback returned
Job: 3 Send metadata to device started
DEBUG: 13.9 Starting get_collections: on_connect
DEBUG: 13.9 Renaming rules: {}
DEBUG: 13.9 Formatting template: {value}{category:| (|)}
DEBUG: 13.9 Sorting rules: []
Job: 2 Get list of books on device finished
No details available.
DEBUG: 3006.9 USBMS: starting sync_booklists
DEBUG: 3010.9 USBMS: finished sync_booklists
DeviceJob: 3 Send metadata to device done, calling callback
DeviceJob: 3 Send metadata to device callback returned
Job: 3 Send metadata to device finished
No details available.

[davidfor]

Originally Posted by SandManSlim

ok, so I re-uploaded my calibre library from my wifi device to the calibre library on my pc. I tried automatic management, and it is still taking around 40 - 50 minutes to send metadata.

On the laptop you were using, how long did this take? And what calibre version was it?

It might be because I changed the tags(Im_reading) for every one of my books. The reason I am doing that is because the "im currently reading list" on my device doesn't show up correctly if I don't have the tags closed, read, or Im_currently reading tagged to all of my books). I was actually going off an earlier post that you made in another thread.

https://www.mobileread.com/forums/sho...d.php?t=192688

OK, that makes sense. The driver does an update for each book in one of these states. So, the more books, the longer it takes.

You may be wondering why I just don't switch to manual. Well I like being able to change the metadata on the fly without having to send another copy of the book to the device, while deleting the older copy.

The only metadata the driver changes on the device is the reading status and the shortlist entries. Nothing else gets changed. If you change the title of a book, you have to delete it from the device and resend it to get the update.

The other reason is that my "currently reading" section also gets screwed up(just like if I don't have the correct tags in auto management, all my "read" books become 99%, or my currently reading is not saving) when I use manual management. Is there something I need to change in manual management to correct this issue?

The database on the device stores a reading status (Unread, Im_Reading, Read and Closed) and the percent read. Looking at the database on my Touch, the percent read doesn't always get set to 100 when the book is marked as Read. So, if something happens that the status gets set back to "Im_Reading", the device will show the 99%.

If metadata management is automatic, the driver sets all books to unread. Then for each book that has the tag for another status, it updates the status. It is likely the that if device sees a book as unread but a percent read other than zero, that it shows this rather than unread.

If metadata management is manual, nothing should be changed in the devices database. The driver reads the status and shows it in the device list, but it doesn't change it. If the read status is being lost, then something else is going on.

I have not tried transferring something with manual management yet, so I do not know if it takes 40 minutes for the metadata as well, but I will check that out.

It shouldn't take that long, but I would be interested to hear.

Last but not least, my Kobo is still trying to process something regardless of whether or not I change anything in it's library. I am under the assumption that I might just need to hard reset the device to get rid of this issue at least.

That might be a book that isn't being processed properly. If there is a problem processing a book, then the device stops processing it and any more on the device. So, the book doesn't get added to the database. The next time you connect to USB and disconnect, it will try again. Unfortunately, it is a manual process to work out which is the bad book. You need to look at the file system and compare it to what is displayed in the device library. You can do this when connected by looking at the device list in calibre.

With some of the other problems, this might be a sign that the database is bad. It might be processing the book and adding it to the in memory list of books. When it writes the database, if there is a problem, it might not be able to save the changes.

Your loss of reading status might also be a symptom. You read the book, it gets marked as read but when you connect to USB the status is lost because it can't write the database and it rereads it from an old state. A test for this would be to connect to the PC without calibre and then do a safe eject and see if the status is lost. If it is, the database is bad.

I will be trying calibre in debug mode tonight, I will upload the log when it is finished.

Ok. I will be interested to see it.