Books Read Checkmarks Vanished

[DoctorOhh]

I just noticed today that none of my books were marked as read. After investigating I know it happened on March 6th while testing whichever beta version happened to exist at that time. I restored from backup so all is fine.

I'm not sure how it happened. If you think the logs would go back that far I'd be glad to send them.

My recollections are fuzzy. I uninstalled and reinstalled CC a couple of times that morning (including wiping the data and cache). I also deleted all of my books via an Android file manager at one point. I also changed from using ID for file name to template naming, then back again.

If no one else has seen this then I would chalk it up to my messing around with the files on my device in a unique manner.

[chaley]

Hmmm....

I would like to know why this happened, but I suspect that it is no longer possible to do the forensics. Too many things have changed, both in CC and in calibre. On the other hand, it is good to know that it did happen, so I can keep it in mind when looking for ways to stress test this stuff.

[DoctorOhh]

Originally Posted by chaley

it is good to know that it did happen, so I can keep it in mind when looking for ways to stress test this stuff.

Just to be clear it was in calibre that the marked read checks disappeared. It was strange. But all is well now.

I have been without power for 4 days due to an ice storm and it is time to try and place my entire library back on my device.

Thanks for taking the time to improve the app.

[chaley]

I think I know what happened. There is an unfortunate interaction between scan for books and syncing. If the sync columns are set when CC scans for books then calibre's data will be overwritten with null values. I am fixing this now.

The workaround is not to set the sync columns until CC has connected for the first time and scanned for books.

[DoctorOhh]

Originally Posted by chaley

I think I know what happened. There is an unfortunate interaction between scan for books and syncing. If the sync columns are set when CC scans for books then calibre's data will be overwritten with null values. I am fixing this now.

The workaround is not to set the sync columns until CC has connected for the first time and scanned for books.

Sounds like you may be on to something. Thursday morning I'll see if I can't recreate the results.

[chaley]

I know the problem is there because it happened to me today during upgrade testing. It took me several moments to figure out what had happened to my db. Fortunately, like you I had a backup and could get back to a known state so I could debug it.

Although it would be great if you could confirm it, given the size of your library the test would take ages. Only bother if you have those ages to spend.

Unfortunately the fix requires a change to calibre as well as to CC, so I can't make a new CC release until Friday.

[DoctorOhh]

Originally Posted by chaley

I know the problem is there because it happened to me today during upgrade testing. It took me several moments to figure out what had happened to my db. Fortunately, like you I had a backup and could get back to a known state so I could debug it.

It is nice to know I'm not alone. As it turns out Dropbox had 1080 backups of my metadata.db file over the last thirty days. Since I haven't added any books since March 4th it was easy to restore to a backup that included the #read column data.

Originally Posted by chaley

Although it would be great if you could confirm it, given the size of your library the test would take ages. Only bother if you have those ages to spend.

This is what the beta process is all about. Since I currently only have 100 books on my device I'll be glad to test it. What are the circumstances required to replicate the error?

[chaley]

Originally Posted by DoctorOhh

This is what the beta process is all about. Since I currently only have 100 books on my device I'll be glad to test it. What are the circumstances required to replicate the error?

The steps I used to make it fail are:

1. Reinitialize CC. You can uninstall/reinstall or use the app manager to clear CC's data. The result is the same.
2. Ensure that there are books in CC's library folder (normally Calibre_Companion) that scan for new books will find.
3. Start CC.
4. Update CC's settings to what you want. In particular, add the sync column lookup names. I also set the UUID as file name check box. Ensure that "Scan for new books" is checked (it is by default).
5. Connect as a wireless device. Let it finish syncing. The read information in calibre should now be gone for any book that is matched as on-device. It isn't necessary to disconnect to note the data loss.

If you alter step four and not put in the lookup names then the data will not be lost. After your first sync completes you can enter the lookup names without triggering the bug. This is why we haven't seen it to date in the prerelease program, because everyone had a working CC install (I think).

[DoctorOhh]

Originally Posted by chaley

The steps I used to make it fail are:

1. Reinitialize CC. You can uninstall/reinstall or use the app manager to clear CC's data. The result is the same.
2. Ensure that there are books in CC's library folder (normally Calibre_Companion) that scan for new books will find.
3. Start CC.
4. Update CC's settings to what you want. In particular, add the sync column lookup names. I also set the UUID as file name check box. Ensure that "Scan for new books" is checked (it is by default).
5. Connect as a wireless device. Let it finish syncing. The read information in calibre should now be gone for any book that is matched as on-device. It isn't necessary to disconnect to note the data loss.

If you alter step four and not put in the lookup names then the data will not be lost. After your first sync completes you can enter the lookup names without triggering the bug. This is why we haven't seen it to date in the prerelease program, because everyone had a working CC install (I think).

I followed your directions and the books were still marked. So I looked closer and saw that you said to have read books that matched on device so while I'm still connected I figure I would transfer some books to my device that were labeled as read and start again, but as soon as the books were on the device the check marks vanished.

I hope that it is consistent with what you expected.

Update:

1. I disconnected then connected again and sent a new set of read books to my device and the same thing happened, but this time not only the new books lost the read mark, the books on the device that matched also lost their marks when the new books were sent.

2. I tried one more time this time with scan for books unchecked and the same thing as in update 1 happened.

[chaley]

Originally Posted by DoctorOhh

I followed your directions and the books were still marked. So I looked closer and saw that you said to have read books that matched on device so while I'm still connected I figure I would transfer some books to my device that were labeled as read and start again, but as soon as the books were on the device the check marks vanished.

I hope that it is consistent with what you expected.

No, that is a new problem. The data in newly-sent books was being lost. I just fixed that.

Update:

1. I disconnected then connected again and sent a new set of read books to my device and the same thing happened, but this time not only the new books lost the read mark, the books on the device that matched also lost their marks when the new books were sent.

2. I tried one more time this time with scan for books unchecked and the same thing as in update 1 happened.

It isn't clear to me whether you had followed all the steps or if this was a normal connection. If it was an "all steps" connection then I suspect that the difference between what I saw and what you are seeing is that the metadata in the books is correct. I would need to check to be sure, but I think that none of the books on my device have the correct custom columns already in them.

Losing the checkmarks on the newly-sent books is the same new problem as above. Losing all the checkmarks when you send new books implies that CC and calibre disagreed on what the checkmark should be, but this doesn't feel right.

I suggest that we hold off any further experiments until we are both running the same codebase. Both CC and calibre have changed to fix the bugs I found earlier in the week (and that you were trying to repeat) and to fix the bug you found today. Both will be released tomorrow.

[DoctorOhh]

Originally Posted by chaley

I suggest that we hold off any further experiments until we are both running the same codebase. Both CC and calibre have changed to fix the bugs I found earlier in the week (and that you were trying to repeat) and to fix the bug you found today. Both will be released tomorrow.

Makes sense to me to wait. I'll try again when CC and calibre are updated.

[DoctorOhh]

With the updates everything works as expected.

[chaley]

Originally Posted by DoctorOhh

With the updates everything works as expected.

Excellent. Thanks for letting me know.