Bug Report re Wireless Sync

sternbyname · 08-14-2018, 04:05 AM

Bug Report re Wireless Sync: If you delete a book in Calibre on the PC then import a different version, then when you wireless sync Calibre Companion it appears to update to the new book, ie the database seems to be changed, but in fact the epub is unchanged, ie if you open the book on your phone it is still the old version of the book. I've changed book versions a few times with out-of-copyright books, ie when I find a better version, and I simply don't know how many books in CC are in fact old epub versions now. I'm running the latest version of CC on a Samsung Galaxy S7, and the latest version of Calibre on a Windows 10 PC.
How to reproduce: Run 'Wireless Device/sync books' in Calibre Companion. Delete a book on the PC, eg Great Expectations by Charles Dickens. Import a new version of it (eg Gutenberg or FeedBooks free versions), ie with the same title and author name. Run 'Wireless Device/sync books' in Calibre Companion again. It looks in CC like you have the new version of Great Expectations, but if you open the book to read it you will find it is the old version, ie the one that was deleted on the PC.

chaley · 08-14-2018, 05:04 AM

If you want the actual books to be updated use "Wireless device connection/Sync newer book files". This makes calibre check the timestamp on the book in the calibre library and send it to CC if the book is newer than the one in CC. Go to Settings / Connecting to calibre then check the appropriate box. Uncheck the other Wireless device connection type if you want CC + calibre to always check the timestamp. For more info see our FAQ What does "Sync newer book files" (Settings / Connecting to calibre) do?

This cannot be the default for two reasons:

Checking the timestamp in calibre can be very slow, causing syncing to also be slow.
The calibre viewer can change the timestamp of a book. We don't want to download the book every time that happens

Also note that calibre does not store the fact that you changed the book, so it can't tell the difference between editing and replacing a book.

sternbyname · 08-14-2018, 07:11 AM

Hi, Thanks for your prompt reply. However unfortunately it is not correct. I already had the checkbox ticked, and I have unticked the other one as you suggest to be sure, and the problematic behaviour is exactly the same.
With respect, Calibre behaves quite differently when you delete and replace a book rather than editing it. When you edit it the epub file remains in the same directory. When you delete and replace then the directory name is different, with a different number. Whether this is what confuses CC's sync system I don't know.
If you try reproducing the problem as above you will see the issue. By the way, if you then edit the new book and sync again then CC is correctly updated, ie syncing after editing works fine but syncing after delete-and-replace does not.

chaley · 08-14-2018, 07:27 AM

Ahh, you are actually deleting the calibre book record then re-adding it? Why? It seems that you should replace the format in the book record, not re-create the record. It is trivial to replace the format; simply show the book details for the book in question then drag the new book file onto the cover image.

In any event, the problem is that the file name template you are using for sending books to CC does not use the book ID, so CC thinks that the two books are the same calibre record when in fact they are not. This is the same situation that arises if you have different books with identical title and author in which case the last book sent wins. To avoid this difficulty you should add the book id to the template (see note #3 of this FAQ). Once you do this then any replaced books will no longer show as on-device and won't sync metadata.

As for not knowing if books are up to date, you can solve that problem by resending all the books to your device. Connect, search for OnDevice:true, select them all, and send them.

sternbyname · 08-14-2018, 10:26 AM

Thanks. It sounds like you understand the bug now.
I understand that I can use workarounds, and am aware of these, I just thought it was worth reporting the bug.
It may just be me but I'd simply like the CC books to replicate the ones on my PC, that's it, and inbuilt functionality to do that would be great.
I appreciate the great work the CC team is doing.
Cheers.

theducks · 08-14-2018, 01:32 PM

Quote:

Originally Posted by sternbyname

Thanks. It sounds like you understand the bug now.
I understand that I can use workarounds, and am aware of these, I just thought it was worth reporting the bug.
It may just be me but I'd simply like the CC books to replicate the ones on my PC, that's it, and inbuilt functionality to do that would be great.
I appreciate the great work the CC team is doing.
Cheers.

It is not a BUG.
Every title in Calibre gets an {id} assigned. You see this if you look unfrt the skirts of the library folder: title (###)

when you delete the book (remove from Library vs remove format), the ID is discarded and never reused.
When you NOW add a BOOK, it gets a new ID.

The book on your device matches the now non-existent ID (AKA no match). That is why it does not get replaced.
If you replace a FORMAT (or add a different one), the ID stay the same and replace should work

sternbyname · 08-14-2018, 02:06 PM

Sure, and that would be fine if CC actually did that coherently, but it doesn't. From the users point of view the default behaviour does not work as expected, or indeed logically. When you delete and reimport a book in Calibre then on sync CC changes the image etc to show the new book information, but the epub is for the old book. That's a bug from the users point of view. If you connect to the content server it wrongly claims that you have the new book on your device, which you don't. That's a bug from the user's point of view. CC should logically either leave it unchanged as you suggest above, ie treat the old book as completely different from the new book, or change it properly to the new book. Either would be logical and understandable, but the current behaviour makes it look like it's doing one thing while it's actually half doing another.
OK, so maybe CC has been programmed to do that (but surely this behaviour is an unfortunate side-effect rather than deliberate?) but it surely looks very like a bug from the user end.

chaley · 08-14-2018, 02:29 PM

Quote:

Originally Posted by sternbyname

OK, so maybe CC has been programmed to do that (but surely this behaviour is an unfortunate side-effect rather than deliberate?) but it surely looks very like a bug from the user end.

This is not a bug in CC. It is how calibre devices have worked for my 8 years of being a calibre developer. Books are matched by calibre using their file name on the device, something that is built very deeply into calibre and will not change. The file name is controlled by a template, and in CC's case also by a CC-side template. Neither CC nor calibre know that you did a presto-chango and created a new book that has the same device file name as the old one.

If you want books to change when their calibre book ID changes then you must add the book ID to the file name used by the device.

sternbyname · 08-14-2018, 03:53 PM

Yes, Calibre works great and has in my experience too been remarkably robust for many years. It works well when adding, deleting or changing books, presto or otherwise, never getting confused or half-changing records.
Regarding CC, I would suggest that either changing or not changing the book record when the Calibre ID changes would be just fine. It's half-changing some bits of the record but not others that is the problem for me. But hey, if that's what it's supposed to do then I can certainly work around it.
Thanks for your time.

sternbyname · 08-14-2018, 04:47 PM

Oh, sorry, I forgot to say, I do already use the ID in the file name used on my mobile device, ie {title} - (authors} ({id}), if that's what you meant. The changing-the-record-but-not-the-epub-file issue occurs in that situation. The old id is retained in the file name after a wireless sync, ie the epub is unchanged. However if I subsequently edit the new book and do another wireless sync the file name is changed to use the new ID so it is all then synced correctly.

theducks · 08-14-2018, 06:15 PM

Quote:

Originally Posted by sternbyname

Oh, sorry, I forgot to say, I do already use the ID in the file name used on my mobile device, ie {title} - (authors} ({id}), if that's what you meant. The changing-the-record-but-not-the-epub-file issue occurs in that situation. The old id is retained in the file name after a wireless sync, ie the epub is unchanged. However if I subsequently edit the new book and do another wireless sync the file name is changed to use the new ID so it is all then synced correctly.

Calibre use internal metadata for matching, not the file name. Note ID can be different if you have different Calibre installs (different computers), because ADD uses the next available ID. So if you add books to different systems, transfe (eg copy to Library), the ID will end up different between systems.
I ended up dropping {ID} from the file name a bunch of years ago for thos reasons.

DoctorOhh · 08-15-2018, 09:18 AM

Quote:

Originally Posted by sternbyname

OK, so maybe CC has been programmed to do that (but surely this behaviour is an unfortunate side-effect rather than deliberate?) but it surely looks very like a bug from the user end.

This is not a CC issue, it is a unique user work case issue that rarely ever occurs in calibre. The reason this does not occur in calibre is because if folks want to update a book they simply drag the new version to the book details panel of the existing book and the book is updated without the need for a new ID. This process updates only the book, not any of the metadata.

There is no reason I can see for deleting a book then re-adding the book to update that book in calibre. This process creates a completely new book as far as calibre is concerned. This work flow is unique to you.

sternbyname · 08-16-2018, 12:51 AM

Brilliant! It's not a bug, just a user who doesn't do things the way they should, like good ole folks who are all just exactly like you.
It's ok, I've got the message - no more bug reports on CC.

Dopedangel · 08-16-2018, 11:16 PM

Quote:

Originally Posted by sternbyname

Brilliant! It's not a bug, just a user who doesn't do things the way they should, like good ole folks who are all just exactly like you.
It's ok, I've got the message - no more bug reports on CC.

To your point of wanting to sync all books in calibre with calibre companion. That is possible for that you need to use the reading list plugin in calibre. Use inlibrary:false to make a list and use remove from device option in reading list plugin. Make a 2nd list using ondevice:false and send books device option in reading list plugin. This will automatically remove and add books like you want when ever you connect calibre companion to calibre.

I agree with what the others have said about this not being a bug and your use case being bit out of the ordinary. At the same time I do see how sync books dialog in calibre companion might confuse people if they are new to it. I think it might be better of as sync updated books. You don't need to be so angry about people not agreeing with you.

sternbyname · 08-17-2018, 07:48 AM

Thanks Dopedangel, I'll give it a go (and thanks Chaley for closing the 'feature request' thread).
And to reiterate, I really appreciate the work the CC team (and Calibre team of course) are doing - it's great software that I use all the time. I only submit feature requests and (occasionally) 'unexpected behaviour' reports for good software - I know myself it's not helpful if software needs major work anyway.
Cheers.

08-14-2018, 04:05 AM	#1
sternbyname Member Posts: 16 Karma: 10 Join Date: Apr 2014 Device: none	Bug Report re Wireless Sync Bug Report re Wireless Sync: If you delete a book in Calibre on the PC then import a different version, then when you wireless sync Calibre Companion it appears to update to the new book, ie the database seems to be changed, but in fact the epub is unchanged, ie if you open the book on your phone it is still the old version of the book. I've changed book versions a few times with out-of-copyright books, ie when I find a better version, and I simply don't know how many books in CC are in fact old epub versions now. I'm running the latest version of CC on a Samsung Galaxy S7, and the latest version of Calibre on a Windows 10 PC. How to reproduce: Run 'Wireless Device/sync books' in Calibre Companion. Delete a book on the PC, eg Great Expectations by Charles Dickens. Import a new version of it (eg Gutenberg or FeedBooks free versions), ie with the same title and author name. Run 'Wireless Device/sync books' in Calibre Companion again. It looks in CC like you have the new version of Great Expectations, but if you open the book to read it you will find it is the old version, ie the one that was deleted on the PC.

08-14-2018, 05:04 AM	#2
chaley Grand Sorcerer Posts: 11,741 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	If you want the actual books to be updated use "Wireless device connection/Sync newer book files". This makes calibre check the timestamp on the book in the calibre library and send it to CC if the book is newer than the one in CC. Go to Settings / Connecting to calibre then check the appropriate box. Uncheck the other Wireless device connection type if you want CC + calibre to always check the timestamp. For more info see our FAQ What does "Sync newer book files" (Settings / Connecting to calibre) do? This cannot be the default for two reasons: Checking the timestamp in calibre can be very slow, causing syncing to also be slow. The calibre viewer can change the timestamp of a book. We don't want to download the book every time that happens Also note that calibre does not store the fact that you changed the book, so it can't tell the difference between editing and replacing a book.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How to report a bug	osmin	Calibre	4	06-20-2017 05:56 PM
Bug report	mtravellerh	PocketBook	303	11-03-2013 07:21 PM
How to report a bug???	tmlee51	Calibre	2	02-14-2012 11:07 PM
Can't get bug report to work	AlexBell	Conversion	6	09-14-2011 04:49 AM

08-14-2018, 07:11 AM	#3
sternbyname Member Posts: 16 Karma: 10 Join Date: Apr 2014 Device: none	Hi, Thanks for your prompt reply. However unfortunately it is not correct. I already had the checkbox ticked, and I have unticked the other one as you suggest to be sure, and the problematic behaviour is exactly the same. With respect, Calibre behaves quite differently when you delete and replace a book rather than editing it. When you edit it the epub file remains in the same directory. When you delete and replace then the directory name is different, with a different number. Whether this is what confuses CC's sync system I don't know. If you try reproducing the problem as above you will see the issue. By the way, if you then edit the new book and sync again then CC is correctly updated, ie syncing after editing works fine but syncing after delete-and-replace does not.

08-14-2018, 07:27 AM	#4
chaley Grand Sorcerer Posts: 11,741 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	Ahh, you are actually deleting the calibre book record then re-adding it? Why? It seems that you should replace the format in the book record, not re-create the record. It is trivial to replace the format; simply show the book details for the book in question then drag the new book file onto the cover image. In any event, the problem is that the file name template you are using for sending books to CC does not use the book ID, so CC thinks that the two books are the same calibre record when in fact they are not. This is the same situation that arises if you have different books with identical title and author in which case the last book sent wins. To avoid this difficulty you should add the book id to the template (see note #3 of this FAQ). Once you do this then any replaced books will no longer show as on-device and won't sync metadata. As for not knowing if books are up to date, you can solve that problem by resending all the books to your device. Connect, search for OnDevice:true, select them all, and send them.

08-14-2018, 10:26 AM	#5
sternbyname Member Posts: 16 Karma: 10 Join Date: Apr 2014 Device: none	Thanks. It sounds like you understand the bug now. I understand that I can use workarounds, and am aware of these, I just thought it was worth reporting the bug. It may just be me but I'd simply like the CC books to replicate the ones on my PC, that's it, and inbuilt functionality to do that would be great. I appreciate the great work the CC team is doing. Cheers.

08-14-2018, 02:06 PM	#7
sternbyname Member Posts: 16 Karma: 10 Join Date: Apr 2014 Device: none	Sure, and that would be fine if CC actually did that coherently, but it doesn't. From the users point of view the default behaviour does not work as expected, or indeed logically. When you delete and reimport a book in Calibre then on sync CC changes the image etc to show the new book information, but the epub is for the old book. That's a bug from the users point of view. If you connect to the content server it wrongly claims that you have the new book on your device, which you don't. That's a bug from the user's point of view. CC should logically either leave it unchanged as you suggest above, ie treat the old book as completely different from the new book, or change it properly to the new book. Either would be logical and understandable, but the current behaviour makes it look like it's doing one thing while it's actually half doing another. OK, so maybe CC has been programmed to do that (but surely this behaviour is an unfortunate side-effect rather than deliberate?) but it surely looks very like a bug from the user end.

08-14-2018, 03:53 PM	#9
sternbyname Member Posts: 16 Karma: 10 Join Date: Apr 2014 Device: none	Yes, Calibre works great and has in my experience too been remarkably robust for many years. It works well when adding, deleting or changing books, presto or otherwise, never getting confused or half-changing records. Regarding CC, I would suggest that either changing or not changing the book record when the Calibre ID changes would be just fine. It's half-changing some bits of the record but not others that is the problem for me. But hey, if that's what it's supposed to do then I can certainly work around it. Thanks for your time.

08-14-2018, 04:47 PM	#10
sternbyname Member Posts: 16 Karma: 10 Join Date: Apr 2014 Device: none	Oh, sorry, I forgot to say, I do already use the ID in the file name used on my mobile device, ie {title} - (authors} ({id}), if that's what you meant. The changing-the-record-but-not-the-epub-file issue occurs in that situation. The old id is retained in the file name after a wireless sync, ie the epub is unchanged. However if I subsequently edit the new book and do another wireless sync the file name is changed to use the new ID so it is all then synced correctly.

08-16-2018, 12:51 AM	#13
sternbyname Member Posts: 16 Karma: 10 Join Date: Apr 2014 Device: none	Brilliant! It's not a bug, just a user who doesn't do things the way they should, like good ole folks who are all just exactly like you. It's ok, I've got the message - no more bug reports on CC.

08-17-2018, 07:48 AM	#15
sternbyname Member Posts: 16 Karma: 10 Join Date: Apr 2014 Device: none	Thanks Dopedangel, I'll give it a go (and thanks Chaley for closing the 'feature request' thread). And to reiterate, I really appreciate the work the CC team (and Calibre team of course) are doing - it's great software that I use all the time. I only submit feature requests and (occasionally) 'unexpected behaviour' reports for good software - I know myself it's not helpful if software needs major work anyway. Cheers.