Kindle sync, tweet, and facebook- revision 10439

[jacr]

Calibre 0.8.19 included changes intended to allow social sharing of non amazon items to work on Kindle. The way it did this is by not writing the ASIN out to the mobi file if a preference option is set.

However this does not work as intended. Removal of the ASIN does not allow sharing on the kindle.

There are three functions a kindle user might wish to do which are affected by the ASIN:

1) Sync last page read between devices
2) Tweet or facebook a quote from a book
3) Tweet or facebook when book is completed

Function 1 requires CDEContentType=EBOK, and a unique ASIN. ASIN does not have to be valid, ie does not have to point to valid amazon product
Function 2 requires either no CDEContentType, or CDEContentType=PDOC (note that CDEContentType=EBOK will allow tweeting, but will not include the quote, so it's kinda pointless)
Function 3 requires CDEContentType=EBOK, and a valid ASIN, ie pointing to an amazon product


Therefore the best way to fix sharing (and as a bonus, keep "last page read" sync) would be:
1) If book is linked to Amazon, store the ASIN in metadata
2) Check if book has an ASIN in metadata. If so, put ASIN in ASIN EXTH, and keep the CDEContentType of EBOK.
3) If book does not have an ASIN in meta data, check the new preference option added in last release (share_not_sync). If set, do not write CDEContentType=EBOK header (this will allow sharing of quotes). If not set, write CDEContentType=EBOK. In both cases write the calibre unique id as the ASIN.

This gives the option for users to link their some books to amazon, and preserves the choice (sharing or sycning) for content that isn't on amazon.

From a forum search, it looks like siebert posted a patch to pull ASINs from the amazon meta data plugin and put them into the EXTH. However I'm not sure if this patch was ever applied.

If someone can confirm my thinking above, and maybe give me some history on siebert's patch, I can merge the patch with the latest changes, test the various scenarios, and supply a new patch.

[kovidgoyal]

I do not want to add ASINs downloaded by the metadata fetch to calibre produced MOBI files. ASINs are intended to identify unique editions and the MOBI file pointed to by an ASIN could differ substantially from the calibre produced MOBI version of that file. Whether that leads to any actual problems anywhere in Amazon's byzantine system, is unclear, but it is not something I think is a good idea. For example, what happens to people that use calibre to generate MOBI files for submission to Amazon's DTP platform? What happens if Amazon decides to implement some kind of cross-user notes compare/sync feature?

Instead, I can have the share_not_sync option simply prevent writing of the cdetype, if that is what is needed to enable sharing.

[jacr]

I understand you're wary of breaking parts of Amazons functionality as we don't know what's going on there. I also understand your concern that future changes by amazon could break functionality. However:

1) I've tested the logic and scenarios I listed above and they work. I can do further testing before providing a patch, including other scenarios if you wish.
2) We're already exposed to the risk that future changes could break functionality. We're putting a random ASIN into the file, which currently enables last page sync to work- there's no guarantee this will keep working.
3) The current interaction between calibre mobi files and the kindle sharing/sync is significantly broken- of the three functions listed above only one will work at a time. This isn't intended as a criticism of calibre- it's amazon behavior. However I can't change amazon, but I can change calibre. If I've told calibre that a given non amazon book is the same as a given amazon book, then all of these functions can be made to work.

Regarding your specific concerns: a user generating MOBI files for submission to Amazon's DTP platform will not be linking the book to an amazon book using the metadata plugin, so behaviour will be as current. I'm not sure what the issue would be with the cross user notes?


Why not let the user decide on whether to include the linked, via a configuration option on the mobi output? Leave it off by default, and let users who want the full sharing/sync functionality turn it on?

I'm happy to implement, test, and maintain this. I think it would be of value to kindle users.

[kovidgoyal]

Originally Posted by jacr

I understand you're wary of breaking parts of Amazons functionality as we don't know what's going on there. I also understand your concern that future changes by amazon could break functionality. However:

1) I've tested the logic and scenarios I listed above and they work. I can do further testing before providing a patch, including other scenarios if you wish.

Try setting the cdetype to PDOC and see if syncing works with a uuid as ASIN

2) We're already exposed to the risk that future changes could break functionality. We're putting a random ASIN into the file, which currently enables last page sync to work- there's no guarantee this will keep working.

I'm not worried about the functionality breaking in the future, I'm worried about the side effects of ASIN collisions. Currently calibre uses a UUID of type 4. That cannot collide with an ASIN.

3) The current interaction between calibre mobi files and the kindle sharing/sync is significantly broken- of the three functions listed above only one will work at a time. This isn't intended as a criticism of calibre- it's amazon behavior. However I can't change amazon, but I can change calibre. If I've told calibre that a given non amazon book is the same as a given amazon book, then all of these functions can be made to work.

The problem is the books are explicitly not the same. A calibre MOBI is quite likely to have gone through a conversion/metadata update.

Regarding your specific concerns: a user generating MOBI files for submission to Amazon's DTP platform will not be linking the book to an amazon book using the metadata plugin, so behaviour will be as current.

Relying on users not doing something is rather optimistic. In my years of supporting calibre I have learned that if something can be done, someone will do it.

I'm not sure what the issue would be with the cross user notes?

Annotations will be keyed by a MOBI location, The location will point to different actual places in different MOBI files with the same ASIN. This will lead to strange bugs.

Why not let the user decide on whether to include the linked, via a configuration option on the mobi output? Leave it off by default, and let users who want the full sharing/sync functionality turn it on?

Because this kind of thing is likely to lead to hard to debug problems in a closed system which I will then have to answer the questions for.

I'm happy to implement, test, and maintain this. I think it would be of value to kindle users.

Feel free to run calibre from source and maintain the patch in your local copy.

[kovidgoyal]

Another undesirable effect of implementing this is that content sharing will work for some books (those that the user happens to have downloaded metadata for) and not for others. Which will lead to endless "bug reports", all of which I will have to handle.

[jacr]

Originally Posted by kovidgoyal

Try setting the cdetype to PDOC and see if syncing works with a uuid as ASIN

Nope, only CDETYPE=EBOK allows syncing.

Originally Posted by kovidgoyal

Feel free to run calibre from source and maintain the patch in your local copy.

Doing just that

One correction to my comments above, in case it's of use to anyone in the future: sharing of content doesn't work with CDECONTENTTYPE=EBOK. I've updated my first post to reflect this.

Your latest CDECONTENTTYPE patch works for me-ie when share_not_sync option is checked, I can now share quotes.

Cheers

[kovidgoyal]

Originally Posted by jacr

Nope, only CDETYPE=EBOK allows syncing.

Pity hopefully someday Amazon will realize their users may want to read more than Amazon purchased content on their Kindles/Kindle apps.

Your latest CDECONTENTTYPE patch works for me-ie when share_not_sync option is checked, I can now share quotes.

Cheers

Glad to hear it.

[tomsem]

In light of the new, improved Personal Documents functionality, do there need to be some changes to this 'random ASIN' thing that calibre does?

Amazon seems to run .mobi files through some conversion, not simply storing the file in the PD library. The file size is different, file extension is .azw, presumably they are adding a UID somewhere also. I haven't looked at the resultant .azw to see what exactly is in there but I trust someone involved in calibre dev will be doing so.

In the meantime I am turning on the 'Enable sharing' MOBI output preference, as this apparently turns off the 'random ASIN' behavior—in case that interferes with what Amazon is doing.

Thanks.

[kovidgoyal]

I suspect they are adding a PDOC based ASIN to the files and recording it on their servers as the file is processed. I doubt this would work for sideloaded books loaded via USB. I suggest you simply use the email functionality on calibre to send your books to the device to use this feature.