Kobo, why won't you sync my sideloaded books?

[speakingtohe]

Calibre would probably have to store bookmarks for a variety of formats or somehow get the last page read from a variety of devices/apps. Then there would have to be a method of keeping track of your different devices/apps and send the information to either all or whichever you choose.

AFAIK there is no way to automatically connect to a device wirelessly, the device has to connect to the content server although some may be able to be connected manually to calibre . And there is no automatic way to differentiate between two different kindle keyboards for example when they are hooked up via USB, and you can only have one connected at once.

Calibre will send files via USB and it may even send them over wireless? Syncing seems a bit more complex.

Helen

[jackastor]

Copyrights aside, there is no need for Kobo to transfer any books from readers where the book is purchased. Instead all they need to do is just copy positioning information the currently being read ebooks that are stored on the device and memory cards. The way this would work is if the book is located on all devices or programs that are kobo, reading data would be synchronized. We basically do this all the time when back the data base up and reformat the reader and/or a firmware goes rogue or we just don't like the new firmware because they broke something. So as long as the books are located on all kobo devices and programs, yes this means you would have to ensure that you, yourself physically copied all your sideloaded books that you wished were synced to all your devices/programs and thus allowing kobo to identify the position you were in the books so that the information could be synced across.

regards

Jack

[TechniSol]

As I said a while back in the thread, all that Kobo would need to transmit and store would be some unique identifier that would identify the book(scrambled or not), and a page number. The only other requirement might be to transmit a public key if it was necessary to encrypt/decrypt the title. I don't see why Kobo would need to store any side loaded books, only their titles and the internal page number reference that Kobo understands so that Kobo reader on various platforms and with differing font sizes or page sizes would sync accurately.

This would require minimal resources. As far as maintaining the same books across all synced devices, if a book was not present on a particular device there would be no database entry and no way to sync it on that device until the book was transferred or side loaded by the user and a database entry was created.

Again, I was only asking for a page synchronization mechanism, not a syncing of content. I would hardly expect them to devote storage for a great number of ebooks for a great number of users, but I think the penalty of storing the highest page number that any sync so far had yielded along with an identifier would be relatively little burden, likely less than 100 bytes per title.

[Sharkus]

Interesting question about the UniqueID side of things. How would you ensure that this would be unique across multiple devices / apps? What would you base the ID on? file name, hash of file name? something else?
Would you expect the same file, when removed and reinstalled to get the same unique ID?

[PeterT]

Originally Posted by Sharkus

Interesting question about the UniqueID side of things. How would you ensure that this would be unique across multiple devices / apps? What would you base the ID on? file name, hash of file name? something else?
Would you expect the same file, when removed and reinstalled to get the same unique ID?

Ideally the uniqueid needs to be tied to the ePubs itself, possibly an UUID from the metadata itself. Remember we need the id to be consistent wherever the book is installed.

[Sharkus]

Originally Posted by PeterT

Ideally the uniqueid needs to be tied to the ePubs itself, possibly an UUID from the metadata itself. Remember we need the id to be consistent wherever the book is installed.

Exactly. If it's only unique for one app / platform it's somewhat useless to say the least.

That would be the first step, figuring out what to use to get an id that can be used across multiple platforms.

There are other challenges though, for example, how to specify the position in the book? the chapter level, yes, that could be done simply by just getting the path to the current file being read, but what about the position in that file. I know what is done for downloaded books, but that's part of the processing when the book is added to the system, and so it's not something the app / device does, so how to go about doing that?

[speakingtohe]

Probably too simple, but if it is the same title, with the same author, and being used in only Kobo devices/apps would not a transfer of percentage read be close enough?

helen

[robko]

Originally Posted by speakingtohe

Probably too simple, but if it is the same title, with the same author, and being used in only Kobo devices/apps would not a transfer of percentage read be close enough?

helen

I'm not sure "close enough" works in this scenario. Are you going to consider it close enough if it cuts off the 1st few words or sentences of the page you were reading because the device you're on now has a different page size? So you're going to need to be down to at least the word level.

[eschwartz]

Just verify with the file's sha1 checksum. Anything other than that may well be a different(ly formatted) book.

[davidfor]

Originally Posted by Sharkus

Exactly. If it's only unique for one app / platform it's somewhat useless to say the least.

That would be the first step, figuring out what to use to get an id that can be used across multiple platforms.

There are other challenges though, for example, how to specify the position in the book? the chapter level, yes, that could be done simply by just getting the path to the current file being read, but what about the position in that file. I know what is done for downloaded books, but that's part of the processing when the book is added to the system, and so it's not something the app / device does, so how to go about doing that?

The firmware already has the location that could be transferred. The reading position is an identifier for the book plus the location inside the book. The location inside the book for sideloaded epubs uses the Adobe RMSDK method. This is basically the path of the internal file being read plus a walk of the HTML inside the file counting tags and the levels of nesting. Plus the number of characters within the lowest tag. An example is:

OEBPS/Text/Chapter08.xhtml#point(/1/4/2/1:0)

I can already transfer this between my devices using my calibre plugin. This works as long as it is the same epub on both devices. And from a quick look at the database on a Sony, it wouldn't be to hard to move this to that device.

The way Kobo does this for kepubs is pretty horrible. Adding all the spans for the ids is just ripe for breakage somewhere along the way. It is no more robust than the above. I think it would have been better for Kobo to use the RMSDK method, or, better still, the method that is part of the epub3 standard.

[TechniSol]

I don't know how the page number is stored internally with each firmware or app, but there must be a relationship between it and the file, so there should be no great obstacle to translating it to a universal value that can be interpreted on another device. One would assume(look out) that the same algorithms might have been migrated across platforms. If not, translation should be minimal, but requiring a different conversion on each platform.

As for identifying the book I'd suggest that whatever mechanism is employed in the internal database now would be sufficient, but, if not, the filename should be sufficient assuming the customer would have to use the same file on all their sideloaded devices. I'm not asking for content syncing, only page number syncing, and only for exactly the same files, not between kepub and epub formats or anything else...

If the data must be masked prior to transferring it to Kobo's servers, a common conversion or hashing algorithm could be used as long as Kobo never attempts to dehash or decrypt it anywhere but on the user's devices. It should slide in past any privacy requirements. Or if we need to make it complicated the user could enter a code or string used to perform the hashing and be required to enter it on their other devices so Kobo would never have the ability to decrypt the transmitted data as long as it never transmits the code or string used -assuming Kobo doesn't feel the need to employ cryptographic analysis techniques to figure out the hash or encryption key used. Let's be honest, the requirement for the filename of the book to be obscured doesn't have to defy cryptographic analysis as long satisfying privacy is only a minimal concern.

I'm just trying to open the same exact epub to the same "page" or thereabouts on differing devices. Personally, considering the difference in display size between a cell phone and a tablet plus or minus a page or two might be the best that could be done, but all this must have already been worked out to a great degree if it can be done for kepubs across platforms and devices...

[eschwartz]

Originally Posted by TechniSol

I don't know how the page number is stored internally with each firmware or app, but there must be a relationship between it and the file, so there should be no great obstacle to translating it to a universal value that can be interpreted on another device. One would assume(look out) that the same algorithms might have been migrated across platforms. If not, translation should be minimal, but requiring a different conversion on each platform.

As for identifying the book I'd suggest that whatever mechanism is employed in the internal database now would be sufficient, but, if not, the filename should be sufficient assuming the customer would have to use the same file on all their sideloaded devices. I'm not asking for content syncing, only page number syncing, and only for exactly the same files, not between kepub and epub formats or anything else...

If the data must be masked prior to transferring it to Kobo's servers, a common conversion or hashing algorithm could be used as long as Kobo never attempts to dehash or decrypt it anywhere but on the user's devices. It should slide in past any privacy requirements. Or if we need to make it complicated the user could enter a code or string used to perform the hashing and be required to enter it on their other devices so Kobo would never have the ability to decrypt the transmitted data as long as it never transmits the code or string used.

Nah, as long as they could decode it it's going to be a legal problem (unless it's opt-in). And asking people to enter encryption strings is not the way to go. I still say the encryption on the data they sync should be done via public-private keys, with the private key stored on the server encrypted with the Kobo login password. Unless they pull an Adobe, they don't have the password, so it becomes meaningless. And it survives password changes.

[TechniSol]

I have no problem with whatever method is used to obscure the name of the book for privacy concerns, but really could care less about that end of it as several possible methods exist, and so pose no obstacle to accomplishing the simple goal of a page sync.

[shalym]

Originally Posted by eschwartz

Nah, as long as they could decode it it's going to be a legal problem (unless it's opt-in). And asking people to enter encryption strings is not the way to go. I still say the encryption on the data they sync should be done via public-private keys, with the private key stored on the server encrypted with the Kobo login password. Unless they pull an Adobe, they don't have the password, so it becomes meaningless. And it survives password changes.

So why not make it opt-in? Is that how Amazon solved the legal issue?

Shari

[eschwartz]

Originally Posted by shalym

So why not make it opt-in? Is that how Amazon solved the legal issue?

Shari

Amazon has you send the books to their server, and then it's actually stored on their cloud. That's pretty opt-in...

I was theorizing on ways to make it frictionless. Of course, they're never going to do ANY of this....