Sending multiple files from a single book

rpspringuel · 08-29-2014, 02:59 PM

As part of my playing around with apnx files it occurred to me that it would be nice if when you send a book to the device, calibre could be configured to send multiple files from the library, not just the ebook itself.

For instance, when using an ebook purchased from Amazon that comes with an apnx file with real page numbers, it would be nice if I could add that apnx file to my calibre library (in addition to the book itself). Further, when sending that book to the Kindle that apnx file should be sent along with it, instead of calibre calculating a new one.

To implement this my thought was to change the Kindle driver so that when it gets to the point where its ready to calculate the apnx file, it first looks into the library to see if there is apnx file there that it should use. If there is, rather than calculating one, it copies over that file instead. If there isn't, then it calculates an apnx file as normal.

However, upon examining the Kindle driver code, it appears that the function which is in charge of ordering up apnx generation (upload_cover) is not aware of the path to the book in the library (at least in its arguments). The book has already been sent to the device and it is the filename and path for that copy of the book which are used in this process.

Thus, it appears that I would have to modify the USBMS driver (where upload_cover is called) to add an argument specifying the path back to the library entry for the book. Of course, more than the Kindle driver depends on the USBMS driver, so this modification would have cascading effects through the source code for other device drivers (changes I can't check myself because I don't own those devices).

Is there a less intrusive way of getting the library path information in order to check for the apnx file?

Alternatively, and this could be done more generally, would it be possible to tell Calibre that when sending a book to a particular device, it should upload multiple files? This would be a sort of inverse of what currently happens when deleting a book from a device. Upon deleting, Calibre knows to look for and also delete the appropriate companion files on the device. Each driver has the DELETE_EXTS property which contains a list of the file extentions which should be deleted in addition to the book file. A similar LOAD_EXTS property could similarly specify the file extensions of files which are to be loaded to the device along with each book (assuming said files have been loaded into the library).

eschwartz · 08-29-2014, 03:44 PM

This is something I want.

(Will be watching this thread to see what comes of it...)

kovidgoyal · 08-30-2014, 12:03 AM

You absolutely should not be accessing the library in driver code. It runs in a separate thread. While the library API post calibre 1.0 is thread safe, it is still not a good idea.

To implement this would require changing a fair bit of calibre internals, not just the Kindle driver.

rpspringuel · 08-30-2014, 04:56 PM

Okay, then time to brainstorm some possible work arounds.

Right now it is possible to add an apnx file to the library for a book, so that step is taken care of. However, the user has no way of instructing calibre to send this format to the device. The "Send to device" option automatically sends the book itself using the preference list for the driver to determine the format. This should not be messed with, but we still need a way to instruct calibre to send the apnx file.

The "Send specific format to..." option allows the user to pick the format that the book is sent in, but only displays formats which are device compatible. For my kindle that's azw3, mobi, pdf, and txt. I suppose apnx could be added to this list, but I'm not sure this is the best solution as it might imply that apnx was a book format (since it would appear in a list with other book formats).

When a book is displayed, the sidebar has a "Formats" line which lists the apnx file (along with anything else for the book). Right clicking on this entry displays a context menu with "Copy", "Delete the * format", and "Save the * format to disk" as options (where * is the name of the format). Perhaps "Send the * format to device" could be added to this list? This would allow users to send any file in their library to the device. There is, of course, no guaruntee that the device would know what to do with the file, but at least in this particular case it is potentially useful.

eschwartz · 08-30-2014, 11:57 PM

You still need to put the apnx in the right place. On the KK and I think any pre-fw5.x devices, this would match up. For the KT/KPW it must go in the .sdr folder.

tl;dr
apnx as a send-to-device format would be useless without going to considerable trouble by switching send templates constantly.

kovidgoyal · 08-31-2014, 12:19 AM

There is no elegant way to accomplish this short of changing how the send to device mechanism works in calibre. Which is way too much disruption for the use case.

Amazon have recently added support for embedding apnx files inside azw3 in PAGE sections. Hopefully sometime in the future they will update their kindle firmware to be able to read PAGE section as well, at which time this whole issue will go away. You can then create a simple plugin for calibre to embed apnx files into awz3 files.

davidfor · 08-31-2014, 12:25 AM

I haven't looked at it for a while, but doesn't upload_books in the USBM driver.py get the paths to the original books? If so, you could override that, look for an apnx file in that directory and copy that to the appropriate place on the device.

kovidgoyal · 08-31-2014, 12:31 AM

@davidfor: It does not get original paths, remember that sending to device updates metadata, and it does so in a copy of the original file so as to leave the original file undisturbed.

davidfor · 08-31-2014, 12:45 AM

@Kovid: That's what I couldn't remember. Whether the metadata had been updated at this point or if it was done by something called in the loop in this method.

I think what was in my mind was that the Kobo driver needs the path to the cover. It gets the path from the metadata. Something like that might be possible, but whether it is a good idea is another matter.

kovidgoyal · 08-31-2014, 12:48 AM

It's not a good idea, the device drivers must not read from files in the library. Reading from a file locks it in windows. So since the device drivers run in their own thread, if you let the device drivers read from files in the library, you can have random failures dues to locked files while operating on files in the library.

rpspringuel · 09-05-2014, 04:42 PM

Quote:

Originally Posted by kovidgoyal

There is no elegant way to accomplish this short of changing how the send to device mechanism works in calibre. Which is way too much disruption for the use case.

Amazon have recently added support for embedding apnx files inside azw3 in PAGE sections. Hopefully sometime in the future they will update their kindle firmware to be able to read PAGE section as well, at which time this whole issue will go away. You can then create a simple plugin for calibre to embed apnx files into awz3 files.

Well, then I guess I'll just be patient.

08-29-2014, 02:59 PM	#1
rpspringuel Enthusiast Posts: 40 Karma: 10 Join Date: Feb 2014 Device: Kindle 4	Sending multiple files from a single book As part of my playing around with apnx files it occurred to me that it would be nice if when you send a book to the device, calibre could be configured to send multiple files from the library, not just the ebook itself. For instance, when using an ebook purchased from Amazon that comes with an apnx file with real page numbers, it would be nice if I could add that apnx file to my calibre library (in addition to the book itself). Further, when sending that book to the Kindle that apnx file should be sent along with it, instead of calibre calculating a new one. To implement this my thought was to change the Kindle driver so that when it gets to the point where its ready to calculate the apnx file, it first looks into the library to see if there is apnx file there that it should use. If there is, rather than calculating one, it copies over that file instead. If there isn't, then it calculates an apnx file as normal. However, upon examining the Kindle driver code, it appears that the function which is in charge of ordering up apnx generation (upload_cover) is not aware of the path to the book in the library (at least in its arguments). The book has already been sent to the device and it is the filename and path for that copy of the book which are used in this process. Thus, it appears that I would have to modify the USBMS driver (where upload_cover is called) to add an argument specifying the path back to the library entry for the book. Of course, more than the Kindle driver depends on the USBMS driver, so this modification would have cascading effects through the source code for other device drivers (changes I can't check myself because I don't own those devices). Is there a less intrusive way of getting the library path information in order to check for the apnx file? Alternatively, and this could be done more generally, would it be possible to tell Calibre that when sending a book to a particular device, it should upload multiple files? This would be a sort of inverse of what currently happens when deleting a book from a device. Upon deleting, Calibre knows to look for and also delete the appropriate companion files on the device. Each driver has the DELETE_EXTS property which contains a list of the file extentions which should be deleted in addition to the book file. A similar LOAD_EXTS property could similarly specify the file extensions of files which are to be loaded to the device along with each book (assuming said files have been loaded into the library).

08-31-2014, 12:25 AM	#7
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	I haven't looked at it for a while, but doesn't upload_books in the USBM driver.py get the paths to the original books? If so, you could override that, look for an apnx file in that directory and copy that to the appropriate place on the device. Last edited by davidfor; 08-31-2014 at 12:26 AM. Reason: Beaten by Kovid, so I'm probably wrong

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Multiple editions for a single book	Townk	Calibre	3	08-18-2014 03:34 AM
Create single book out of multiple files?	bumpkinboy	Editor	4	04-08-2014 06:41 AM
Sending multiple files to kindle 2	Bonanza43	Calibre	2	07-29-2011 02:07 PM
Converting multiple HTML files into a single hyperlinked PDF?	Jürgen Hubert	Reading and Management	6	01-11-2010 07:44 AM
How do you handle multiple stories in a single book?	Sabardeyn	Calibre	1	06-24-2009 02:42 PM

08-29-2014, 03:44 PM	#2
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	This is something I want. (Will be watching this thread to see what comes of it...)

08-30-2014, 12:03 AM	#3
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You absolutely should not be accessing the library in driver code. It runs in a separate thread. While the library API post calibre 1.0 is thread safe, it is still not a good idea. To implement this would require changing a fair bit of calibre internals, not just the Kindle driver.

08-30-2014, 04:56 PM	#4
rpspringuel Enthusiast Posts: 40 Karma: 10 Join Date: Feb 2014 Device: Kindle 4	Okay, then time to brainstorm some possible work arounds. Right now it is possible to add an apnx file to the library for a book, so that step is taken care of. However, the user has no way of instructing calibre to send this format to the device. The "Send to device" option automatically sends the book itself using the preference list for the driver to determine the format. This should not be messed with, but we still need a way to instruct calibre to send the apnx file. The "Send specific format to..." option allows the user to pick the format that the book is sent in, but only displays formats which are device compatible. For my kindle that's azw3, mobi, pdf, and txt. I suppose apnx could be added to this list, but I'm not sure this is the best solution as it might imply that apnx was a book format (since it would appear in a list with other book formats). When a book is displayed, the sidebar has a "Formats" line which lists the apnx file (along with anything else for the book). Right clicking on this entry displays a context menu with "Copy", "Delete the * format", and "Save the * format to disk" as options (where * is the name of the format). Perhaps "Send the * format to device" could be added to this list? This would allow users to send any file in their library to the device. There is, of course, no guaruntee that the device would know what to do with the file, but at least in this particular case it is potentially useful.

08-30-2014, 11:57 PM	#5
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	You still need to put the apnx in the right place. On the KK and I think any pre-fw5.x devices, this would match up. For the KT/KPW it must go in the .sdr folder. tl;dr apnx as a send-to-device format would be useless without going to considerable trouble by switching send templates constantly.

08-31-2014, 12:19 AM	#6
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	There is no elegant way to accomplish this short of changing how the send to device mechanism works in calibre. Which is way too much disruption for the use case. Amazon have recently added support for embedding apnx files inside azw3 in PAGE sections. Hopefully sometime in the future they will update their kindle firmware to be able to read PAGE section as well, at which time this whole issue will go away. You can then create a simple plugin for calibre to embed apnx files into awz3 files.

08-31-2014, 12:31 AM	#8
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@davidfor: It does not get original paths, remember that sending to device updates metadata, and it does so in a copy of the original file so as to leave the original file undisturbed.

08-31-2014, 12:45 AM	#9
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	@Kovid: That's what I couldn't remember. Whether the metadata had been updated at this point or if it was done by something called in the loop in this method. I think what was in my mind was that the Kobo driver needs the path to the cover. It gets the path from the metadata. Something like that might be possible, but whether it is a good idea is another matter.

08-31-2014, 12:48 AM	#10
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It's not a good idea, the device drivers must not read from files in the library. Reading from a file locks it in windows. So since the device drivers run in their own thread, if you let the device drivers read from files in the library, you can have random failures dues to locked files while operating on files in the library.