extract or export annotation or highlight from personal PDF kindle

jonie · 03-17-2018, 08:33 PM

Hello there.

I am using the kindle to read scientific papers and do a lot of highlights. I intended to use Calibre and the annotations plugin for retrieval of those. I recently realized that highlights in personal pdf files are neither saved to the Myclippings.txt file nor can they be uploaded or bulk shared to my local amazon account (I would prefer a calibre solution anyway).
Every python and java script I found would do the same, extract the myClippings.txt.
However, it seems that the notes and highlights for personal pdfs are saved as pointers (i hope this is the right expression ?) in the .str folder in a .pds or .pdt binary file. Those however are a complete enigma to me.
Does anyone know if it is possible to extract the information with calibre?

The only solution that I could come up with so far would be setting up a facebook account and sharing each highlight individually (= tapping on each highlighted line in each book on the paperwhite) and then try to retrieve all posts from that account with some APU. ... rather discouraging.
Any idea or help would be greatly appreciated.

kovidgoyal · 03-17-2018, 11:50 PM

Basically someone would need to reverse engineer those binary formats.

jonie · 03-18-2018, 05:46 AM

Ok , I guess that is not a trivial task. And the Amazon people probably do not have an interest in helping. Thank you anyway, I guess I have to keep to sharing my notes on facebook or twitter to extract them. Maybe I should try to convert the pdf to an ebook format in the future, but for formulas , figures and formatting I don t think it works very well. (At least it did not last time I tried).

PS: Thanks for the great tool (Calibre). I wish something as thought through and open would exist for science paper-management.

davidfor · 03-18-2018, 10:09 AM

@jonie: I maintain the Annotations plugin. For Kindles, this reads the My Clippings.txt file and put this into a column in calibre for the books. I haven't heard that annotations for PDFs are not in the My Clippings.txt file. That might just mean that the people who use PDFs already know that they don't appear there. I'm not a Kindle user, so I can't test myself.

Kindle does store the annotations in other files. From a little looking a while ago, this would be a better place to get the annotations from. But, either the format isn't known or it encrypted somehow. I can't remember off the top of my head, but my memory was that it wasn't practical to use for this purpose.

jonie · 03-19-2018, 04:08 AM

@davidfor: at least if you do not send the pdf via amazon email they don't. Probably most people convert pdf anyway and researchers jprobably still just print their articles

. I think they use different formats on different kindles and as kovidgoyal said they would have to be reverse engineered.

I found some code that tries the reverse engineering but it dates from 2011 and is for .pdr which is not the format used on my paperwhite.
https://code.google.com/archive/p/ki...kis/Usage.wiki

NanetteK · 06-24-2018, 01:21 AM

@jonie I had an extensive chat with Amazon about this yesterday. They confirmed not syncing annotations from other than Amazon-purchased books (unfortunately).

However, I don't know if you found it, but you can send/download your annotations in an *.html-file.

In the Kindle-app (which I use on Android devices and on my PC), in the book, click on the little 'file'-button in the options on top to get acces to your 'notes'. Than there is a 'share'-icon and you can export & send it to Gmail, Dropbox, Evernote, etc.
(On PC-app it has an upload-button.)
(Sidenote: it uses 'locations' instead of 'pages' but at least you can get the content in one go.)

jonie · 06-27-2018, 03:58 AM

@Nanettek : I guess the option is only present in the apps? At least in my paperwhite I did not find that option. maybe they'll update it someday ....

Or did you find a solution to get annotations from pdf s on your reader that you did not get via kindlestore to the cloud?

In the end I uploaded everything to a twitter accoun setup especially for that purpose and then copied it into a textfile ... at least now I know the annotations by heart . . . . .

NanetteK · 06-29-2018, 01:40 AM

@Jonie Sorry, don't know about Paperwhite but I have to assume it also backs it up to your account which you can then view in the app on your PC.

I've also noticed an update in the app(s) that has to do with notes & flashcards. I haven't taken the time to research yet, but it seems they're developping stuff in that area.

I can't imagine there is no faster/better way than having to create a twitteraccount en go through there; what a work you must have had with that.
Luckily you can see the upside of that (knowing them by heart).

03-17-2018, 08:33 PM	#1
jonie Junior Member Posts: 4 Karma: 10 Join Date: Mar 2018 Device: kindle paperwhite	extract or export annotation or highlight from personal PDF kindle Hello there. I am using the kindle to read scientific papers and do a lot of highlights. I intended to use Calibre and the annotations plugin for retrieval of those. I recently realized that highlights in personal pdf files are neither saved to the Myclippings.txt file nor can they be uploaded or bulk shared to my local amazon account (I would prefer a calibre solution anyway). Every python and java script I found would do the same, extract the myClippings.txt. However, it seems that the notes and highlights for personal pdfs are saved as pointers (i hope this is the right expression ?) in the .str folder in a .pds or .pdt binary file. Those however are a complete enigma to me. Does anyone know if it is possible to extract the information with calibre? The only solution that I could come up with so far would be setting up a facebook account and sharing each highlight individually (= tapping on each highlighted line in each book on the paperwhite) and then try to retrieve all posts from that account with some APU. ... rather discouraging. Any idea or help would be greatly appreciated.

06-24-2018, 01:21 AM	#6
NanetteK Junior Member Posts: 5 Karma: 10 Join Date: Jun 2018 Device: none	@jonie I had an extensive chat with Amazon about this yesterday. They confirmed not syncing annotations from other than Amazon-purchased books (unfortunately). However, I don't know if you found it, but you can send/download your annotations in an .html-file. In the Kindle-app (which I use on Android devices and on my PC), in the book, click on the little 'file'-button in the options on top to get acces to your 'notes'. Than there is a 'share'-icon and you can export & send it to Gmail, Dropbox, Evernote, etc. (On PC-app it has an upload-button.) (Sidenote: it uses 'locations' instead of 'pages' but at least you can get the content in one go.) Last edited by NanetteK; 06-24-2018 at 02:27 AM.*

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Updating books with highlight and annotation	Julius Caesar	Amazon Kindle	6	08-18-2015 10:18 AM
Calibre annotations and highlight extract plugin for the Nook Simple Touch?	einkuser	Plugins	15	01-24-2015 08:13 PM
Does Calibre preservation of annotation, bookmark, highlight work w/ Paperwhite?	Julius Caesar	Amazon Kindle	3	08-26-2013 11:49 AM
Glo Highlight/Annotation compatibility Kobo/Calibre/Adobe Acrobat Pro	kobokobe	Kobo Reader	8	05-19-2013 08:02 PM
PRS-650 prs 650 notes annotation export pdf	bjrnfrdnnd	Sony Reader	8	03-10-2011 12:19 AM

03-17-2018, 11:50 PM	#2
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Basically someone would need to reverse engineer those binary formats.

03-18-2018, 05:46 AM	#3
jonie Junior Member Posts: 4 Karma: 10 Join Date: Mar 2018 Device: kindle paperwhite	Ok , I guess that is not a trivial task. And the Amazon people probably do not have an interest in helping. Thank you anyway, I guess I have to keep to sharing my notes on facebook or twitter to extract them. Maybe I should try to convert the pdf to an ebook format in the future, but for formulas , figures and formatting I don t think it works very well. (At least it did not last time I tried). PS: Thanks for the great tool (Calibre). I wish something as thought through and open would exist for science paper-management.

03-18-2018, 10:09 AM	#4
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	@jonie: I maintain the Annotations plugin. For Kindles, this reads the My Clippings.txt file and put this into a column in calibre for the books. I haven't heard that annotations for PDFs are not in the My Clippings.txt file. That might just mean that the people who use PDFs already know that they don't appear there. I'm not a Kindle user, so I can't test myself. Kindle does store the annotations in other files. From a little looking a while ago, this would be a better place to get the annotations from. But, either the format isn't known or it encrypted somehow. I can't remember off the top of my head, but my memory was that it wasn't practical to use for this purpose.

03-19-2018, 04:08 AM	#5
jonie Junior Member Posts: 4 Karma: 10 Join Date: Mar 2018 Device: kindle paperwhite	@davidfor: at least if you do not send the pdf via amazon email they don't. Probably most people convert pdf anyway and researchers jprobably still just print their articles . I think they use different formats on different kindles and as kovidgoyal said they would have to be reverse engineered. I found some code that tries the reverse engineering but it dates from 2011 and is for .pdr which is not the format used on my paperwhite. https://code.google.com/archive/p/ki...kis/Usage.wiki

06-27-2018, 03:58 AM	#7
jonie Junior Member Posts: 4 Karma: 10 Join Date: Mar 2018 Device: kindle paperwhite	@Nanettek : I guess the option is only present in the apps? At least in my paperwhite I did not find that option. maybe they'll update it someday .... Or did you find a solution to get annotations from pdf s on your reader that you did not get via kindlestore to the cloud? In the end I uploaded everything to a twitter accoun setup especially for that purpose and then copied it into a textfile ... at least now I know the annotations by heart . . . . .

06-29-2018, 01:40 AM	#8
NanetteK Junior Member Posts: 5 Karma: 10 Join Date: Jun 2018 Device: none	@Jonie Sorry, don't know about Paperwhite but I have to assume it also backs it up to your account which you can then view in the app on your PC. I've also noticed an update in the app(s) that has to do with notes & flashcards. I haven't taken the time to research yet, but it seems they're developping stuff in that area. I can't imagine there is no faster/better way than having to create a twitteraccount en go through there; what a work you must have had with that. Luckily you can see the upside of that (knowing them by heart).

Advert

Advert