Annotation import Kindle PW3

[Quoth]

I thought it should be format

Chapter date
Highlight
Note

However the datestamp part starts with "Unknown" and repeats between Highlight and Note.

Kindle PW3 with most up todate software. Same on mobi or AZW3 books with no DRM.
There is an extra blank paragraph (mentioned in Formatting tool) after each highlight. I'd prefer that after each note and I need Highlight before note as is. For readability (NOT in output in Metadata) I've added an under line at start of each highlight.

Unknown Tue 08 Jan 2019 01:15:41 PM
_ Yet as one focused more and more on

Unknown Tue 08 Jan 2019 01:16:00 PM
Test pw3
Unknown Tue 08 Jan 2019 01:16:20 PM
_ had started innocently enough. He woke

Unknown Tue 08 Jan 2019 01:16:38 PM
Test ch3

<div class="user_annotations" style="margin:0"><div class="annotation" genre="" hash="f0ba67761b40a3b527301ee60a984d39" location_sort="000000" reader="Kindle" style="margin:0 0 0.5em 0"><table cellpadding="0" width="100%" style="background-color:LightGray;color:black;font-size:75%;margin:2.0em;" color="Gray"><tbody><tr><td class="location" style="text-align:left">Unknown</td><td class="timestamp" uts="1546953341.0" style="text-align:right">Tue 08 Jan 2019 01:15:41 PM </td></tr></tbody></table><p class="highlight" style="margin:1.4em;">Yet as one focused more and more on</p><p class="highlight" style="margin:1.4em;"></p></div><div class="annotation" genre="" hash="8bffb64d9758f0893067f83ff91747bc" location_sort="000000" reader="Kindle" style="margin:0 0 0.5em 0"><table cellpadding="0" width="100%" style="background-color:LightGray;color:black;font-size:75%;margin:2.0em;" color="Gray"><tbody><tr><td class="location" style="text-align:left">Unknown</td><td class="timestamp" uts="1546953360.0" style="text-align:right">Tue 08 Jan 2019 01:16:00 PM </td></tr></tbody></table><p class="note" style="font-size:100%;margin:0;">Test pw3</p><p class="note" style="font-size:100%;margin:0;"></p></div><div class="annotation" genre="" hash="d966a37596b7b4642db38101929ebd2e" location_sort="000000" reader="Kindle" style="margin:0 0 0.5em 0"><table cellpadding="0" width="100%" style="background-color:LightGray;color:black;font-size:75%;margin:2.0em;" color="Gray"><tbody><tr><td class="location" style="text-align:left">Unknown</td><td class="timestamp" uts="1546953380.0" style="text-align:right">Tue 08 Jan 2019 01:16:20 PM </td></tr></tbody></table><p class="highlight" style="margin:1.4em;">had started innocently enough. He woke</p><p class="highlight" style="margin:1.4em;"></p></div><div class="annotation" genre="" hash="cf5c0640d7edb453eb4a8547eb5f03e9" location_sort="000000" reader="Kindle" style="margin:0 0 0.5em 0"><table cellpadding="0" width="100%" style="background-color:LightGray;color:black;font-size:75%;margin:2.0em;" color="Gray"><tbody><tr><td class="location" style="text-align:left">Unknown</td><td class="timestamp" uts="1546953398.0" style="text-align:right">Tue 08 Jan 2019 01:16:38 PM </td></tr></tbody></table><p class="note" style="font-size:100%;margin:0;">Test ch3</p><p class="note" style="font-size:100%;margin:0;"></p></div></div>

[davidfor]

How are you doing the annotations import? With the Annotations plugin? Or the experimental method on calibre?

[Quoth]

I have no idea how the "experimental method" is employed.
On Kobo I always used the Kobo Utilites which puts a pop up window. I think it's an Annotations Plug in (it seems to do the same with Kobo H2O as it does with Kindle PW3. It puts the selected (it pops up a window with list of books, doesn't seem to allow book selection in advance) annotation in my custom column, in side pane under cover and in the 2nd custom metadata tab if you go edit metadata.

[davidfor]

Originally Posted by FrustratedReader

I have no idea how the "experimental method" is employed.

It is an option on one of the device related buttons when a device is connected. For supported devices, it finds all annotations and adds them to the comments field. I think only Kindles and Kobo devices are supported.

On Kobo I always used the Kobo Utilites which puts a pop up window. I think it's an Annotations Plug in (it seems to do the same with Kobo H2O as it does with Kindle PW3. It puts the selected (it pops up a window with list of books, doesn't seem to allow book selection in advance) annotation in my custom column, in side pane under cover and in the 2nd custom metadata tab if you go edit metadata.

The Annotations plugin supports both Kindles and Kobo devices. Again, it retrieves all annotations from the device for books in the calibre library. But, once you have the list of books with annotations, you can choose not to import some of them. Where the annotations are put is configurable. As is some of the layout for how of the stored annotations.

For Kindles, the Annotations plugin reads the My Clippings.txt from the device. It parses this to get the details. The file isn't perfect and sometimes doesn't include all the details properly. Or it is possible that the exact layout has changed. I haven't had any reports of problems suggesting that there has been a change.

For your example, the word "Unknown" should be where the location of the annotation in the book. From memory, there are several ways this is stored by Kindles. "Unknown" is used if this is missing or can't be determined. If this isn't the case in the "My Clippings.txt", I need to see the file and possibly a debug log.

For the layout, you can change that in the configuration for the plugin.

[Quoth]

I can't why the location is being repeated between the highlight and the note.
The order is otherwise fine, and I was able to change bold/italic/ indent etc.

I'll see what is in my clippings on Kindle. It seems to do the same bug with the kobo H2O (normally I use the Kobo Utilities)

[Quoth]

The my clippings.txt has the details (It's a useless sort of thing if you are annotating a number of books at once):

Europa's Fairy Book (Joseph Jacobs)
- Your Highlight on page 71 | location 1098-1098 | Added on Friday, 2 November 2018 17:19:07

the King of the Birds whistled loud and shrill, and
==========
Europa's Fairy Book (Joseph Jacobs)
- Your Highlight on page 71 | location 1097-1098 | Added on Friday, 2 November 2018 17:22:25

the King of the Birds, and told him what he wanted. So the King of the Birds whistled loud
==========
Europa's Fairy Book (Joseph Jacobs)
- Your Highlight on page 71 | location 1097-1098 | Added on Friday, 2 November 2018 17:22:42

And after a time he found the King of the Birds, and told him
==========
Europa's Fairy Book (Joseph Jacobs)
- Your Note on page 71 | location 1097 | Added on Friday, 2 November 2018 17:22:59

Tezste
==========
Conspiracies and Rooks (Ray McCarthy)
- Your Highlight on page 9 | location 132-133 | Added on Friday, 2 November 2018 17:23:24

as to what the mysterious letter implied. Why did Alice
==========
Conspiracies and Rooks (Ray McCarthy)
- Your Note on page 9 | location 133 | Added on Friday, 2 November 2018 17:23:41

Test 2
==========
Europa's Fairy Book (Joseph Jacobs)
- Your Note on page 73 | location 1134 | Added on Sunday, 4 November 2018 17:56:22

Illustration
==========
Europa's Fairy Book (Joseph Jacobs)
- Your Highlight on page 73 | location 1134-1134 | Added on Sunday, 4 November 2018 17:56:22


==========
Europa's Fairy Book (Joseph Jacobs)
- Your Highlight on page 73 | location 1134-1134 | Added on Sunday, 4 November 2018 18:04:51


==========
Europa's Fairy Book (Joseph Jacobs)
- Your Highlight on page 73 | location 1134-1134 | Added on Sunday, 4 November 2018 18:04:53


==========
Book of Martyrs (Complete) (John Foxe)
- Your Bookmark on page 82 | location 1254 | Added on Sunday, 4 November 2018 20:14:43


==========
Kindle Paperwhite User’s Guide, 7th Edition (Amazon)
- Your Highlight at location 96-96 | Added on Sunday, 4 November 2018 21:48:53

display above the keyboard. Tap the
==========
Kindle Paperwhite User’s Guide, 7th Edition (Amazon)
- Your Highlight at location 96-96 | Added on Sunday, 4 November 2018 21:48:58

based upon your entries display above the keyboard. Tap the
==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Highlight at location 33-34 | Added on Wednesday, 21 November 2018 20:24:58

set number 6813. In detail, you might find some passingly familiar objects, places, animals and people – rest assured
==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Highlight on page 1 | location 54-54 | Added on Tuesday, 8 January 2019 12:46:01

The sky blossoms fell, each one swirling to its own individual path, shaped and distorted by
==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Note on page 1 | location 54 | Added on Tuesday, 8 January 2019 12:46:36

Test annotation on pw3
==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Highlight on page 1 | location 57-57 | Added on Tuesday, 8 January 2019 12:46:53

Focusing on the individual it seemed to
==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Note on page 1 | location 57 | Added on Tuesday, 8 January 2019 12:47:37

Test 2 pw3
==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Highlight on page 4 | location 107-107 | Added on Tuesday, 8 January 2019 13:15:41

Yet as one focused more and more on
==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Note on page 4 | location 107 | Added on Tuesday, 8 January 2019 13:16:00

Test pw3
==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Highlight on page 37 | location 696-696 | Added on Tuesday, 8 January 2019 13:16:20

had started innocently enough. He woke
==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Note on page 37 | location 696 | Added on Tuesday, 8 January 2019 13:16:38

Test ch3

==========

[Quoth]

All of the above have Unknown and date/time line repeated not just before each highlight, but also before each note, thus separating note and highlight.

I want:
Chapter - datestamp
Highlighted
Note
<blank line>

I'm getting
Unknown - datestamp
Highlight
<blank line>
Unknown - datestamp
Note

[davidfor]

Originally Posted by FrustratedReader

The my clippings.txt has the details (It's a useless sort of thing if you are annotating a number of books at once):

Which is at least partly the point of the plugin: to gather the annotations for a book together.

Europa's Fairy Book (Joseph Jacobs)
- Your Highlight on page 71 | location 1098-1098 | Added on Friday, 2 November 2018 17:19:07

the King of the Birds whistled loud and shrill, and

==========

Well that's different to expected. The plugin is expecting the words "page" and "location" to start in capitals. That is breaking the location info. I have posted a beta with a fix for this in the Annotations plugin thread. I cannot test this as I don't have a Kindle. Try it and tell me how it goes.

[davidfor]

Originally Posted by FrustratedReader

I can't why the location is being repeated between the highlight and the note.

For the example you posted the HTML source, it isn't. That sample has four different timestamps. That looks like different types of highlight, bookmark or annotation.

The order is otherwise fine, and I was able to change bold/italic/ indent etc.

I'll see what is in my clippings on Kindle. It seems to do the same bug with the kobo H2O (normally I use the Kobo Utilities)

Well, it isn't going to be the same bug for the Kobo device. The method the annotations are stored in is completely different. And if you are using the horrible annotations fetch in the Kobo Utilities plugin, the code is also completely unrelated.

Or are you referring to the bug on the device discussed in the Kobo forum? If so, it is even less related.

[Quoth]

Regular source in Text Box in Custom meta data <comments>:
<1st annotation>
<datestamp>Unknown Tue 08 Jan 2019 01:15:41 PM
<Highlight>Yet as one focused more and more on
<blank line as per Annotation Configuration info>
<datestamp>Unknown Tue 08 Jan 2019 01:16:00 PM
<note>Test pw3
<second annotation>
<datestamp>Unknown Tue 08 Jan 2019 01:16:20 PM
<highlight>had started innocently enough. He woke
<blank line as per Annotation Configuration info>
<datestamp>Unknown Tue 08 Jan 2019 01:16:38 PM
<Note>Test ch3


I was only saying that though the Kobo and Kindle are obviously completely different, the bug in the rendering or final processing of the fetched annotation is the same:
1) No location/chapter, only Unknown and Date-Time stamp
2) Repetition of Unknown and Date-Time stamp before "Note" as well as "Source Highlight.

The blank line between Highlight and Note sections seems to be by design. It would make more sense to have it after the Note, which logically is after the Highlight in the default.

I tried the HR option and it added an HR between Highlight and Note Datestamped parts, not just between each overall annotation (Location-Date, Highlight source, entered Note).
I'll have lunch and look at updated Plug in.

Thanks.

[Quoth]

It SHOULD be:
<datestamp><chapter & location info> Tue 08 Jan 2019 01:15:41 PM
<Highlight>Yet as one focused more and more on
<note>Test pw3
<blank line or HR by config>
<second annotation>
<datestamp><chapter & location info> Tue 08 Jan 2019 01:16:20 PM
<highlight>had started innocently enough. He woke
<Note>Test ch3
<blank line or HR by config>

[Quoth]

Hm...
I restarted Calibre twice after adding plugin
I added another annotation (and accidentally a bookmark)
There are three notes. All start with "Test". The first time no notes appeared, only highlights, so I ejected PW3, restarted the second time and reconnected after deleting previous annotations in Calibre.
It's now got location, but that's no use to me, I need Chapter text (and maybe % into it if possible) as I might not have highlighted a phrase unique in the book.
Copied from Custom Metadata tab for book:

Thu 01 Jan 1970 01:00:00 AM
Thu 01 Jan 1970 01:00:00 AM
107 Tue 08 Jan 2019 01:15:41 PM
Yet as one focused more and more on

107 Tue 08 Jan 2019 01:16:00 PM
Test pw3
585 Thu 10 Jan 2019 02:21:27 PM
Test 4
585 Thu 10 Jan 2019 02:21:28 PM
“You’re an orphaned Solaran, from a remote village. You never knew your parents

668 Thu 10 Jan 2019 02:20:19 PM
696 Tue 08 Jan 2019 01:16:20 PM
had started innocently enough. He woke

696 Tue 08 Jan 2019 01:16:38 PM
Test ch3

It still seems to repeat the location & timestamp at the Note.

HERE is the "my clippings.txt" of relevant part:

==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Highlight at location 33-34 | Added on Wednesday, 21 November 2018 20:24:58

set number 6813. In detail, you might find some passingly familiar objects, places, animals and people – rest assured
==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Highlight on page 1 | location 54-54 | Added on Tuesday, 8 January 2019 12:46:01

The sky blossoms fell, each one swirling to its own individual path, shaped and distorted by
==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Note on page 1 | location 54 | Added on Tuesday, 8 January 2019 12:46:36

Test annotation on pw3
==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Highlight on page 1 | location 57-57 | Added on Tuesday, 8 January 2019 12:46:53

Focusing on the individual it seemed to
==========
Of North Blood Drawn (Magen Book 1) (Watterson, C. J.)
- Your Note on page 1 | location 57 | Added on Tuesday, 8 January 2019 12:47:37

Test 2 pw3
==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Highlight on page 4 | location 107-107 | Added on Tuesday, 8 January 2019 13:15:41

Yet as one focused more and more on
==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Note on page 4 | location 107 | Added on Tuesday, 8 January 2019 13:16:00

Test pw3
==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Highlight on page 37 | location 696-696 | Added on Tuesday, 8 January 2019 13:16:20

had started innocently enough. He woke
==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Note on page 37 | location 696 | Added on Tuesday, 8 January 2019 13:16:38

Test ch3

==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Bookmark on page 36 | location 668 | Added on Thursday, 10 January 2019 14:20:19


==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Bookmark on page 36 | location 668 | Added on Thursday, 10 January 2019 14:20:26


==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Bookmark on page 36 | location 668 | Added on Thursday, 10 January 2019 14:20:36


==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Note on page 31 | location 585 | Added on Thursday, 10 January 2019 14:21:27

Test 4
==========
Of North Blood Drawn (Magen) (Ciaran MacUaitéar)
- Your Highlight on page 31 | location 585-586 | Added on Thursday, 10 January 2019 14:21:27

“You’re an orphaned Solaran, from a remote village. You never knew your parents
==========

[Quoth]

Also blank line between Annotations makes sense, between Highlighted Source and Note is not useful.

[Quoth]

I didn't mean to have bookmarks. They seem to have changed GUI to remove them, or I've forgotten!

[davidfor]

Originally Posted by FrustratedReader

Also blank line between Annotations makes sense, between Highlighted Source and Note is not useful.

As I said, you can change the layout of the annotations in the plugins configuration. Fiddling with the CSS should be able to do this. Or just select the option to add a horizontal line between them.