Pocketbook Annotations Extraction

[michailbachtin]

Dear Pocketbook users now using the latest software releases:

WARNING: ANNOTATIONS ARE A PROBLEM

The history:

I loved the format of annotations offered by Toxaris's Annotation Extractor based on the Sony annotations extractor: see the thread here:

https://www.mobileread.com/forums/sho...d.php?t=170051

The Toxaris annotation exporter does the same thing: it extracts the sql from a file entitled books.db on any Kobo or Sony ereader, and the result looks great, as you can see on Toxaris's website:

https://toxaris.nl/en/annotation-exporter/

His thread for annotations at Mobileread.com is here:

https://www.mobileread.com/forums/sho...d.php?t=262913

In previous versions of the pocketbook software, the highlights and annotations for each ebook (whether epub or mobi) were stored in a separate file in the system directory of the device as an html file.

The new software for the Pocketbook series of eReaders has changed recently to the sql format: precisely what the Sony and Kobo eReaders use.

You can guess my next question:

Is there anyone out there developing an annotations extractor for the new Pocketbook software? Or (even better) would it be possible to add to the annotations extractor already available a new configuration for the Pocketbook Touch HD (which uses the same software as the latest Pocketbook Inkpad 2 etc)?

The link to the original code for the annotations extractor is here:
https://www.mobileread.com/forums/sho...d.php?t=170051

The sql code stored on the Pocketbook HD 631 is located in a books.db file.

Here's what it looks like:

http://www55.zippyshare.com/v/FWqtwM6m/file.html

The file is located in the \system\config folder of the Pocketbook ereader.

I suppose anyone working on the problem will also need information for the SD Card in order to get the notes from that.

Could anyone help me? I know there are thousands of Pocketbook users out there who would appreciate it.

Let me know if I can help!

Cheers,

B.

[Toxaris]

I am looking at it, but it does not resemble the databases from Sony and Kobo. That is not surprising, why make it easy...

I *think* I have found the structure, but it will be some time before I can incorporate it into the program. I have a lot of other programming stuff to do for other programs first and this has, at least for me, lower priority (sorry...).

However, I will post my findings on the GitHub page for the annotation program.

[NASCARaddicted]

Thanks, michailbachtin, for pointing out this problem.

In the past I used the PB Touch (I think it is nowadays called Basic Touch), and I loved the annotations and how they were stored. Just copy the html file and everything was fine. I could use them very easily on my computer.

Now I got myself a PB Touch HD and nothing is fine. Whenever I want to use the annotations, I have to copy them manually them from the reader.


and of course, a big thanks to Toxaris, for trying to find a solution. I hope you are successful.

[nhedgehog]

I tried to verify your problem but couldn't.
PB626 latest firmware:
Annotations are stored under:
/System/config/Active Content/ebookfilename_A_blablabla.html

[NASCARaddicted]

Originally Posted by nhedgehog

I tried to verify your problem but couldn't.
PB626 latest firmware:
Annotations are stored under:
/System/config/Active Content/ebookfilename_A_blablabla.html

It's like that on the PB 622, too.
But the PB Touch HD is number 631, and here, the annotations are stored in one db file.

[nhedgehog]

Mmmh thats a Damocles sword hanging over us waiting for the firmware update for our other pocketbook devices. Glad that there is allways the forced update option to go back in firmware.
I had a look into the db file michailbachtin provided. There is no image in it and the entries seem to link to some file in the system. Thats why I doubted that the annotation are only written into this db. But you certainly could prove more infos by opening an ebook, making some text annotations and some scribble ones, document what you have done and where and provide the db. Also there are some filemanagers that let you search for files changed on a specific date. With that all the files on the internal memory that are changed by the annotation could be found.

[Skeeve]

Can someone tell me where the file is? I just created one comment but I'M unable to locate the file where it is stored

[michailbachtin]

I should add that the tech-support I contacted at Pocketbook Inc was surprised by the issue. When I wrote to them in English and Russian, they merely repeated that the Pocketbook Touch HD stored the annotations much like every other Pocketbook device, i.e. as an html in the directory "system/config/active contents". Only when I wrote back to them a third time in Russian did they respond with an acknowledgement that there is currently NO WAY TO COPY ANNOTATIONS from the device to your computer.

Which is why I bought the device in the first place.

I simply do not understand why manufacturers haven't recognized the business model for eReaders specifically for students, academics, and scholars: people who highlight and annotate, but also need to access these highlights and annotations on the devices we use to write about what we've read.

Previously, Pocketbook provided a partial solution. The highlights and notes were accessible, but the format of the notes was (and remains for the other Pocketbook eReaders) incredibly ugly.

Solutions of higher quality have always arisen among ambitious developers who work outside the box. The initial version of the annotation extractor for the Sony eReaders was a beautiful and thoughtful contribution: one that offered a separate html-file for each book featuring parallel columns of highlights next to annotations with approximate page numbers of the corresponding epub file. (You could instantly see the difference between text that was highlighted and text that was your own annotation.) The only drawback on the Sony devices was the ignorance of Sony: they limited the number of exportable highlighted characters to 200 for each annotation. Even on books with no DRM protection.

Toxaris instantly recognized the annotation-extractor created by the legendary "Yoths" in 2012 as the best of its kind in the industry. Of course, genius recognizes genius, and Toxaris also recognized that Sony eReaders were a dying breed: even the last in the Sony series didn't offer a backlit screen. But the Kobo devices did, and they also used a similar file-system for annotations. So Toxaris (a generous spirit of the eBook community) adapted the annotation exporter for Kobos.

So why not just give up on Pocketbook eReaders and buy a Kobo? Well, the Pocketbook eReaders offer far more flexibility in two major fields: The dictionary options for Pocketbook eReaders (if you know what you're doing) are immense, whereas the Kobo dictionary options are rudimentary. Moreover, the TTS modules (Text to Speech) for the Pocketbook eReaders are the best in the industry. (Google Voice does not even come close.) You can choose among any European language (and sometimes several voices for a single language) and practice pronunciation while reading, say, in French, German, Russian, Italian, Spanish, even Welsch!

So (damn it) just when Pocketbook offers, with its Touch HD 631 model, its most mature piece of hardward, with 8GB of internal memory (lots of space for TTS modules) and a mSD card slot for 32GB of books, with, I hasten to add, one of the best screens on the market, what does Pocketbook Inc do? They spoil it all by removing ALL options for annotation and highlight extraction.

We need to tell these manufacturers what we customers need. Otherwise, I see a dark future for Pocketbook Inc. and for other competitors to the Amazon-Kindle ecosystem. The Kindles are so popular that hundreds of amateur programmers have come up with their own annotation support -- and that has evolved into one of the best systems for integrating notes with Calibre, but also with Evernote and with browser apps for Chrome. Moreover, highlighting and annotations are easier to do on a Kindle. (Just try to highlight from one page to the next on a Kobo, for instance. Worse, the latest Pocketbook modus for highlighting is a giant step back from what was offered in earlier versions of the Pocketbook OS.) Trouble is: Kindles limit you to one format and they don't have an mSD, which means you are stuck in Kindle's Amazon cloud for content, with all the potential loss of privacy that entails.

And what kills me (and what has decimated the potential business model of Pocketbook eReaders for serious readers such as students and academics) is that the technical support at Pocketbook Inc. is entirely ignorant of the problem.

As I mentioned, I wrote to them both in English and in Russian about the issue with the Touch HD 631, and received two replies with the false claim that nothing had changed. (At that point, of course, I simply returned my device to the store and counselled my students NOT to buy a Pocketbook. After writing to Pocketbook a third time with screenshots demonstrating that the "active contents" folder with HTML-files containing highlights no longer exists in the OS of the Touch HD, I finally got an acknowledgement that the annotations don't work.

And the response: Maybe, someday, they will...

It makes me wonder who Pocketbook thinks it manufactures for. Yes, they have a good share of the Russian market. But they should think of themselves as a high-end manufacturer, not a cheap manufacturer of one-size-fits-all Kindles (with whom they will never be able to compete).

There ought to be a large market for the Pocketbook Touch HD. There won't be, unless they change.

[nhedgehog]

What I see here is a heck of monologues. Do you answer our questions too?

[rkomar]

I doubt it, since he sent the device back already. That was just for venting his spleen.

[michailbachtin]

To rkomar (above):

Actually, I'm still following this thread closely.

To nhedgehog:

The DB file does not actually contain the annotations. It contains the file names and locations of the annotations from epubs. That system seems to have its advantages. In the Sony and Kobo, you get the page numbers next to the annotations when you extract them with the Toxaris annotation software.

Again to nhedgehog:

There is no "forced update option to go back in firmware" in the Touch HD 631. As NASCARaddicted already noted above, you're in the wrong thread for info about the PB626, a device that uses entirely different note-taking and dictionary functions and that has no Text-to-Speech option.

As for venting spleen (this from a fellow Canadian: rkomar):

I am in regular contact with the developers at Pocketbook international. It helps that I speak Russian (although I'm a Canadian), and that some of the technical staff there read English.

Anna Cherkes of Pocketbook responded (for the third time) two days ago with the following:

"We have forwarded your feedback to the R&D department and the option to copy the notes from the device onto PC will be realized in the future updates for the device."

For those who have not read the messages carefully above: the messages here only concern the Pocketbook HD Touch 631. My purported "venting" includes the message that this device is, from a technical perspective, really the best thing on the market for philologists, except that (and this really nullifies its market value) you cannot extract the highlights and annotations. Furthermore, the technical staff at Pocketbook were seemingly entirely unaware of the issue when I first contacted them.

What I have done is to inform the people at Pocketbook, not only about the problem, but also about how the Toxaris software (for the Kobo and Sony) formats annotations in the most useful manner, and why they should provide a DB format compatible with the Toxaris annotation extractor (or create a downloadable software like it).

Pocketbook promised to fix the issue in January. As of February the 8th, the statement now reads "future updates".

So I'll let everyone know what happens. The very minute that Pocketbook fixes this, I'm buying a Touch HD 631. Technically--a great device.

FOR THOSE WHO WANT TO HELP SOLVE THIS ISSUE: I ENCOURAGE EVERYONE TO WRITE TO ANNE CHERKES AT POCKETBOOK AND REQUEST AN ANNOTATION EXTRACTION SOFTWARE OPTION.

Anna Cherkes <help@pocketbook-int.com>

[jbacelar]

Originally Posted by michailbachtin

FOR THOSE WHO WANT TO HELP SOLVE THIS ISSUE: I ENCOURAGE EVERYONE TO WRITE TO ANNE CHERKES AT POCKETBOOK AND REQUEST AN ANNOTATION EXTRACTION SOFTWARE OPTION.

Anna Cherkes <help@pocketbook-int.com>

Better: annotations and the highlights.

[Toxaris]

Originally Posted by jbacelar

Better: annotations and the highlights.

It is unfortunate that I do not have the time at this moment. The database from the Pocketbook that was provided to me stores the highlight in the database including the location and the annotation made (if any). Unfortunately it is in a different formatting than Sony and Kobo use, otherwise adding support for the Pocketbook was not that difficult.
So, when I get around to it of course both the highlight and the annotation will be exported and you can choose to have the highlight expanded both sides for easier identification. The original highlight will be highlighted in the text.

The sad part is, that I do not expect to have the time for at least 5-6 months.

[jbacelar]

Originally Posted by Toxaris

The sad part is, that I do not expect to have the time for at least 5-6 months.

Thank you very much Toxaris!
Five or six months is much better than ever.

[NASCARaddicted]

FOR THOSE WHO WANT TO HELP SOLVE THIS ISSUE: I ENCOURAGE EVERYONE TO WRITE TO ANNE CHERKES AT POCKETBOOK AND REQUEST AN ANNOTATION EXTRACTION SOFTWARE OPTION.

Thanks for your feedback and your close work with the people at Pocketbook.

But I have one question: is there really a benefit if you use a DB file for the annotations?

That system seems to have its advantages. In the Sony and Kobo, you get the page numbers next to the annotations when you extract them with the Toxaris annotation software.

I don't know if this is really a benefit. The page numbers only makes sense if you use a PC software like the ADE ebook Reader from Adobe. At least if you want to edit them on a computer.

Personally, I would love it, if they would go back to the old html style, like the did on previous devices like the PB622. You don't need any special software to extract anything, just copy the html file to your computer.

However, when I think back, to the old days: my first ebook-Reader (Bebook One) had no touch screen, so you couldn't do any annotations.
Back then, I had to use a special hardware for annotations, called pen and paper. I am glad that this days are over.