Another bug in m92 1.7 20120726 - serious.

janek · 08-09-2012, 01:43 PM

In 1.7 20120726 merging scribbles and annotations with pdf works erratically. Sometimes it works as normal but sometimes it does not work at all - no output file is produced! It also happened to me that an output file was produced but no scribbles/annotations were merged with it. I tested this with a dozen files I have on my reader and that I had annotated previously to find that about one fifth of them is affected. Exporting annotations to text file does not seem to be affected.

Now, the strangest bit is that the same file, uploaded fresh to m92 with a new file name and annotated anew, can be succesfully merged!
On the other hand, if the filename of the affected document is changed and then the same file with old name is re-uploaded to the device, annotations and scribbles can be seen in the newly uploaded file (they are lost in the old one*) but still cannot be merged.
It follows from this that the bug is related to boox, not to the annotated file.

Since I can't figure what is causing this (database failure? bug in software?) it means annotations/scribbles cannot be relied on anymore. It's probably not just me that they only make sense if they can be transferred to PC. This is really upsetting.

* Changing the file name of an annotated document makes annotations and scribbles disappear - they cannot be seen anymore in the document with a changed name. If the file name is changed back to the original one, annotations and scribbles reappear. I consider this a bug, too.

PF4Mobile · 08-09-2012, 01:48 PM

Is this in the new reader?

janek · 08-09-2012, 01:52 PM

Quote:

Originally Posted by PF4Mobile

Is this in the new reader?

In the old one.

iampiti · 08-09-2012, 02:36 PM

I noticed this too. Since I hadn't used the merge annotations option before (yes, It's true :P) I didn't know this was a bug of this latest firmware.
As Janek said, exporting to text file works correctly.

janek · 08-09-2012, 03:14 PM

BTW, does anyone know where datafiles with user annotations/scribbles are located? They are not in /root or any of its subdirectories (database files therein contain nothing of interest), not in /home, not in /media or /mnt, not in /var. I wanted to peek around to see if there's anything unusual about them that could possibly explain this bug, but I can't find them !

Edit: Oh okay, they are in a hidden directory /media/flash/.onyx.
Be sure to enable hidden files display in otfm to avoid asking silly questions

Edit2: Here's what i've found after finally locating the *.sketch files containing annotations/scribbles:
These files are very simple Sqlite database files, which can be read with any sqlite editor out there. There is one for each pdf file read by the user, and each contains information about both scribbles and annotations. There are just two tables in the database: annotations and scribbles, each consisting of just three columns: id, page position and data, the last of which contains actual scribble information and is of binary type, so cannot be edited in a straightforward way. As much as I could tell by comparing problematic files with unproblematic ones, there's no difference in how these binary fields are composed. I experimented a little with the database of one my "unmergable" files by eliminating individual rows in the annotations table, one by one, and then uploading the database back to boox to see if this changes anything. In this way i could check if it is a particular annotation that is triggering the problem. This worked to some extent, because it turned out that removing all annotations but one indeed allowed me to successfully merge the remaining annotation(s). On the other hand, it seems irrelevant which annotation is left out, as long as there's only one left. This is the bad news. Good news is that scribbles seem to be unaffected by the bug - it only prevents merging the annotations, and scribbles if you also have annotations in your file - so if you dont use annotations, you are less unsafe.

Conclusions:
1. The "unmergable" database files are not corrupt - they can be successfully read with any sqlite editor
2. The binary data in database is not corrupt - since it does not matter which annotaion is left in the file to be merged, it cannot be a problem with binary data.
3. Hence it follows that the problem has to do with merging procedure in the reader, and for that reason,
4. The problem cannot be fixed by a user, at least not in a simple way
5. Still, if you have both scribbles and annotations in your file and need to export them to a pdf file, do the following:
a. using otfm or ftp, go to /media/flash/.onyx/ or /media/sd/.onyx/ directory of your Boox, depending on the location of your pdf file (you need to enable hidden files display first)
b. copy the database file that corresponds to your pdf (the file is named yourfile.pdf.sketch) to your pc
c. do a backup copy of that file
d. make sure you really did a backup copy
e. using sqlite editor such as sqliteman, open the database file
f. go to annotations table and delete all rows in it, then close the transaction (apply changes to database file)
g. copy the file back to the directory in your Boox where you got it from
h. merge your scribbles and mourn your annotations.
i. you can copy your backed-up copy of original sketch file back to Boox so that you can enjoy your annotations on your Boox again while our friends at Onyx work long and indeed hard hours removing the bug <sigh>

Mono · 08-09-2012, 06:38 PM

If I understand it well, everything works fine, if there is any amount of scribles and up to one annotation in the document.

Is that right?

janek · 08-10-2012, 03:09 AM

Quote:

Originally Posted by Mono

If I understand it well, everything works fine, if there is any amount of scribles and up to one annotation in the document.

Is that right?

Yes, this is how it worked with my test files. But clearly, with Boox software you never know... it may still offend your mother or harrass your dog.

jbsavitz · 08-10-2012, 01:29 PM

Hi,

I have an M92 and an i62HD. The M92 has firmware from Onxy Europe and the i62HD has firmware from this forum (20120717) and from what I can tell the firmware on the i62HD is similar to yours.

I've noticed that I'm able to merge annotations into the .pdf file, but have not tried this yet with a lot of .pdf files. I did have one situation where the software would not merge the annotations into the .pdf file and thus did not create a copy of the .pdf file with annotations merged. I haven't tested enough to come to a conclusion.

What I'm curious about is whether you're getting any abnormal terminations when you select the hide margins view and when you do select hide margins if the layout shifts from single page to scrolling pages?

I have your M92 version of firmware, but have not loaded it yet into my M92 for testing.

Please advise.

Mono · 08-10-2012, 01:39 PM

It might explain, why we had different results with scribbling/anotating and switching between old/new pdf reader.

It did work for me (I do not use annotations) and did and did not for you. The whole problem might not be switching back and forth 2 readers, but the general issue in in current pdf reader scribbling/annotation export to pdf...

Of course, one never knows, what else is behind the curtain....

I did not look, if the Onyx scrible/annotation system changed since say March 2012. That time the *.sketch files were placed in .Onyx folder which was in folder where was the document (it was not in system part not normaly accessible to user). And so, it was normaly accessable from PC by any filebrowser the same way like the documents.

If one wanted to backup scribles/annot., the best way was to copy the whole folder structure including .Onyx folders.

I one wanted to move doc, say XXX.pdf, including scribles&ann. - doc file and .Onyx/XXX.sketch file have to be moved/coppied.

janek · 08-10-2012, 02:24 PM

Quote:

Originally Posted by jbsavitz

Hi,
What I'm curious about is whether you're getting any abnormal terminations when you select the hide margins view and when you do select hide margins if the layout shifts from single page to scrolling pages?

I have your M92 version of firmware, but have not loaded it yet into my M92 for testing.

Please advise.

I didn't notice any such effects. Merging annotations is most certainly unrelated to viewing mode.

Quote:

Originally Posted by Mono

It might explain, why we had different results with scribbling/anotating and switching between old/new pdf reader.

It did work for me (I do not use annotations) and did and did not for you. The whole problem might not be switching back and forth 2 readers, but the general issue in in current pdf reader scribbling/annotation export to pdf...

Yes, it may be the same bug.

Quote:

Originally Posted by Mono

I did not look, if the Onyx scrible/annotation system changed since say March 2012. That time the *.sketch files were placed in .Onyx folder which was in folder where was the document (it was not in system part not normaly accessible to user). And so, it was normaly accessable from PC by any filebrowser the same way like the documents.

It didn't. My old scribbles/annotations work out of the boox (pun intended). Also, there's nothing in them that could change - they are indeed very simple.

Edit: Now that i've browsed through past posts in this forum I can see the same bug has already been reported in April. So okay, it's been four months now - it is the time to conclude Onyx is not giving a sh*t about that. Perhaps Zaltys scripts will be more reliable?

Booxtor · 08-13-2012, 04:53 AM

Hi, Janek,

Could you please send me this affected PDF Files (where merging does not work)?
In my colection a could not reproduce this problem.

Thanks!

Knorke · 08-13-2012, 05:29 AM

Hello Janek, hello Booxtor,

I observed by testing with very simple self-made pdf documents a problem with special characters / letters/ signs. At the moment the characters ß, ö, µ are problematic if they occur in the comment text of the annotation.

https://www.mobileread.com/forums/sho...85#post2183185

janek · 08-13-2012, 06:21 AM

Quote:

Originally Posted by Booxtor

Hi, Janek,

Could you please send me this affected PDF Files (where merging does not work)?
In my colection a could not reproduce this problem.

Thanks!

Booxtor, as I wrote, the bug is probably unrelated to the pdf file being annotated. If i annotate the same file again (under a new name), annotations can be merged. Yet if I change file names so that old scribbles appear in the new file - they can't be merged.

Anyway, one of the files that failed to merge is this

Please check your PM to see the *.sketch file that I generated while annotating this very file.

Quote:

Originally Posted by Knorke

Hello Janek, hello Booxtor,

I observed by testing with very simple self-made pdf documents a problem with special characters / letters/ signs. At the moment the characters ß, ö, µ are problematic if they occur in the comment text of the annotation.

https://www.mobileread.com/forums/sho...85#post2183185

It is possible this is yet another bug. I don't use German umlauts in my annotations nor greek letters. I do use non-kosher characters (Polish), but in many files where I use them, I'm still able to merge annotations.

Knorke · 08-13-2012, 06:53 AM

Hi janek,

thank you for your pdf file. It seems to have not so many special characters. Can you upload the exported txt-file of the comments for the working and not working case please? So we could reproduce it in a better way.

Have you used the orginal (marked) text* of the text for the comment or have you put in your own (german) words?

Thanks!

* in the new beta pdf version the marked text is put in into the comment box; in older versions the text field is blank (but if you press 'OK' the marked text is written into the comment)

An additional comment after reading your last edit: I observed the bug never using only characters as a,b,c,d...

iampiti · 08-13-2012, 01:05 PM

This is obviously a very serious bug. As you can see in the post above it also happened to me. If I can do anything to help just tell me.

08-09-2012, 01:43 PM	#1
janek Groupie Posts: 175 Karma: 23456 Join Date: Feb 2012 Device: Boox m92	Another bug in m92 1.7 20120726 - serious. In 1.7 20120726 merging scribbles and annotations with pdf works erratically. Sometimes it works as normal but sometimes it does not work at all - no output file is produced! It also happened to me that an output file was produced but no scribbles/annotations were merged with it. I tested this with a dozen files I have on my reader and that I had annotated previously to find that about one fifth of them is affected. Exporting annotations to text file does not seem to be affected. Now, the strangest bit is that the same file, uploaded fresh to m92 with a new file name and annotated anew, can be succesfully merged! On the other hand, if the filename of the affected document is changed and then the same file with old name is re-uploaded to the device, annotations and scribbles can be seen in the newly uploaded file (they are lost in the old one) but still cannot be merged. It follows from this that the bug is related to boox, not to the annotated file. Since I can't figure what is causing this (database failure? bug in software?) it means annotations/scribbles cannot be relied on anymore. It's probably not just me that they only make sense if they can be transferred to PC. This is really upsetting. Changing the file name of an annotated document makes annotations and scribbles disappear - they cannot be seen anymore in the document with a changed name. If the file name is changed back to the original one, annotations and scribbles reappear. I consider this a bug, too.

08-09-2012, 03:14 PM	#5
janek Groupie Posts: 175 Karma: 23456 Join Date: Feb 2012 Device: Boox m92	BTW, does anyone know where datafiles with user annotations/scribbles are located? They are not in /root or any of its subdirectories (database files therein contain nothing of interest), not in /home, not in /media or /mnt, not in /var. I wanted to peek around to see if there's anything unusual about them that could possibly explain this bug, but I can't find them ! Edit: Oh okay, they are in a hidden directory /media/flash/.onyx. Be sure to enable hidden files display in otfm to avoid asking silly questions Edit2: Here's what i've found after finally locating the .sketch files containing annotations/scribbles: These files are very simple Sqlite database files, which can be read with any sqlite editor out there. There is one for each pdf file read by the user, and each contains information about both scribbles and annotations. There are just two tables in the database: annotations and scribbles, each consisting of just three columns: id, page position and data, the last of which contains actual scribble information and is of binary type, so cannot be edited in a straightforward way. As much as I could tell by comparing problematic files with unproblematic ones, there's no difference in how these binary fields are composed. I experimented a little with the database of one my "unmergable" files by eliminating individual rows in the annotations table, one by one, and then uploading the database back to boox to see if this changes anything. In this way i could check if it is a particular annotation that is triggering the problem. This worked to some extent, because it turned out that removing all annotations but one indeed allowed me to successfully merge the remaining annotation(s). On the other hand, it seems irrelevant which annotation is left out, as long as there's only one left. This is the bad news. Good news is that scribbles seem to be unaffected by the bug - it only prevents merging the annotations, and scribbles if you also have annotations in your file - so if you dont use annotations, you are less unsafe. Conclusions: 1. The "unmergable" database files are not corrupt - they can be successfully read with any sqlite editor 2. The binary data in database is not corrupt - since it does not matter which annotaion is left in the file to be merged, it cannot be a problem with binary data. 3. Hence it follows that the problem has to do with merging procedure in the reader, and for that reason, 4. The problem cannot be fixed by a user, at least not in a simple way 5. Still, if you have both scribbles and annotations in your file and need to export them to a pdf file, do the following: a. using otfm or ftp, go to /media/flash/.onyx/ or /media/sd/.onyx/ directory of your Boox, depending on the location of your pdf file (you need to enable hidden files display first) b. copy the database file that corresponds to your pdf (the file is named yourfile.pdf.sketch) to your pc c. do a backup copy of that file d. make sure you really did a backup copy e. using sqlite editor such as sqliteman, open the database file f. go to annotations table and delete all rows in it, then close the transaction (apply changes to database file) g. copy the file back to the directory in your Boox where you got it from h. merge your scribbles and mourn your annotations. i. you can copy your backed-up copy of original sketch file back to Boox so that you can enjoy your annotations on your Boox again while our friends at Onyx work long and indeed hard hours removing the bug <sigh> Last edited by janek; 08-09-2012 at 05:47 PM.*

08-10-2012, 01:29 PM	#8
jbsavitz Connoisseur Posts: 73 Karma: 530904 Join Date: Apr 2011 Device: Sony PRS-950, Onyx Boox M92 and i62HD	Hi, I have an M92 and an i62HD. The M92 has firmware from Onxy Europe and the i62HD has firmware from this forum (20120717) and from what I can tell the firmware on the i62HD is similar to yours. I've noticed that I'm able to merge annotations into the .pdf file, but have not tried this yet with a lot of .pdf files. I did have one situation where the software would not merge the annotations into the .pdf file and thus did not create a copy of the .pdf file with annotations merged. I haven't tested enough to come to a conclusion. What I'm curious about is whether you're getting any abnormal terminations when you select the hide margins view and when you do select hide margins if the layout shifts from single page to scrolling pages? I have your M92 version of firmware, but have not loaded it yet into my M92 for testing. Please advise. Last edited by jbsavitz; 08-10-2012 at 01:33 PM.

08-13-2012, 06:53 AM	#14
Knorke Enthusiast Posts: 33 Karma: 66 Join Date: Aug 2012 Device: Onyx Boox M92	Hi janek, thank you for your pdf file. It seems to have not so many special characters. Can you upload the exported txt-file of the comments for the working and not working case please? So we could reproduce it in a better way. Have you used the orginal (marked) text* of the text for the comment or have you put in your own (german) words? Thanks! * in the new beta pdf version the marked text is put in into the comment box; in older versions the text field is blank (but if you press 'OK' the marked text is written into the comment) An additional comment after reading your last edit: I observed the bug never using only characters as a,b,c,d... Last edited by Knorke; 08-20-2012 at 04:06 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Bug: M92 - Annotations in PDF disappearing!	Defenderland	Onyx Boox	15	05-11-2013 03:35 PM
M92 scribling bug?	Mono	Onyx Boox	2	03-08-2012 07:33 PM
Boox M92. Spring Bargains. The price of M92 stays reduced for 14 days :)	Booxtor	Onyx Boox	5	03-02-2012 05:16 AM
Embedded font bug or CSS bug in ADE	JSWolf	ePub	10	06-11-2011 02:34 PM
PRS-505 bug or eBookLib bug?	porkupan	Sony Reader	3	10-07-2007 10:44 PM

08-09-2012, 01:48 PM	#2
PF4Mobile Guru Posts: 629 Karma: 3526 Join Date: Jun 2011 Device: Kobo Touch, Nook Touch, EEE 800 Note, Entourage PE, finally M92	Is this in the new reader?

08-09-2012, 02:36 PM	#4
iampiti Connoisseur Posts: 69 Karma: 57510 Join Date: Feb 2012 Device: Onyx M92	I noticed this too. Since I hadn't used the merge annotations option before (yes, It's true :P) I didn't know this was a bug of this latest firmware. As Janek said, exporting to text file works correctly.

08-09-2012, 06:38 PM	#6
Mono Mono Posts: 699 Karma: 13333 Join Date: Jan 2012 Device: Boox M92	If I understand it well, everything works fine, if there is any amount of scribles and up to one annotation in the document. Is that right?

08-10-2012, 01:39 PM	#9
Mono Mono Posts: 699 Karma: 13333 Join Date: Jan 2012 Device: Boox M92	It might explain, why we had different results with scribbling/anotating and switching between old/new pdf reader. It did work for me (I do not use annotations) and did and did not for you. The whole problem might not be switching back and forth 2 readers, but the general issue in in current pdf reader scribbling/annotation export to pdf... Of course, one never knows, what else is behind the curtain.... I did not look, if the Onyx scrible/annotation system changed since say March 2012. That time the *.sketch files were placed in .Onyx folder which was in folder where was the document (it was not in system part not normaly accessible to user). And so, it was normaly accessable from PC by any filebrowser the same way like the documents. If one wanted to backup scribles/annot., the best way was to copy the whole folder structure including .Onyx folders. I one wanted to move doc, say XXX.pdf, including scribles&ann. - doc file and .Onyx/XXX.sketch file have to be moved/coppied.

08-13-2012, 04:53 AM	#11
Booxtor Booxtor Posts: 1,126 Karma: 2305664 Join Date: Jun 2011 Location: Germany Device: a lot of..	Hi, Janek, Could you please send me this affected PDF Files (where merging does not work)? In my colection a could not reproduce this problem. Thanks!

08-13-2012, 05:29 AM	#12
Knorke Enthusiast Posts: 33 Karma: 66 Join Date: Aug 2012 Device: Onyx Boox M92	Hello Janek, hello Booxtor, I observed by testing with very simple self-made pdf documents a problem with special characters / letters/ signs. At the moment the characters ß, ö, µ are problematic if they occur in the comment text of the annotation. https://www.mobileread.com/forums/sho...85#post2183185

08-13-2012, 01:05 PM	#15
iampiti Connoisseur Posts: 69 Karma: 57510 Join Date: Feb 2012 Device: Onyx M92	This is obviously a very serious bug. As you can see in the post above it also happened to me. If I can do anything to help just tell me.