Apparently, the DR1000S stores the metadata no longer in XML files (like the iLiad did), but in SQLite (v3) databases. Could any of you early adaptors post a scribbled on PDF file and the corresponding metadata.db file (or other .db files, if they appear after scribbling)?

This would allow me to see if the new scribbles format can be reverse engineered easily. Working in a 100% Linux shop and needing to merge scribbles on PDFs, this is a crucial part of my purchasing decision ...

Thanks in advance,
-Thorsten

Try this one. As I use a Mac any Windows software isn't much use to me anyway. I'd like to be able to merge the scribbles & pdfs too. There is only a scribble on the first page.

In order to upload, I have had to rename the metadata.db file to a .txt extension, just change it back to .db

Thanks :2thumbsup:!
There is only a scribble on the first page.
That's consistent with the table annotations in metadata.db (one has to lookup the filename via an index into file_metadata though).

The scribble itself is encoded as 4444 byte of binary data :eek:, which is mostly structured in 12byte records. Why would iRex use this obfuscated approach instead of a table in the datebase :blink:?

Could you describe the scribble (or do you have a photo), so that I can test ideas?

The scribble is just a handwritten script that says "This is a test". Of course I forgot you can't read the scribble on the pdf.
Here's the photo:)

Here's the photo:)And here's the decoded scribble: 17000This is of course still very far from a usable program, because I need to figure out how zoom, rotation etc. are encoded. Also I need to reaquaint myself with java if I want to use the iText library.

Now would be a good time to buy a unit:). Unfortunately, the german distributor has sold his copies and I have to wait:(

Well, your are getting ahead pretty nicely. You may even beat iRex :smack:

I have included a picture of a sketch I made, and the resulting "merged" PDF from the iRex PC application... judge for yourself. Anybody had better luck?

I have also included a zipped folder with the default MeetingNotes.pdf, the 1024x1280 Blankpage.pdf I made and properly "templated" (the one thing that actually works in the PC app) and BlankPage 001.pdf, which contains 5 more or less vertical and 5 horizontal lines, one each for every pen size. (plus the corresponding metadata.db) Maybe that will still help in the decoding. I deleted the .db file just prior to doing the test, so this one should be spot on.

I lost the .db with the sketch when I was playing around with files. Definetely, the drawing contained in the metadata will be applied to simply any PDF with the corresponding filename, even if the content has changed.

Allovertheglobe, can you tell me how you managed to merge the pdf & scribble. That functionality doesn't appear to work for me. Although you can see the menu function of the irex companion software, it is inaccessable in the irex companiuon software as delivered. Do I have to download anything else?

It's a pity you cannot merge in the DR1000s itself.

I have included a picture of a sketch I made

Hey! You'r a quite talented drawer! Very nice and cute. :2thumbsup

But the pdf is awful. I hope they correct this very fast.

Gogolo

Allovertheglobe, can you tell me how you managed to merge the pdf & scribble. That functionality doesn't appear to work for me. Although you can see the menu function of the irex companion software, it is inaccessable in the irex companiuon software as delivered. Do I have to download anything else?

It's a pity you cannot merge in the DR1000s itself.
I'm using the slightly updated version (either from the auto-update, or included in the "Card Content" download from Irex's website)

I just selected the pdf in question, the green "Merge" icon became active, and I clicked it. But it might as well not have worked considering the result...

Curious, I downloaded yet another updated version last night. Just as well I keep checking their website as their inbox doesn't work. I still cannot get the merge to operate.

Was that done in the writepad though? Maybe the merge only works on the writepads & not other pdfs?

I'll have to try again tomorrow when I've got access to a PC.

Curious, I downloaded yet another updated version last night. Just as well I keep checking their website as their inbox doesn't work. I still cannot get the merge to operate.

Was that done in the writepad though? Maybe the merge only works on the writepads & not other pdfs?

I'll have to try again tomorrow when I've got access to a PC.
Good point. I did indeed use a blank page that I created, templated and put in Writepads.

I just tried it now with some other PDF document, and it still "works".

I have included a picture of a sketch I made, and the resulting "merged" PDF from the iRex PC application... judge for yourself. Anybody had better luck?



Nifty... ...This is they type of stuff I would be interested in picking up an iRex for. Though, from the way things sound and from what I've seen thus far- I might be asking too much, too soon.

Still, thanks for the sample sketch.

Well I'm a frustrated little scribbler. I cannot get any of 3 Windows XP machines, nor a Mac running parallels to merge my scribbles into any pdf document. The function is just "greyed out" I can create templates.

OH and the DR1000s draws the line at opening a 36Mb pdf, it just decided enough was enough. Shame. up to 25Mb was okay.

Howard

Upon further investigation:

- The Merge only works for PDF files for now, not images. The reader supports it, even templating, but not the PC Companion.

- The resulting merged PDF seems to have a scaling issue, which may explain the thick lines. The PDF is way too big in any other PDF viewer (the scribble too small relatively speaking) BUT back on the reader it looks the right size (except for the super thick line)

It seems the scribble is stored as coordinates on the metadata.db, and the PC companion replaces every straight line with a black rectangle of the same length but a fixed width, then layers that on top inside the PDF file (which may explain why that is not possible with image files which do not normally support layers (or vectors))

It seems the scribble is stored as coordinates on the metadata.dbThe scribble is a list of strokes and each stroke is a list of points. If I interpret the coordinates of each point as pixels on a 1024x1280 screen and connect them by straight lines, I get a good approximation for the two examples that I have seen so far.

the PC companion replaces every straight line with a black rectangle of the same length but a fixed width,From your example (thanks!), I could figure out where the width of each stroke is stored. The unit is probably again device pixels.

not possible with image files which do not normally support layers (or vectors))That's no excuse: the software could always draw on a bitmap of the image (cf. The Gimp, ImageMagick or Photoshop). It's not completely trivial to make joints and diagonal strokes look nice, but it's not rocket science either.

There are some more bytes that I could not interpret yet (one of them must be the color and there should also be zoom, offset and rotation information, but there's more). I would have to purchase a reader in order to experiment with the settings to decode more of them. Unfortunately, the german distributor is currently sold out:(.

BTW: here's what I know about the format so far (the short integers are store low byte before high byte):
scribble =
{ header : header;
strokes : stroke list }

header =
{ b01_36 : 36bytes;
(* 0x010000000000000000000000000000000000000000000000 000000000004000000050000 *)
num_strokes : 2bytes; (* more likely 16bit short than byte *)
b38_39 : 2bytes (* Always 0x0000? *) }

stroke =
{ stroke_header : stroke_header;
points : point list }

stroke_header =
{ b01 : 1byte; (* Always 0x00? *)
b02_03 : 2bytes; (* Always 3? 16bit short or byte?*)
b04_05 : 2bytes; (* Always 0x0000? *)
b06_09 : 4bytes; (* Always 0xB4276842 or 0x5A5A3143? *)
b10_17 : 8bytes; (* Always 0x0000000000000000? *)
width : 2bytes; (* 16bit short or byte? (the latter would suffice) *)
b20_21 : 2bytes; (* Always 0x0000? *)
num_points : 2bytes; (* more likely 16bit short than byte *)
b24 : 1byte (* Always 0x00? *) }

point =
{ b01 : 1byte; (* Always 0x00? *)
y : 2bytes;
b04_05 : 2bytes; (* Always 0x00? *)
x : 2bytes;
b08_09 : 2bytes; (* Always 0x00? *)
b10_12 : 4bytes (* Always 0x000000 for the first point,
0x88D32F or 0xE8021B for the middle points and
0xF0B115 or 0xB8A215 for the final point? *) }

That's no excuse: the software could always draw on a bitmap of the image (cf. The Gimp, ImageMagick or Photoshop). It's not completely trivial to make joints and diagonal strokes look nice, but it's not rocket science either.
I agree, but you may have noticed that most things DR1000 that Irex has provided so far are very slightly rough around the edges :rolleyes:
So I think they will get this sorted out... eventually. Maybe you can beat them to it! :D

(BTW ImageMagick is one great tool)

OH and the DR1000s draws the line at opening a 36Mb pdf, it just decided enough was enough. Shame. up to 25Mb was okay.

Howard

Seriously?? Well curse. I've been opening pdfs on my iliad that are 40Mb, easy enough - takes a while to load the first time, then just gets on with it.
My DR1000s is due to arrive tomorrow and judging from information like this, let alone the battery-life revelations and scribble-merge issues (does anyone know how to do this on a Mac OS X - the homebrew someone put together for the iliad worked beautifully) it's looking like I may have spent several hundred bucks pointlessly.
All I want is a big-screen iliad! That would do me fine.

Well it might just be that one pdf, however it definitely doesn't want to open on my DR1000s (35.5Mb) It just sits there thinking about it and then crashes back to the menu.
All my other pdfs work, but they are smaller. I'd like to know if your 40Mb pdf does open. If so I'll try re-creating this problem one again.

regards, Howard

I've been opening pdfs on my iliad that are 40Mb, easy enough - takes a while to load the first time, then just gets on with it. Did you add swap to your iLiad (see Swap on iLiad (http://wiki.mobileread.com/wiki/Swap_on_iLiad))? The DR1000S has 128 MB of RAM (vs the iLiad's 64 MB), which should allow larger PDF files. If we get shell access on the DR1000S, then adding swap on the SD card should be straight forward.

I've been opening pdfs on my iliad that are 40Mb, easy enough - takes a while to load the first time, then just gets on with it.Did you add swap to your iLiad (see Swap on iLiad (http://wiki.mobileread.com/wiki/Swap_on_iLiad))? The DR1000S has 128 MB of RAM (vs the iLiad's 64 MB), which should allow larger PDF files. If we get shell access on the DR1000S, then adding swap on the SD card should be straight forward.
Just a suggestion: Maybe this discussion about large PDFs should move into its own top-level thread instead of being buried inside this one about scribbles? I'm sure other people would be interested in this issue too.

Did you add swap to your iLiad (see Swap on iLiad (http://wiki.mobileread.com/wiki/Swap_on_iLiad))? The DR1000S has 128 MB of RAM (vs the iLiad's 64 MB), which should allow larger PDF files. If we get shell access on the DR1000S, then adding swap on the SD card should be straight forward.

Um... Sorry... this is going to sound really pathetic but I don't know if I did or not. I'm very un-techy - when I got the iLiad, I got the 'developer's package' and carefully followed other mobileread members' instructions to install a bunch of extras to optimize it, but that was months ago and I cannot for the life of me remember what I did and didn't do. And I never knew what it was I was doing! I'll be throwing myself on the mercy of those less incompetent than me to do the same to the DR1000s if the chance comes up.

allovertheglobe
New thread started :)

Now, back to scribbles. I still cannot get a merge to work. (It did work once & very well too, on a writepad document I had created but never since on anything) I'm smacking the teddy bear as I type :( I've reinstalled the directory structure on 2 SD cards & tried 3 different PCs.

It is frustrating as I really like the scribble functionality & if Thomega can do anything to help Mac users he has my vote

I have also raised it as an issue with iRex, though I suspect if yourself and others can get it to work, I'm missing something obvious.

Attached is a little python program I cobbled together to do merging on the Mac. Don't have a device to test with at the moment, but it seems to work with the examples posted earlier in this thread. Just run and hit file->merge pdf, then select a PDF. The metadata.db needs to be in the same dir as the PDF. It should generate a <input filename>_merged.pdf

It's very rough around the edges, and will quite likely only work on 10.5 (it may also just fail to work for unexplained reasons :) ). The source is available in the .app if anyone's interested.

If a merge doesn't work right, I can try look into it if I have the pdf and db.

Cheers,
Ross

Hmm, I'm using it in Windows XP on a bootcamp partition on my Mac. I have done it both by running the app straight from the card in a cardreader as well as from the DR1000. Both in the WritePads folder and in my own folders.

It only works with PDFs as far as I can tell, but even when I tried with a PNG image, it saw there was scribble and made the Merge icon active (even though it didn't do anything)

You have certainly gone out of the way trying various things for something that should be straight forward. I'm stumped. The only thing I can think of is that perhaps the matadata.db is corrupted and not read correctly. Did you try deleting it and start over with a fresh scribble (maybe restarting the reader to be extra careful)?

ross,
Many thanks. It works after a fashion. However as you can see, Page 2 of the enclosed doc, for some reason it has rotated the scribble by 90deg. :blink:

I've enclosed the metadata, the merged output and the original pdf. (Change the txt to db for the metadata)

(It's 90deg i other scribbled pdfs too) but thanks :)

allovertheglobe, I have hit the reset button, I've tried in the reader, on the PC on the Mac.

I haven't tried scribbling on any images as I haven't got any on the device but pdfs and scribbling is failing at the merge on mine. I did try deleting all the metadata.db files & doing a card reset from the system menu, still no merge.

If Ross, can get my 90deg tilt fixed , I'll be happy as I could then at least merge from home, even if not at the office.

I've enclosed the metadata, the merged output and the original pdf. (Change the txt to db for the metadata)

(It's 90deg i other scribbled pdfs too) but thanks :)


Ah, turns out I wasn't handling rotation or scaling of the scribbles :) That PDF is also a bit strange, so the scaling doesn't work quite right yet.
I've attached an update version of the app that should work a bit better (at least it gets orientation right).

If someone has the time, I'd appreciate more examples of scribbled PDFs so I can check if I'm doing the scaling and rotating right. A PDF of a grid with scribbled crosses on the grid-intersections would be really helpful. Any other non-working merged scribbles would be useful.

Cheers,
Ross

People can use the ones from my thread:
http://www.mobileread.com/forums/showthread.php?t=30729

I'll post one myself later if I get around to it. (And try your script)

Ross, you are rapidly becoming my hero :) The pdf is a "non-standard" size as it is an instruction leaflet and yes the writing has slipped down a bit in your DR Merge.

As you request, here is a quadranted page with crosses, numbers and some text on it. It is the Quad template from grumpyoldman. Now, as it is a writepad, I can actually merge this in the PC software but pdfs elsewhere, are not seen in the companion software I have.

I enclose the original metadata & Quad template.
The result of the PC merge & the result of your DR Merge.

thank you, Howard

The grid seems to line up now, and the drenalin manual text looks right-ish (I need what seems to be an arbitrary scaling factor - oh well). Also added a progress indicator during merging. Now to figure out how to make the merged PDFs not so ridiculously huge.

Cheers,
Ross

I'm not happy with my scaling "formula", which is just a glorified fudge factor, but a least I win on file size :cool: ...

17181, 17182, 17183

The program is written in O'Caml (http://caml.inria.fr/) and is portable to all machines runnig O'Caml (i.e. all machines :cool:), since it only uses camlpdf (http://www.coherentgraphics.co.uk/camlpdf.html) and ocaml-slqlite3 (http://ocaml.info/home/ocaml_sources.html#ocaml-sqlite3). I will release it under the GPL, as soon as I have received a DR1000S and had the chance to convince myself that the scaling is done right.

(I can give the current sources to people who want to compile it without any handholding ;).)

OK, here it is: 17187.

The Makefile assumes that you have O'Caml installed and that you have wget in your path to get the two required libraries. There is no autoconf yet and no documentation. There are many options missing (color, opacity, time stamps, &c.) and the scaling and rotation can be seriously off -- not to mention that you machine migh spontaneously combust :eek:. You have been warned. Enjoy :D.

I've tried Ross's new version of DR Merge & things now line up. The file size is an interesting one. I wonder if in the companion software the merge then does a further compression to reduce it again?
thanks again everyone.

The file size is an interesting one. I wonder if in the companion software the merge then does a further compression to reduce it again?I don't think so. Ross' uses Quartz drawing primitives for each line segment that are apparently terribly ineffecient (my acroread takes forever to render them). I build the lines using PDF operators in one piece and append it directly to each page. I can save another factor of two by using PDF transforms and the DR's integer pixel coordinates instead of my already scaled floats that are typically twice as long...

you may be correct. If I can figure out how to get your programme to run on the mac I'll give it a go.

The speed of resposne does vary dramatically depending on which pdf reader you use to open them though. In the MAc, acrobat is slow, preview quite quick & likewise the quick view.
On the PC, acrobat isn't too bad.

I am new to this group, but I am very happy other people have the same problems. I am trying a IRex 1000s and seeing the discussion I wonder if IReX wants to sell this product at all!! The idea is great, but if they do not deliver any decent SW to do this things and they do not react to any email I send them, this let me wonder. I really hope you figure out the solution for the merging of PDF files and metadata.db. Thanks a lot in advance for anything you may find out!