Currently PDF challenged--help!

[Alfy]

Originally Posted by Xenophon

Many documents are nearly that simple. But PostScript (and its descendent, PDF) are really fully general page description languages. That means, among other things, that any glyph can be placed at any location on the page, in any size, rotation, thickness, etc. And there's a very rich language for computing those locations. In PostScript, it's turing-complete (that is, you can use it to compute anything you can compute in any ordinary programming language). I'm not sure whether PDF is quite that complex. In any case, the simplest PDF files might not be difficult to de-tag and reflow, but add in even a little of that complexity and it gets a whole lot more difficult.

PDF really wasn't originally designed for reflow. It's really a page description language. Period.

Xenophon

P.S. I invite more knowledgeable geeky types to correct any mistakes I've made in this explanation.

Thanks for the additional info.

I went on the ADE forum to ask the question there, and here is the answer I got:

It's not an issue with the 505, this is expected behavior. When you zoom in on PDF, to contents are "reflowed" - essentially stripping a lot of the formating so we can enlarge the font sizes. Because of technical limitations with PDF files (and the current implementation of the reflow algorithms), we do not reflow across pages, so you will get gaps between pages.
Also because of limitations with with the PDF file structure itself (it is not an easily reflowable content), line breaks will appear in odd places.

The bits I find interesting:
- They talk of the "current implementation" as being a cause, so there might be hope for a brighter future.
- They say "we", but what does it mean? I assume it's the PRS software that reflows text, not ADE?

I completely understand PDF weren't originally designed for reflow, but I still find it amazing how difficult it is, sometimes impossible, to get a proper html doc from a PDF. Even Acrobat pro can't get it right 90% of the time...

[pilotbob]

Originally Posted by Alfy

It's not an issue with the 505, this is expected behavior.

This is the typical "working as designed" answer. I've used it myself as a developer. However, this doesn't mean their design is correct. Why did they decide on this behavior. What makes it better? I would ask them that.

BOb

[DaleDe]

Originally Posted by Alfy

But if my understanding of PDF format is correct, it's just text and pictures which are encapsulated using page structures and tags to "fix" the exact look. Unless the document has an exceptionally complicated layout, would it not be easy for the PRS505's software to simply remove the tags to generate a simple flow?

I am asking because I, like many, have been trying unsuccessfully to reflow PDF documents for my sony reader, and my biggest hurdle was removing all the tags and page structures from a 400 pages document and replacing them with the bare minimum to produce an acceptable result (otherwise, even using the "export to html" from acrobat pro, I still got some very weird results). Obviously, I abandoned after a while, but I was wondering with the proper coding skills whether it would be not to difficult to automatize the process? And then even better, to include in the reader's own software?

Why do you believe it is necessary to redo the tags? It is not clear to me that the Sony implementation even uses the tags or needs them. Digital Editions does not.

Dale

[DaleDe]

Originally Posted by Alfy

Thanks for the additional info.

I went on the ADE forum to ask the question there, and here is the answer I got:

It's not an issue with the 505, this is expected behavior. When you zoom in on PDF, to contents are "reflowed" - essentially stripping a lot of the formating so we can enlarge the font sizes. Because of technical limitations with PDF files (and the current implementation of the reflow algorithms), we do not reflow across pages, so you will get gaps between pages.
Also because of limitations with with the PDF file structure itself (it is not an easily reflowable content), line breaks will appear in odd places.

The bits I find interesting:
- They talk of the "current implementation" as being a cause, so there might be hope for a brighter future.
- They say "we", but what does it mean? I assume it's the PRS software that reflows text, not ADE?

I completely understand PDF weren't originally designed for reflow, but I still find it amazing how difficult it is, sometimes impossible, to get a proper html doc from a PDF. Even Acrobat pro can't get it right 90% of the time...

The PRS software was written by the ADE team so they have ownership of it. This is why they say "we". They will need to pull two pages at a time in order to flow past the page boundary. That is doable but they haven't done it. The original PDF group did that for the PocketPC implementation.

PDF is based loosely on Post Script and its ilk PageMaker. A document is really a set of glyphs and a set of locations on the page. It looks to you like a book but it is really an illusion. In many cases a word may not even show up with the letters side by side in the database. If the document has been edited then the data can be in a totally different portion of the file in some cases. It can get very messy. If they PDF was created in one clean shot then it will translate pretty easy but not all PDF files were done that way. The ADE group is new and young and they are working hard I believe but they have a steep learning curve.

Dale

[pilotbob]

Originally Posted by DaleDe

The ADE group is new and young and they are working hard I believe but they have a steep learning curve.

Dale

That's a silly excuse. Adobe is the inventor of PDF and has been creating tools to create/read/edit them for quite a while. Adobe claims they are the best tools on the market. They have more PDF knowledge in house than any other company. If you are saying that they can't leverage that knowledge to create DE then they are doing something wrong. I would bet there are a few people on the DE team that were on the Acrobat team.

BOb

[Alfy]

Originally Posted by pilotbob

That's a silly excuse. Adobe is the inventor of PDF and has been creating tools to create/read/edit them for quite a while. Adobe claims they are the best tools on the market. They have more PDF knowledge in house than any other company. If you are saying that they can't leverage that knowledge to create DE then they are doing something wrong. I would bet there are a few people on the DE team that were on the Acrobat team.

BOb

Actually, I would say the Adobe team as a whole has been quite unable to make the format compatible or transferrable to anything else. I got the Acrobat 9 Pro version recently, and have been meddling around with the export to Word or to html features, and without being broken, it's quite incapable of producing satisfying results. But then, Adobe themselves say it is not their intention, so perhaps the format is indeed THAT incompatible with... Well, just anything else!

By the way, I found one way that seems to be giving good results when converting a PDF into a html: when running ReadIris OCR on a PDF text based document, the result looks quite good. It's a bit ridiculous, considering there's nothing to OCR, but that might prove an effective tool for converting PDFs. It'll do nothing for protected PDFs, obviously.

[DaleDe]

Originally Posted by Alfy

By the way, I found one way that seems to be giving good results when converting a PDF into a html: when running ReadIris OCR on a PDF text based document, the result looks quite good. It's a bit ridiculous, considering there's nothing to OCR, but that might prove an effective tool for converting PDFs. It'll do nothing for protected PDFs, obviously.

Nice find. There is something to OCR and that is exactly why PDF is so tough to translate. The OCR scans the presented screen which solves the problem of the page data being scattered all over the page instead of being sequential.

Dale

[DDHarriman]

Yes.

The best solution so far to revert from PDF looks to see it as it was basic designed to be, an exact reproduction of the paper document it mimics.
So, one just must treat it as a paper document and, if it’s not protected, apply OCR over it.

Omnipage pro 16 and Finereader pro 9 do an outstanding job in this camp.

[Alfy]

MMMhhh... I guess I can make it work, although it requires quite a bit of work. The recognition works fine, and for text with pictures inlined, without wrap around, it should work fine, certainly better than any other conversion process I've tried so far. Alas, both ReadIris and Omnipage html output include arbitrary page breaks to correspond to the PDF version. Anyone here is knowledgeable enough with either of these software to know if there's an option to remove this behaviour?

[DaleDe]

Originally Posted by Alfy

MMMhhh... I guess I can make it work, although it requires quite a bit of work. The recognition works fine, and for text with pictures inlined, without wrap around, it should work fine, certainly better than any other conversion process I've tried so far. Alas, both ReadIris and Omnipage html output include arbitrary page breaks to correspond to the PDF version. Anyone here is knowledgeable enough with either of these software to know if there's an option to remove this behaviour?

Once you have the html it should be easy to correct this behavior. You can just search and replace the br.

Dale

[DDHarriman]

In Omnipage Pro, and choosing the output to Html 3.2, in the screen to save and choosing formatted text, choose option (right side) and check if insert page breaks is activated, if so deactivate it and save again.

Also I advise you to test with the output options to see the ones that give you better results.

[Jim Lester]

Originally Posted by pilotbob

This is the typical "working as designed" answer. I've used it myself as a developer. However, this doesn't mean their design is correct. Why did they decide on this behavior. What makes it better? I would ask them that.

BOb

We do not reflow across pages to reduce the demands for both memory and processor on mobile devices. For instance, in order to get a proper page count for a reflowed PDF (if we were to reflow across pages), the entire PDF would need to be loaded and rendered.

[gwynevans]

Originally Posted by Jim Lester

We do not reflow across pages to reduce the demands for both memory and processor on mobile devices. For instance, in order to get a proper page count for a reflowed PDF (if we were to reflow across pages), the entire PDF would need to be loaded and rendered.

On the other hand, Mobipocket readers seem to manage pretty well with an approximate page count, so maybe that should be dropped & see where you get? Wouldn't that mean that all you need is to do would be to continue to read in the following pages until you'd filled the screen, which seems as if it shouldn't be significantly more than you currently do...

If someone really wants the accurate page count, then they would still be able to use the non-reflowed view to locate it.

[Alfy]

Originally Posted by Jim Lester

We do not reflow across pages to reduce the demands for both memory and processor on mobile devices. For instance, in order to get a proper page count for a reflowed PDF (if we were to reflow across pages), the entire PDF would need to be loaded and rendered.

But when do you need an accurate page count? Yes, there are the table of content and glossaries, but for many novels, those don't even exist. Plus, if it is easy to let the documents reflow at the cost of the page count, then why not make it an option?

ì am not sure how important ebooks are for Adobe and the PDF format, but if the format is to be successful on the current generation of readers (and probably the following ones, a screen the size of a paperback is what people really want in the end), SOME way has to be found to make the docs appear correctly without complex user's intervention like the one I mention above in the thread...

[Alfy]

By the way, Jim Lester is the person from Adobe who nicely answered when I went to their forum to enquire about reflow. Thanks for taking the time to post here!