Is there a document somewhere that describes the makeup of a Plucker pdb file?

In what ways is it different from other files with the pdb extension? Are there lots of flavors of pdb or just a handful?

Anything that would help clarify this for me (a non-Palm owner) would be greatly appreciated.

Roger Sperberg

Incidentally, I just looked into this. There's a document in the Plucker CVS that describes their file format internally.

http://www.tinbergen.net/DBFormat.html

There's also the PalmDOC format, which I think you want to know which is like

http://homepage.mac.com/pauljlucas/software/txt2pdbdoc/

Then there's also iSilo in pdb format and eReader in Pdb format.

.pdb simply means the Palm DataBase. A lot of applications use this -container- format. Contents, of course, differ.

Thanks! Just the information I was looking for.

You might want to download the DBFormat.html to your local machine, if you haven't yet because I put it online for you as a curtesy. I will remove it soon again. (You can always get it in the cvs)

Thank you. I have done so.

I appreciate your posting it. I'm not familiar enough with the CVS to have gotten it there.

You don't need to know CVS. You can use the web interface to view the latest copy (http://cvs.plkr.org/index.cgi/*checkout*/docs/DBFormat.html?rev=HEAD&content-type=text/html).

I posted the latest version from CVS on my site, Laurens. It is no problem.

But I was unaware of the web client. Nice.

I am curious though, rsperberg, ...what do you need it for?

Because FBReader had implemented Plucker support and aportis doc support (along with html and FB2 and other file formats), I wondered whether the Plucker pdb is similar to other Palm pdb formats.

And along the way I suggested that uBook e-reader add Plucker support, since there's no e-reader distributed that runs on Windows desktops. I think it would be nice to be able to create a Plucker file and then check it on the machine I'm using to make it.

Ah, that explains. Thanks for all the fish.

It would be nice if there were one e-reader that spanned Palm, Windows and Linux (not to mention Mac OS and WinCE/PPC).

Even though David Rothman at Teleread has gone on for quite a while about "the Tower of eBabel," I think the issue of incompatibility is only going to get worse. We're going to see many handheld-size or "ultra-mobile" computers like the Nokia 770 come out, and the majority of these are likely to run Linux.

And I assume you're saying one e-Reader that supports as much formats as possible?

Difficult. pdb's, as you know, have many formats alone. Not to mention the proprietary formats which would be difficult to implement as well. But,...noone is holding you back from making one.

See the Internet Tablet Users blog for Feb 8:
http://www.internettablettalk.com/blog/?p=237

:)

I made the most startling discovery today....

... I never did know you were a member of the openreader group, Roger.

On hindsight, I should have known. Hah.

Well, maybe not. I only joined a week or two ago.

I predicate my involvement on its moving to a formal standards body like OASIS. But that's off-topic here.

As to formats, if you look in the FBReader forum, you will see that I propose expanding beyond the existing formats to allow any arbitrary XML vocabulary.

It's another case of "if browsers can do it, and XML editors can do it, why not our e-readers?"

Haha. Do not worry, I have you firmly fixed in a specific category in my mind. An in all honest, I think OpenReader is a bit of a deadend. It has not made significant progress in a long time.
At any rate, I understand you wish to move to a new XML format that supersedes all the old formats and is much more capable then the rest?

The point of xml is that it's extensible — if you need another element for your content, you add it. And it's self-documenting. All we need an e-reader to do is display it.

And the point of e-books, it seems to me, is that they're electronic. We should expect them to remain books but still do things print books can't do.

So in those regards you are right — but whether that means a new format or just a format and e-reader that permit extensibility and allow electronicity, I couldn't say.

Well, I would say you ought to decide what direction things have to move before starting to peddle. IE: can modifications to an existing format do what you want...or do you need a new format. Incidentally, I still think xhmtl is well suited with a few minor modifications in rendering methods.

That is a good point.

On the other hand, I think every format so far is abysmal — for someone.

And as for XHTML, you're not traveling very far from the roost to choose that. Most every e-book format out there is derived from it or from pre-XML HTML. It's a vocabulary primarily intended for delivering text and yet it isn't rich enough.

And the rich vocabularies — Docbook, TEI — are stuffed to the gills and yet are unsuitable for general use, or even use for people outside the small domains of technical publishing and academia, respectively.

I don't fancy tacking a replacement vocabulary for handling text onto XHTML. And at the same time I recognize that any attempt to specify all the things that need to be marked up in any domain is doomed to failure.

So I say, let us agree on a framework and a small core set of elements and focus on establishing clear requirements for how content should be delivered to the reading system, and what information about the content must accompany it, and on the other hand what exactly the reading system is responsible for.

And this is for books, not the myriad of other media that also must wrestle with the issue of electronicity.

Will OpenReader do a better job of setting us up for the future? I can't say I like the way it's started if that's the task at hand. I think it stands a better chance than Sony, eReader, Mobipocket, Microsoft, IDPF, Plucker, et al., do from where they are positioned.

Mayhap, mayhap. To be fair, is there a need to travel far from the roost. I suppose we both agree that XML is very extensible. Especially HTML. I agree it is not rich enough, yes. Mainly because they are only capable of representing text in the same way is paper books, which is your point all the time, yes?

So here is what I think you ask/say.

* Find a way to deliver -rich- books. Moving images, sound and pictures and complicated markup.
* Find a good manner to deliver this content to the reading device.

That brings in the following extra demands:

* Docbook and such failed because they are -complicated-. The web succeeded with HTML because it is relatively easy.
* The delivery manner must be easily implemented. If you ask me, a simply container format wil do. And the more standard, the better. Again though, difficulty means resistance. It ought be easy.

Am I correct in these two assumptions?

The most important requirement is that the format must have minimal runtime footprint, so as to run under memory-constrained devices. Putting everything in one giant XML stream as FB2 does is not the way to go here. The XML needs to be uncompressed and parsed into an intermediate binary format for the reader to have acceptable rendering performance. In practical terms, this immediately doubles the storage requirements. That alone is a reason not to go with XML. Also, the solution might work reasonably well for fiction-length text, but it scales extremely poorly to anything bigger. Suppose you'd want to access to access the last page of a huge document like an offline Wikipedia. This would mean you'd have to parse all the preceding hundreds or thousands of pages. Without some form of binary index to facilitate random access, this would become extremely slow. (And then you end up with a binary format.)

IMO, a better solution is to have a common XML-based format that can be converted on the desktop to a precompiled binary format optimized for a particular platform with regard to its file structure (PalmOS can only store databases in PDB format in RAM), endianness (PalmOS 68K is big-endian, while ARM-based devices are little-endian) and character encoding (Windows CE supports 16-bit Unicode, while PalmOS only does 8-bit encodings).

True. From a programming point of view, XML is not entirely pleasant to parse. However, contrary to binary format, it -is- extremely easy to create and...I am not entirely in agreement with you. I did not say the stream could not be compressed onto the device. There are very memory constrained compression routines that work great on text. And storage space increases all the time on devices.

What you are suggesting last sounds remarkably much like Sunrise actually, Laurens. That takes HTML and converts it to binary format for reading on PocketPC or palm. But then again, because the program is yours I suppose it fits your vision. Logically.

True. From a programming point of view, XML is not entirely pleasant to parse. However, contrary to binary format, it -is- extremely easy to create and...I am not entirely in agreement with you. I did not say the stream could not be compressed onto the device. There are very memory constrained compression routines that work great on text. And storage space increases all the time on devices.

First off, my post was not exactly a response to you, but more a general comment. (The FB2 format that Roger mentions is a single XML stream that can, optionally, be stored in a ZIP file.)

My point is that a data format has to be geared towards the environment in which it is going to be used. While it's true that decompression routines like ZLIB work fine even under low-memory conditions, random access across a compressed stream is always going to be very slow. It may have acceptable performance for fiction-length works, but it scales poorly to larger documents.

Relying on storage space increasing in future devices is wrongheaded, IMO. There are plenty of low-end devices being released even today. (The Palm Z22 for instance.) Also, in a multitasking OS, multiple processes compete for memory and programmers will always find ways to fill up that memory. (They used to say 640Kb RAM was going to be more than enough for everyone.)

What you are suggesting last sounds remarkably much like Sunrise actually, Laurens. That takes HTML and converts it to binary format for reading on PocketPC or palm. But then again, because the program is yours I suppose it fits your vision. Logically.

Try opening a really large HTML file on a handheld device and notice how it slows down to a crawl. I've yet to see a handheld browser that can render the one-page version of the MySQL manual succesfully. (Pocket IE stops rendering the page about a third of the way through.) Plucker and iSilo manage to render this page completely, exactly because they optimize the document's data structures for use in memory-constrained environments and scale well to larger documents.

True, but let's be fair. ZLib and many compression systems are not geared for random access compression while said binary format you mention -are- specialized, just as you say, for constrained environments.

And while you are right in saying the a data format has to be mindful of the environment it runs in, I very must -dislike- the thought of having to convert any of those files specifically for one of those devices. The idea was very much to get rid of all the different variations. At least, that is what the topic is about I think.

As for the test on the PDA, I will have to try that later. I do not have the PDA on hand here.

And while you are right in saying the a data format has to be mindful of the environment it runs in, I very must -dislike- the thought of having to convert any of those files specifically for one of those devices. The idea was very much to get rid of all the different variations.

The question is whether that is technically possible and feasible.

Remember that PalmOS devices only handle PDB format when storing databases in RAM. Unless you want to force Palm users to store documents on a memory card, you have a problem right there.

This is not my personal crusade for all intent and purposes. I was merely responding to Roger.

I know Palm only supports pdb...and now we're suddenly porting pdb readers to other platforms? Seriously though, all kind of devices are locked into their own format only. I have a rocketbook 1100. Same problem as palm. Only its own format.

And I do not belief every existing device could even completely fit such a new file format. Simply because it would be -one- format and delivery method...left in the middle if this is xml or binary.

The proper location is as follows:

http://cvs.plkr.org/docs/DBFormat.html?view=co

The proper location is as follows:

http://cvs.plkr.org/docs/DBFormat.html?view=co

Thank you.

Roger Sperberg

Hi,
post #8 stated "I think it would be nice to be able to create a Plucker file and then check it on the machine I'm using to make it."

Has there been any developments in the last 2 years that now allow this?

Has there been any developments in the last 2 years that now allow this? FBReader (http://www.fbreader.org/) reads Plucker and now runs under Windows, and under OSX (>=10.4, on Intel).

Hi,
post #8 stated "I think it would be nice to be able to create a Plucker file and then check it on the machine I'm using to make it."

Has there been any developments in the last 2 years that now allow this?
Plucker has included a tool to do this for over 4 years, and it exists in the source and in cvs: The GTK+ Plucker viewer.

Excellent... thanks for the help guys.