EBook-Tools on CPAN

[AZed]

Originally Posted by nrapallo

No worry, I'm happy to contribute, explain, educate, ....

Well, I hope you have access to a better version of the .IMP standard than can be found at http://krausyaoj.tripod.com/reb1200.htm -- I've just run into a brick wall. EBook::Tools can now recognize IMP files and parse the header data, but the table of contents data I'm getting on the test file I'm using downloaded from Fictionwise bears no resemblance at all to the file types in the documentation.

DEBUG: found toc entry 'AXIB' [92 bytes]
DEBUG: found toc entry '    ' [5865 bytes]
DEBUG: found toc entry 'EJWD' [76 bytes]
DEBUG: found toc entry 'GHEJ' [76 bytes]
DEBUG: found toc entry 'GJOB' [484 bytes]
DEBUG: found toc entry 'GJSB' [45 bytes]
DEBUG: found toc entry 'GLMV' [1497 bytes]
DEBUG: found toc entry 'MVGJ' [1392 bytes]
DEBUG: found toc entry 'MVMR' [136 bytes]
DEBUG: found toc entry 'OZSN' [60 bytes]
DEBUG: found toc entry 'SRWH' [64 bytes]
DEBUG: found toc entry 'WPER' [152 bytes]
DEBUG: found toc entry 'YFOX' [1838 bytes]
DEBUG: found toc entry 'YLAB' [458 bytes]
DEBUG: found toc entry 'YVGH' [52 bytes]

The ' ' file is the DATA.FRK text, but that's the only thing that matches, and if it's using some form of LZSS for compression, it's using a very strange version that doesn't use a sliding window, but instead must be sticking all of its dictionary entries into one of those other resource files, because there's no initial ASCII data at all to work with. It's also not ZLib compression, which I checked just in case.

Unless you have some idea of how to proceed, I'm going to end up releasing 0.4.0 with just minimal support for unpacking a .imp file into a REB1200 .RES directory, but no way to actually access the text.

Update: it just occurred to me that they might be adding a ninth bit before every 8 bits of real data, instead of using the 8th bit as a compression flag. This is something of a long shot, but I can at least try to check for it by staring at the beginning of the bitstream 9 bits at a time and see if I can pull out a pattern. Not tonight, though.

[nrapallo]

Originally Posted by AZed

Well, I hope you have access to a better version of the .IMP standard than can be found at http://krausyaoj.tripod.com/reb1200.htm

No, that's all we've got.

However, I've been able to put down my own .imp reverse-engineering knowledge in some perl code you can have, to see what's required to further your understanding. It's an uphill battle that I would like to help with.

-- I've just run into a brick wall. EBook::Tools can now recognize IMP files and parse the header data, but the table of contents data I'm getting on the test file I'm using downloaded from Fictionwise bears no resemblance at all to the file types in the documentation.

DEBUG: found toc entry 'AXIB' [92 bytes]
DEBUG: found toc entry '    ' [5865 bytes]
DEBUG: found toc entry 'EJWD' [76 bytes]
DEBUG: found toc entry 'GHEJ' [76 bytes]
DEBUG: found toc entry 'GJOB' [484 bytes]
DEBUG: found toc entry 'GJSB' [45 bytes]
DEBUG: found toc entry 'GLMV' [1497 bytes]
DEBUG: found toc entry 'MVGJ' [1392 bytes]
DEBUG: found toc entry 'MVMR' [136 bytes]
DEBUG: found toc entry 'OZSN' [60 bytes]
DEBUG: found toc entry 'SRWH' [64 bytes]
DEBUG: found toc entry 'WPER' [152 bytes]
DEBUG: found toc entry 'YFOX' [1838 bytes]
DEBUG: found toc entry 'YLAB' [458 bytes]
DEBUG: found toc entry 'YVGH' [52 bytes]

The .res filename is what you've found above; you need to look at the next four-letter resource filetypes like 'BGcl', 'PNG ', 'Styl', etc.

The ' ' file is the DATA.FRK text, but that's the only thing that matches, and if it's using some form of LZSS for compression, it's using a very strange version that doesn't use a sliding window, but instead must be sticking all of its dictionary entries into one of those other resource files, because there's no initial ASCII data at all to work with. It's also not ZLib compression, which I checked just in case.

If the .imp is compressed then the text will be LZSS compressed using a 14 bit dictionary size with a 3 bit sliding window. I've figured out the algorithm used (using some trial and error 'guessing') and produced a program called deimp.exe in the Fictionwise ebookwise forum. I'll provide you with what I've learned so far.

Unless you have some idea of how to proceed, I'm going to end up releasing 0.4.0 with just minimal support for unpacking a .imp file into a REB1200 .RES directory, but no way to actually access the text.

Update: it just occurred to me that they might be adding a ninth bit before every 8 bits of real data, instead of using the 8th bit as a compression flag. This is something of a long shot, but I can at least try to check for it by staring at the beginning of the bitstream 9 bits at a time and see if I can pull out a pattern. Not tonight, though.

I'm grateful that you are willing to venture into this uncharted terrority. I tried my hand at creating a IMP.pm based on the PDB.pm you include with EBook-Tools. I'm almost finished the header parse routine as well as the resource filetypes separation into "records". I think it best that we collaborate now to avoid anymore hair-pulling...

I'll contact you via PM to see how best to pick up where I've left off.

Thanks again for doing this for me (and the ebookwise community).

[AZed]

Originally Posted by nrapallo

I'm grateful that you are willing to venture into this uncharted terrority. I tried my hand at creating a IMP.pm based on the PDB.pm you include with EBook-Tools. I'm almost finished the header parse routine as well as the resource filetypes separation into "records". I think it best that we collaborate now to avoid anymore hair-pulling...

Oy, yes, especially if you were starting with PalmDoc.pm -- that's a subclass of Palm::PDB that will cause you no end of grief if you attempt to massage it into parsing something that isn't based on PDB records.

I've sent you a reply PM as well.

Thanks again for doing this for me (and the ebookwise community).

My pleasure. I'm glad someone else is finding my code useful.

[nrapallo]

Originally Posted by AZed

Oy, yes, especially if you were starting with PalmDoc.pm -- that's a subclass of Palm::PDB that will cause you no end of grief if you attempt to massage it into parsing something that isn't based on PDB records.

Actually, I think I used an IFF.pm package I found as a starting point as you will soon see with my initial efforts at a IMP.pm package.

I was getting ready use PalmDoc.pm to replace the LZ77 decompress routine with the LZSS decompression routine I discovered and incorporated into deimp.exe until you saved me, er, surprised me that you had started writing IMP.pm.

But remember, go easy on me, I do this all for "fun" not profit!

[nrapallo]

Zed:

BTW, there is some java code, unimp.java, to convert a .imp file into it's .res files written by scotty1024 over in the REB1200 forum in the Yahoo Groups.

I've attached it here to show you how it's done (the program was written July 2002).

[AZed]

The 0.4.2 release, now featuring IMP support, is now on CPAN. IMP files can be unpacked either into .RES directories or HTML, and metadata can be edited directly on the file without unpacking.

Thanks to nrapallo for lots of help putting this together.

[nrapallo]

Originally Posted by AZed

The 0.4.2 release, now featuring IMP support, is now on CPAN. IMP files can be unpacked either into .RES directories or HTML, and metadata can be edited directly on the file without unpacking.

Thanks to nrapallo for lots of help putting this together.

And thanks to you, Zed, for releasing these "tools". It took a while to setup everything so that I could use your ebook.pl, but it was worth it.

I'm off to convert some un-encrypted ereader .pdb ebooks to .epub, now that's a first! Then I'll try some metadata .imp changes. Thanks again!

Are you planning any .epub to .mobi/.pdb/.imp support in your tools?

EDIT: What would be the best way to deal with bugs/comments? I just noticed that your .pdb to .epub allowed the .opf to retain a non-standard media-type in the <manifest> section, e.g. you used media-type="text/plain" instead of media-type="application/xhtml+xml" or media-type="text/x-oeb1-document" for .html files.

[AZed]

Actual bugs or feature requests are best added to the CPAN RT bug tracker, though of course you can always e-mail or PM me at this point as well. Comments can be left here.

The pdb to epub is an odd case — if you don't use the --htmlconvert option (which I just noticed isn't documented properly, and that *is* a bug I'll need to fix), what you're left with really *is* text/plain, and even though it's noncomformant to have that in the OPF file, since it's also nonconformant to have the document not be XHTML, I thought it better to leave it that way as a warning and for consistency.

If you mean that *with* --htmlconvert, unpacking a PDB is still leaving the manifest media-type as text/plain, even though what it points to has been wrapped in html, that's a bug I wasn't aware of.

I wasn't planning to write packers for Mobi/eReader/IMP anytime in the near future, though if there's call for it, I can change my mind. Mobi of course you can use the helper functions on, and unzipping the .epub into a directory and running 'ebook genmobi' on the result should produce what you need if you have the official converter in a findable location.

[nrapallo]

Originally Posted by AZed

Actual bugs or feature requests are best added to the CPAN RT bug tracker, though of course you can always e-mail or PM me at this point as well. Comments can be left here.

The pdb to epub is an odd case — if you don't use the --htmlconvert option (which I just noticed isn't documented properly, and that *is* a bug I'll need to fix), what you're left with really *is* text/plain, and even though it's noncomformant to have that in the OPF file, since it's also nonconformant to have the document not be XHTML, I thought it better to leave it that way as a warning and for consistency.

I didn't use that option, but just unpacked the ereader .pdb and then genepub'ed from the resulting .opf/.html that was unpacked. Now I'm curious if --htmlconvert will fair better.

If you mean that *with* --htmlconvert, unpacking a PDB is still leaving the manifest media-type as text/plain, even though what it points to has been wrapped in html, that's a bug I wasn't aware of.

I wasn't planning to write packers for Mobi/eReader/IMP anytime in the near future, though if there's call for it, I can change my mind. Mobi of course you can use the helper functions on, and unzipping the .epub into a directory and running 'ebook genmobi' on the result should produce what you need if you have the official converter in a findable location.

Fair, but what about more .epub support like metadata editing, "fixing" things within the .epub (which would require an unpack (unzip) and repack routine), etc.

[nrapallo]

Originally Posted by AZed

If you mean that *with* --htmlconvert, unpacking a PDB is still leaving the manifest media-type as text/plain, even though what it points to has been wrapped in html, that's a bug I wasn't aware of.

I converted a free ereader ebook from Fictionwise using this command:

E:\ebooks\EBook-Tools-0.4.2>ebook unpack Verne-80-Days.prc -d My_Book2 --htmlconvert

and got this resulting .opf (see bold part):

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE package PUBLIC "+//ISBN 0-9673008-1-9//DTD OEB 1.2 Package//EN" "http://openebook.org/dtds/oeb-1.2/oebpkg12.dtd">

<package unique-identifier="UUID" xmlns="http://openebook.org/namespaces/oeb-package/1.0/">
  <metadata>
    <dc-metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
      <dc:Identifier scheme="UUID" id="UUID">80F10A5F-6CB4-1014-B40C-9E694293C8DF</dc:Identifier>
      <dc:Title>Around the World in 80 Days</dc:Title>
      <dc:Creator role="aut">Jules Verne</dc:Creator>
      <dc:Subject BASICCode="FIC004000">Classic Literature</dc:Subject>
      <dc:Description>Classic Literature.   62793  words long.</dc:Description>
      <dc:Publisher>Fictionwise, Inc.</dc:Publisher>
      <dc:Type>Novel</dc:Type>
      <dc:Language>en-us</dc:Language>
      <dc:Rights>Copyright &amp;#169;</dc:Rights>
    </dc-metadata>
    <x-metadata>
      <output encoding="utf-8" content-type="text/x-oeb1-document"/>
    </x-metadata>
  </metadata>
  <guide>
    <reference href="Verne-80-Days.html#fpXXXXXXXXXX" title="Table of Contents" type="toc">
    </reference>
  </guide>
  <manifest>
    <item media-type="text/plain" href="Verne-80-Days.html" id="text-main"/>
  </manifest>
  <spine>
    <itemref idref="text-main"/>
  </spine>
</package>

EDIT: I had previously uploaded the Mobipocket .prc version in error and now replace that with the correct ereader. pdb!

[AZed]

I can't replicate this with the file you have provided (though I fixed a related bug when unpacking PalmDoc files, now fixed in svn). I get text/html when I unpack it. Please delete the My_Book2 output directory entirely and try again, to see if you're simply failing to overwrite some older copy somehow.

If you still get this output, please e-mail me a copy of the full output of doing this with the '-vvv' option. My best guess in that case is that mime type detection is broken on your system somehow, but I'll see if I can narrow it down.

[nrapallo]

Originally Posted by AZed

I can't replicate this with the file you have provided (though I fixed a related bug when unpacking PalmDoc files, now fixed in svn). I get text/html when I unpack it. Please delete the My_Book2 output directory entirely and try again, to see if you're simply failing to overwrite some older copy somehow.

I tried it several times, and always got (in the .epub's .opf) the same media-type="text/plain" instead of media-type="application/xhtml+xml" or media-type="text/x-oeb1-document" for .html files.

If you still get this output, please e-mail me a copy of the full output of doing this with the '-vvv' option. My best guess in that case is that mime type detection is broken on your system somehow, but I'll see if I can narrow it down.

I think you are right about the mime-detection being broken! See the Warning I get since I am unable to find a suitable 'shared-mime-info' package for Win XP on freedesktop.org, namely:

WARNING: You don't seem to have a mime-info database.
The shared-mime-info package is available from
http://freedesktop.org/wiki/Software_2fshared_2dmime_2dinfo

I attach the ereader file I used, the log.txt results when using the -vvv option and the unpacked directory (see My_Book6.zip) as well as the .epub created using genepub. Hope this helps narrow down the problem.

Can you create a .epub from my ereader .pdb and post it here with your command line used? Maybe add a switch to help me override mime-detection to produce your results?

[AZed]

From the logs, the mimetype() command isn't working properly on your system, and cross-checking it against my own Windows XP environment shows that it's broken there, too. Ugh. It looks like there are no good mime type detection modules currently available for Perl under Windows.

I'm writing in a mechanism to override autodetection for the next release.

[nrapallo]

Originally Posted by AZed

I'm writing in a mechanism to override autodetection for the next release.

Thanks!

[AZed]

Just noting the 0.4.4 release (I forgot to update this thread for 0.4.3). E-reader mime type handling is working, and a bug involved with running this in CGI is fixed.