Zip spec for ePub

[acts_as_david]

According to the ePub OCF spec, the ePub format should be zipped using the PK Zip compression method, as described in the PKware APPNOTE:

http://www.pkware.com/documents/casestudies/APPNOTE.TXT

The short version of my question: I'm looking for an easy, cross-platform way to accomplish this in a conversion tool I'm writing.

More specifically: I'm designing an ePub library (and tool) for Ruby to handle, manipulate, and create ePub files. Directory and file creation is easy enough, but Zipping them up into an ePub file gets tricky.

Right now, I'm using the following (on OS X, should work in Linux, too):

`zip -X0 Sample.epub mimetype`
`zip -rX9 Sample.epub * -x mimetype`

This is fine and produces a Zip file that looks something like this in my Hex editor:

PK...........;oa
.,............mi
metypeapplicatio
n/epub+zipPK...
......l.;......
..........META-I
NF/PK........%..
;.f......4......
.META-INF/contai
ner.xml

That is, mimetype and 'application/epub+zip' occur at the right offsets, according to the spec.

But no other Zip utility I've found will output this type of Zip file! Not in OS X ('compress') or Linux or Windows (Send to -> Archive). Trying rubyzip and zipruby didn't help. My files never turn out correct using these tools/libraries, and they don't pass epubcheck.

Any suggestions on what library I should use to Zip up my archives?

[pepak]

The most common ZIP library is ZLib.
InfoZip is a command line utility that works on both Windows and Linux and produces the expected output.

[acts_as_david]

Okay, thanks for the suggestions. I think I'll try Zlib first (it's part of the standard Ruby library) and maybe move onto InfoZip if that doesn't work.

One thing I'm not clear on is how many Zip specifications there are? I had thought that any Zip implementation would produce identical Zip files, but as I've said, rubyzip and zipruby didn't.

How can you tell whether an implementation will churn out the right type of file?

[Kolenka]

There is only one ZIP spec, but there are a lot of similar-sounding compression schemes that aren't ZIP (bzip, gzip which don't have an internal file structure, for example, and tend to need you to use the tar format underneath to archive/compress a folder).

One of the other problems you are looking at is that the mimetype needs to be stored rather than compressed in the ZIP archive as the first file. Not all zip libraries even support setting compression for individual files (versus the whole archive).

compress doesn't create ZIP archives, which is a key distinction as well.

[Jellby]

Originally Posted by acts_as_david

Right now, I'm using the following (on OS X, should work in Linux, too):

`zip -X0 Sample.epub mimetype`
`zip -rX9 Sample.epub * -x mimetype`

You may want to add -D to the second line's arguments.

[Valloric]

Originally Posted by acts_as_david

The short version of my question: I'm looking for an easy, cross-platform way to accomplish this in a conversion tool I'm writing.

Take a look at ZipArchive. It has both a commercial license and an open source one (GPL).

It's used in Sigil, and I recommend it highly. Works perfectly across Windows, Linux, Mac OS X and the various BSD's.

If your app will be GPL-compatible, feel free to take a look at the patched ZipArchive used in Sigil: it fixes a few bugs related to python generated ZIP files and improves BSD support.

Originally Posted by pepak

The most common ZIP library is ZLib.
InfoZip is a command line utility that works on both Windows and Linux and produces the expected output.

Zlib is a compression library. It only offers the DEFLATE algorithm used in ZIP, it does *not* understand or provide facilities for the PKZIP format.

You cannot use it (alone) to create a ZIP archive.

[rogue_ronin]

Originally Posted by Jellby

You may want to add -D to the second line's arguments.

Doesn't that eliminate directory structure?

m a r

[acts_as_david]

Originally Posted by Kolenka

There is only one ZIP spec, but there are a lot of similar-sounding compression schemes that aren't ZIP (bzip, gzip which don't have an internal file structure, for example, and tend to need you to use the tar format underneath to archive/compress a folder).

One of the other problems you are looking at is that the mimetype needs to be stored rather than compressed in the ZIP archive as the first file. Not all zip libraries even support setting compression for individual files (versus the whole archive).

compress doesn't create ZIP archives, which is a key distinction as well.

Okay, thanks for the details, Kolenka. You're right that the mimetype file was being compressed when it shouldn't have been, giving me a bad epub file. Thomas Sondergaard, the developer of rubyzip, was kind enough to add compression options to his library, which now does what I want.

Take a look at ZipArchive. It has both a commercial license and an open source one (GPL).

It's used in Sigil, and I recommend it highly. Works perfectly across Windows, Linux, Mac OS X and the various BSD's.

If your app will be GPL-compatible, feel free to take a look at the patched ZipArchive used in Sigil: it fixes a few bugs related to python generated ZIP files and improves BSD support.

Thanks for the suggestion, Valloric, I'll look into ZipArchive and how it's used in Sigil. I'm sure I can get some good ideas from looking at the code.

[Jellby]

Originally Posted by rogue_ronin

Doesn't that eliminate directory structure?

No, it just eliminates the entries for the directories themselves. The files are still stored with their paths. As the man page says, the directory entries are useful when you want to store their permissions, but you don't need the permissions in an ePUB, so you don't need the directory entries.

[rogue_ronin]

I spent an hour or two yesterday trying to figure out how to store a set of files using zip so that the relative path would be correct, and so that it wouldn't store the full path.

-j definitely killed all the directory info; never could find a way to tell zip to store the path info starting from a given directory. Is there such a trick?

I finally had to write a macro that would generate a batch file that would cd to the proper directory and run the zip commands from there.

I'm doing this on a virtual machine of Win2k, although I generally use Linux; the world's greatest text editor requires Windows, sadly...

m a r

[Jellby]

Originally Posted by rogue_ronin

I spent an hour or two yesterday trying to figure out how to store a set of files using zip so that the relative path would be correct, and so that it wouldn't store the full path.

With the zip command, from what I gather from the man page, you can only store relative paths with respect to the current directory. You can also drop any path info (-j) or store absolute paths (-jj).

I'm doing this on a virtual machine of Win2k, although I generally use Linux; the world's greatest text editor requires Windows, sadly...

Hmm... no, vim works fine in linux

[Slash5]

I was playing around in VB trying to create epub files.
The only way I could get it to work was by having a base zip file with mimetype at the right offset and the uncompressed data. I then copied that file to the filename I wanted. I then had to change directory to the files I wanted to add so that pathnames were not included. Then add my files to the canned zipfile.

This was the only reliable way I could get properly formated epubs.

Since I was modifying already created epubs, I eventually decided it was easier to extract the html, modify it and then zip back to the original epub. Solves all the problems.

[rogue_ronin]

@Slash5: me, too, basically!

Originally Posted by Jellby

With the zip command, from what I gather from the man page, you can only store relative paths with respect to the current directory. You can also drop any path info (-j) or store absolute paths (-jj).

That's what I concluded, too. All, or nothing. Still, I worked-around it. But auto-generating, self-executing batch-files seems like overkill, even when it's exactly necessary.

Hmm... no, vim works fine in linux

I said editor, not shreditor.

m a r

ps: Brain shreditor.

[acts_as_david]

Hmm... no, vim works fine in linux

Er... TextMate works just fine in OS X ;-)

Also, running the zip command is working fine for me, re: relative file paths. In other words, running zip (etc., etc.) from within the directory I manually created gives me a proper ePub file. How are you guys running the zip command?

[rogue_ronin]

I was running it as a macro, so it was being called from the current directory, or the root of the drive, which gave me the full path stored in the zip. Not good.

So I had to rewrite the macro as described above to start a new process in the appropriate directory.

m a r