Creating HUGE ebooks from the 2006 Wikipedia CD Selection

[nrapallo]

I kinda hijacked a thread explode - plucker to html when ranting and raving about my 2006 Wikipedia.pdb (plucker) conversion. This thread was copied from there with the assistance of Nate the great.

EDIT: Some HUGE .prc (37MB), .epub (41MB) and REB1200 .imp (50MB) ebooks made from that 2006 Wikipedia CD Selection from the SOS Children website.

If I exclude images, then the resulting ebook filesizes are: 22 MB REB1200 .imp, 12 MB .epub and 9 MB .prc (high-compression) and may be much more useful. A 15 MB .prc (standard compression) is also available for those readers that cannot handle high (Huff-Dic) compression.

...Try this for more previews of my 2006 Wikipedia work-in-progress ebooks
...Be sure to 'Switch View' or 'Show Descriptions/Tags' on the mediafire.com server to read the vital file info!

Thanks a lot! It worked like a charm using explode!

Also, I had to login (thanks to my Sony Clie TH-55 days) to get your upload at 1src.com. Most won't be so lucky.

So with the author's permission (pruss), I attach explode v0.11 to this thread for others to have access to it.

For years, everyone kept telling me that plucker to .html was not possible; and thanks to you, they are ALL wrong now!

I converted the 43MB Wikipedia 2006.pdb in plucker format quickly as follows:

E:\ebooks> explode --directory=Wikipedia Wikipedia.pdb

I'm converting it now to .imp and from there to .mobi/.prc and .epub, so thanks again for your (much appreciated) update to explode.c.

[nrapallo]

Originally Posted by nrapallo

I'm converting it now to .imp and from there to .mobi/.prc and .epub, so thanks again for your (much appreciated) update to explode.c.

WOW!!!!

It converted on it's first run to my REB1200 .imp format, but the resulting file is TOO huge to be used on my ebook reader. After about 4.5 hours of processing, my conversion software (eBook Publisher) created a 151MB file!

The PC viewer can load it, so I attach some screenshots as the "proof" of concept. Hey, it was fun to even be able to try this.

Now onto round two; decrease the resulting .imp file size by eliminating some or all of the images and/or repetitive text. I would need a 10 fold reduction to make this useable and fear that that may not be feasible. But I WILL try.... — (to boldly go where no man has gone before)

[nrapallo]

Originally Posted by nrapallo

I'm converting it now to .imp and from there to .mobi/.prc and .epub, so thanks again for your (much appreciated) update to explode.c.

OK, I've completed the round one conversions of this to .mobi/.prc and .epub and the results were similar to my previous conversion to .imp.

Using Mobipocket Creator with standard compression resulted in a 138MB .prc whereas using calibre resulted in a 141MB .epub (see some sample ADE screenshots). In all three conversions, the images seem to be stored in full 16M color resolution (as extracted from explode.exe) and the culprit for the tremendous filesize increase. The (compressed) text occupied only about 10% (12 to 15MB) of the resulting ebooks.

Time to focus on image resolution reduction, but I may ( will! ) not probably get ANY better compression result than Plucker's 43MB .pdb, so this is an exercise in futility, but I don't mind...

[nrapallo]

Originally Posted by pruss

You could also reduce the compression accuracy of the jpeg files. I've just posted a new version of explode with a --jpeg-quality=xxx switch (untested). I noticed that the previous version always set the quality to 100 (!). If you set it to 85 or so, you might find a nice improvement, unless of course your further conversion tools themselves reduce the jpeg quality.

Actually, I was thinking of using instead (in place) .gif images with 256-colors (yields about 50% filesize savings) and will see if using a 16-color/grayscale .gif will reduce this any further. Worth a shot!

However, I will also try your new --jpeg-quality switch. "Whatever works best" is my motto...

Thanks again!

[nrapallo]

Using 4-bit .gif improved the filesize of the final ebook a lot. The .imp now is 50 MB, .epub now 43 MB and .prc (stnd) is 44 MB & .prc (high compression) is 37 MB. There was very little image quality loss reducing from 16M colors to 16 colors.

Still too big to be useful.

If I exclude images, then the resulting filesizes are: 23 MB .imp, 14 MB .epub and 16 MB .prc (stnd).
much more useable...

I'll experiment a bit more and see if using the Wikipedia 2006 CD source directly instead of the plucker extracted .html/images works just as good or better.

Stay tuned...

[nrapallo]

Originally Posted by nrapallo

Using 4-bit .gif improved the filesize of the final ebook a lot. The .imp now is 50 MB, .epub now 43 MB and .prc (stnd) is 44 MB & .prc (high compression) is 37 MB. There was very little image quality loss reducing from 16M colors to 16 colors.

Still too big to be useful.

WOW!! WOW!! WOW!! I just tried that 50 MB Wikipedia 2006 .imp file on my REB1200 ebook reader and it works flawlessly!!!! All the links, images and text (though needs to be better formatted) were all there and useable.

I had to use Impserve to load it and that took several minutes, but WOW!! WOW!! WOW!!

I'm a bit excited...

p.s. doesn't work on my EBW1150, though!

[nrapallo]

** deleted by user **

[nrapallo]

Originally Posted by nrapallo

Using 4-bit .gif improved the filesize of the final ebook a lot. The .imp now is 50 MB, .epub now 43 MB and .prc (stnd) is 44 MB & .prc (high compression) is 37 MB. There was very little image quality loss reducing from 16M colors to 16 colors.

Still too big to be useful.

If I exclude images, then the resulting filesizes are: 23 MB .imp, 14 MB .epub and 16 MB .prc (stnd).
much more useable...

I'll experiment a bit more and see if using the Wikipedia 2006 CD source directly instead of the plucker extracted .html/images works just as good or better.

Stay tuned...

Some HUGE .prc, .epub and REB1200 .imp ebooks made from that 2006 Wikipedia CD Selection from the SOS Children website.

If I exclude images, then the resulting ebook filesizes are: 22 MB REB1200 .imp, 12 MB .epub and 9 MB .prc (high-compression) and may be much more useful. A 15 MB .prc (standard compression) is also available for those readers that cannot handle high (Huff-Dic) compression.

Just a test... ...Try this for more previews of my 2006 Wikipedia work-in-progress ebooks ...

[nrapallo]

Originally Posted by nrapallo

Some HUGE .prc, .epub and REB1200 .imp ebooks made from that 2006 Wikipedia CD Selection from the SOS Children website.

If I exclude images, then the resulting ebook filesizes are: 22 MB REB1200 .imp, 12 MB .epub and 9 MB .prc (high-compression) and may be much more useful. A 15 MB .prc (standard compression) is also available for those readers that cannot handle high (Huff-Dic) compression.

Just a test... ...Try this for more previews of my 2006 Wikipedia work-in-progress ebooks ...

I am also converting the original 2006 Wikipedia CD Selection source .html but have only managed to compile the Letter A entires. If you want a sneak peak look here.

This version uses tables and keeps the images at their original size resolution, but still shrinks most of them to 150x150 and 16 colors. I know some are unreadable and I may have to revert to their originals when they are important enough to retain. To that end, I also include a slightly larger version with max 150x??? or ???x150 images AND an even larger version with bigger images (max. 450x??? or ???x450). The latter looks really nice, but the resulting ebook may be almost 80-100MB...

Have fun maxing out your readers!

[cbell]

Gargantuan ebooks... Bring them on!

I tried the prc and epub versions on for size. All loaded surprisingly quickly on my reader. I had varying success with the links. I like having images in an ebook and they looked good.

Nick, I'll send you a more detailed PM.

[nrapallo]

In an effort to reduce the ultimate ebook filesize, I have pre-reduced the colours in the images used by the 2006 Wikipedia to only 8 colours/grayscale and uploaded those ebooks here.

I only did this with the ebook (NEW 2006 Wikipedia CD Selection_letter A_images150) with its images shrunk down to a max. 150x??? or ???x150, but saw a noticable 20% filesize reduction.

Since almost all of the current readers only display 4/8 grayscale (iRex offerings excepted), this may be quite tolerable.

Using just the 'Letter A' entries, I noted the following filesize increases, when including images of different # of max. colours:
- NEW 2006 Wikipedia CD Selection_letter A (original with max. 150x150 16 colour images)
- NEW 2006 Wikipedia CD Selection_letter A_images150-8colors (about 10% incr.)
- NEW 2006 Wikipedia CD Selection_letter A_images150 (about 40% increase)
- NEW 2006 Wikipedia CD Selection_letter A_images450 (about 85% increase)

On a related note, I also uploaded here, a TEST 45MB .epub that contains only 8 colour/grayscale images (whereas all the other ones there have 16 colours) and includes ALL Wikipedia ENTRIES even entries for 'Letter B' to 'Letter Z'. This ebook may not render properly as the .html source included there has not yet been "cleaned up" and may have "glitches" that will be fixed in any released/final ebook version!

It looks like, in the end, a .epub with almost original-sized images (450x???) will be about 2/3rds 5/6ths greater than this 45 MB test .epub, namely could be about 75 MB 83 MB.

Well, I'm off to create this test .epub (~83MB) with these larger sized images.... and then later upload to my Gargantuan ebook server.

EDIT: it actually is 83 MB in size

[ahi]

I'd be curious to find out initial loading, page loading, and other response times for an ePub that contains the Summa Theologica five times over, ideally structured... though for the sake of simplicity, it might be easier not to nest the structuring fully.

[nrapallo]

Originally Posted by nrapallo

It looks like, in the end, a .epub with almost original-sized images (450x???) will be about 2/3rds 5/6rds greater than this 45 MB test .epub, namely could be about 75 MB 83 MB.

Well, I'm off to create this test .epub (~83MB) with these larger sized images.... and then later upload to my Gargantuan ebook server.

EDIT: it actually is 83 MB in size

I've just uploaded here a 80+ MB test ebook (.epub and .prc versions only) with all the Wikipedia entries HOWEVER the Letter B to Z entries have not yet been "cleaned up".

This ebook has at most 450x??? or ???x450 sized images with 16 colours and those images, being approx. 68 MB in total, comprise just over 80% of the ebook's filesize. Now that's a big ebook!

I did notice some quirkiness with the .prc version around the Indices for B to Z and the first entry for the '1906 San Francisco Earthquake', but otherwise it looks very good in the PC software Mobipocket Reader. I'll hopefully fix this later.

Anyone willing to max out their ebook reader and try one of these Gargantuan ebooks? If it survives the ordeal, just post here your experiences...