Creating HUGE ebooks from the 2006 Wikipedia CD Selection

nrapallo · 04-08-2009, 11:38 AM

I kinda hijacked a thread explode - plucker to html when ranting and raving about my 2006 Wikipedia.pdb (plucker) conversion. This thread was copied from there with the assistance of Nate the great.

EDIT: Some HUGE .prc (37MB), .epub (41MB) and REB1200 .imp (50MB) ebooks made from that 2006 Wikipedia CD Selection from the SOS Children website.

If I exclude images, then the resulting ebook filesizes are: 22 MB REB1200 .imp, 12 MB .epub and 9 MB .prc (high-compression) and may be much more useful. A 15 MB .prc (standard compression) is also available for those readers that cannot handle high (Huff-Dic) compression.

...Try this for more previews of my 2006 Wikipedia work-in-progress ebooks
...Be sure to 'Switch View' or 'Show Descriptions/Tags' on the mediafire.com server to read the vital file info!

Thanks a lot! It worked like a charm using explode!

Also, I had to login (thanks to my Sony Clie TH-55 days) to get your upload at 1src.com. Most won't be so lucky.

So with the author's permission (pruss), I attach explode v0.11 to this thread for others to have access to it.

For years, everyone kept telling me that plucker to .html was not possible; and thanks to you, they are ALL wrong now!

I converted the 43MB Wikipedia 2006.pdb in plucker format quickly as follows:

Code:

E:\ebooks> explode --directory=Wikipedia Wikipedia.pdb

I'm converting it now to .imp and from there to .mobi/.prc and .epub, so thanks again for your (much appreciated) update to explode.c.

nrapallo · 04-08-2009, 11:50 AM

Quote:

Originally Posted by nrapallo

I'm converting it now to .imp and from there to .mobi/.prc and .epub, so thanks again for your (much appreciated) update to explode.c.

WOW!!!!

It converted on it's first run to my REB1200 .imp format, but the resulting file is TOO huge to be used on my ebook reader. After about 4.5 hours of processing, my conversion software (eBook Publisher) created a 151MB file!

The PC viewer can load it, so I attach some screenshots as the "proof" of concept. Hey, it was fun to even be able to try this.

Now onto round two; decrease the resulting .imp file size by eliminating some or all of the images and/or repetitive text. I would need a 10 fold reduction to make this useable and fear that that may not be feasible. But I WILL try.... — (to boldly go where no man has gone before)

nrapallo · 04-08-2009, 11:52 AM

Quote:

Originally Posted by nrapallo

I'm converting it now to .imp and from there to .mobi/.prc and .epub, so thanks again for your (much appreciated) update to explode.c.

OK, I've completed the round one conversions of this to .mobi/.prc and .epub and the results were similar to my previous conversion to .imp.

Using Mobipocket Creator with standard compression resulted in a 138MB .prc whereas using calibre resulted in a 141MB .epub (see some sample ADE screenshots). In all three conversions, the images seem to be stored in full 16M color resolution (as extracted from explode.exe) and the culprit for the tremendous filesize increase. The (compressed) text occupied only about 10% (12 to 15MB) of the resulting ebooks.

Time to focus on image resolution reduction, but I may ( will! ) not probably get ANY better compression result than Plucker's 43MB .pdb, so this is an exercise in futility, but I don't mind...

nrapallo · 04-08-2009, 11:53 AM

Quote:

Originally Posted by pruss

You could also reduce the compression accuracy of the jpeg files. I've just posted a new version of explode with a --jpeg-quality=xxx switch (untested). I noticed that the previous version always set the quality to 100 (!). If you set it to 85 or so, you might find a nice improvement, unless of course your further conversion tools themselves reduce the jpeg quality.

Actually, I was thinking of using instead (in place) .gif images with 256-colors (yields about 50% filesize savings) and will see if using a 16-color/grayscale .gif will reduce this any further. Worth a shot!

However, I will also try your new --jpeg-quality switch. "Whatever works best" is my motto...

Thanks again!

nrapallo · 04-08-2009, 12:00 PM

Using 4-bit .gif improved the filesize of the final ebook a lot. The .imp now is 50 MB, .epub now 43 MB and .prc (stnd) is 44 MB & .prc (high compression) is 37 MB. There was very little image quality loss reducing from 16M colors to 16 colors.

Still too big to be useful.

If I exclude images, then the resulting filesizes are: 23 MB .imp, 14 MB .epub and 16 MB .prc (stnd).

much more useable...

I'll experiment a bit more and see if using the Wikipedia 2006 CD source directly instead of the plucker extracted .html/images works just as good or better.

Stay tuned...

nrapallo · 04-09-2009, 09:20 AM

Quote:

Originally Posted by nrapallo

Using 4-bit .gif improved the filesize of the final ebook a lot. The .imp now is 50 MB, .epub now 43 MB and .prc (stnd) is 44 MB & .prc (high compression) is 37 MB. There was very little image quality loss reducing from 16M colors to 16 colors.

Still too big to be useful.

WOW!! WOW!! WOW!! I just tried that 50 MB Wikipedia 2006 .imp file on my REB1200 ebook reader and it works flawlessly!!!! All the links, images and text (though needs to be better formatted) were all there and useable.

I had to use Impserve to load it and that took several minutes, but WOW!! WOW!! WOW!!

I'm a bit excited...

p.s. doesn't work on my EBW1150, though!

nrapallo · 04-10-2009, 11:04 AM

** deleted by user **

nrapallo · 08-15-2009, 12:34 PM

Quote:

Originally Posted by nrapallo

Using 4-bit .gif improved the filesize of the final ebook a lot. The .imp now is 50 MB, .epub now 43 MB and .prc (stnd) is 44 MB & .prc (high compression) is 37 MB. There was very little image quality loss reducing from 16M colors to 16 colors.

Still too big to be useful.

If I exclude images, then the resulting filesizes are: 23 MB .imp, 14 MB .epub and 16 MB .prc (stnd).

much more useable...

I'll experiment a bit more and see if using the Wikipedia 2006 CD source directly instead of the plucker extracted .html/images works just as good or better.

Stay tuned...

Some HUGE .prc, .epub and REB1200 .imp ebooks made from that 2006 Wikipedia CD Selection from the SOS Children website.

If I exclude images, then the resulting ebook filesizes are: 22 MB REB1200 .imp, 12 MB .epub and 9 MB .prc (high-compression) and may be much more useful. A 15 MB .prc (standard compression) is also available for those readers that cannot handle high (Huff-Dic) compression.

Just a test...

...Try this for more previews of my 2006 Wikipedia work-in-progress ebooks ...

nrapallo · 08-20-2009, 04:40 PM

Quote:

Originally Posted by nrapallo

Some HUGE .prc, .epub and REB1200 .imp ebooks made from that 2006 Wikipedia CD Selection from the SOS Children website.

If I exclude images, then the resulting ebook filesizes are: 22 MB REB1200 .imp, 12 MB .epub and 9 MB .prc (high-compression) and may be much more useful. A 15 MB .prc (standard compression) is also available for those readers that cannot handle high (Huff-Dic) compression.

Just a test...

...Try this for more previews of my 2006 Wikipedia work-in-progress ebooks ...

I am also converting the original 2006 Wikipedia CD Selection source .html but have only managed to compile the Letter A entires. If you want a sneak peak look here.

This version uses tables and keeps the images at their original size resolution, but still shrinks most of them to 150x150 and 16 colors. I know some are unreadable and I may have to revert to their originals when they are important enough to retain. To that end, I also include a slightly larger version with max 150x??? or ???x150 images AND an even larger version with bigger images (max. 450x??? or ???x450). The latter looks really nice, but the resulting ebook may be almost 80-100MB...

Have fun maxing out your readers!

cbell · 08-21-2009, 08:01 AM

Gargantuan ebooks... Bring them on!

I tried the prc and epub versions on for size. All loaded surprisingly quickly on my reader. I had varying success with the links. I like having images in an ebook and they looked good.

Nick, I'll send you a more detailed PM.

nrapallo · 08-22-2009, 07:59 AM

In an effort to reduce the ultimate ebook filesize, I have pre-reduced the colours in the images used by the 2006 Wikipedia to only 8 colours/grayscale and uploaded those ebooks here.

I only did this with the ebook (NEW 2006 Wikipedia CD Selection_letter A_images150) with its images shrunk down to a max. 150x??? or ???x150, but saw a noticable 20% filesize reduction.

Since almost all of the current readers only display 4/8 grayscale (iRex offerings excepted), this may be quite tolerable.

Using just the 'Letter A' entries, I noted the following filesize increases, when including images of different # of max. colours:
- NEW 2006 Wikipedia CD Selection_letter A (original with max. 150x150 16 colour images)
- NEW 2006 Wikipedia CD Selection_letter A_images150-8colors (about 10% incr.)
- NEW 2006 Wikipedia CD Selection_letter A_images150 (about 40% increase)
- NEW 2006 Wikipedia CD Selection_letter A_images450 (about 85% increase)

On a related note, I also uploaded here, a TEST 45MB .epub that contains only 8 colour/grayscale images (whereas all the other ones there have 16 colours) and includes ALL Wikipedia ENTRIES even entries for 'Letter B' to 'Letter Z'. This ebook may not render properly as the .html source included there has not yet been "cleaned up" and may have "glitches" that will be fixed in any released/final ebook version!

It looks like, in the end, a .epub with almost original-sized images (450x???) will be about 2/3rds 5/6ths greater than this 45 MB test .epub, namely could be about 75 MB 83 MB.

Well, I'm off to create this test .epub (~83MB) with these larger sized images.... and then later upload to my Gargantuan ebook server.

EDIT: it actually is 83 MB in size

ahi · 08-22-2009, 10:43 AM

I'd be curious to find out initial loading, page loading, and other response times for an ePub that contains the Summa Theologica five times over, ideally structured... though for the sake of simplicity, it might be easier not to nest the structuring fully.

nrapallo · 08-24-2009, 12:07 PM

Quote:

Originally Posted by nrapallo

It looks like, in the end, a .epub with almost original-sized images (450x???) will be about 2/3rds 5/6rds greater than this 45 MB test .epub, namely could be about 75 MB 83 MB.

Well, I'm off to create this test .epub (~83MB) with these larger sized images.... and then later upload to my Gargantuan ebook server.

EDIT: it actually is 83 MB in size

I've just uploaded here a 80+ MB test ebook (.epub and .prc versions only) with all the Wikipedia entries HOWEVER the Letter B to Z entries have not yet been "cleaned up".

This ebook has at most 450x??? or ???x450 sized images with 16 colours and those images, being approx. 68 MB in total, comprise just over 80% of the ebook's filesize. Now that's a big ebook!

I did notice some quirkiness with the .prc version around the Indices for B to Z and the first entry for the '1906 San Francisco Earthquake', but otherwise it looks very good in the PC software Mobipocket Reader. I'll hopefully fix this later.

Anyone willing to max out their ebook reader and try one of these Gargantuan ebooks? If it survives the ordeal, just post here your experiences...

nrapallo · 07-13-2010, 04:46 PM

Well, it's been almost a year since I created these HUGE ebooks and would like to update you regarding some usage/download stats, namely:

Code:

Wikipedia_2006 Folder at nrapallo's Gargantuan ebook server (see http://www.mediafire.com/nrapallo  )									
									
File Name							Ebook Size			Downloads			Comments										
2006 Wikipedia CD Selection_1200.imp			49.01 MB			3 Downloads			works on REB1200 without issue!	
2006 Wikipedia CD Selection_noimages_1150.imp		18.84 MB			2 Downloads			"crashes on ebookwise 1150; sorry, will not work!"	
2006 Wikipedia CD Selection_noimages_1200.imp		21.65 MB			1 Downloads			works on REB1200 without issue!	
									
Wikipedia_2006.epub						40.29 MB			55 Downloads				
Wikipedia_2006.prc						36.20 MB			30 Downloads			uses high-compression otherwise would have been 44MB	
									
Wikipedia_2006_noimages.epub				12.30 MB			31 Downloads				
Wikipedia_2006_noimages.prc				8.77 MB			23 Downloads			uses high-compression otherwise would have been 15MB	
Wikipedia_2006_noimages_stnd_compression.prc		15.06 MB			5 Downloads			uses standard compression	
									
Summary Statisctics from nrapallo's Gargantuan ebook server (see http://www.mediafire.com/nrapallo )								
								
Total Storage Used		Total Downloads Served						Est. Bandwidth Served
576.39 MB			202									5.14 GB
From 30 total files		"Since August 15, 2009"						"up to July 13, 2010"

or given that "a picture is worth a thousand words"...

Click image for larger version

Name: nrapallo-Gargantuan ebook server-Summary Stats.jpg
Views: 766
Size: 88.0 KB
ID: 55141

Long live Gargantuan ebooks!!!!

HarryT · 07-14-2010, 01:46 PM

I'm rather a fan of seriously big books myself, Nick

. Keep up the good work!

04-10-2009, 11:04 AM	#7
nrapallo GuteBook/Mobi2IMP Creator Posts: 2,958 Karma: 2530691 Join Date: Dec 2007 Location: Toronto, Canada Device: REB1200 EBW1150 Device: T1 NSTG iLiad_v2 NC Device: Asus_TF Next1 WPDN	deleted by user Last edited by nrapallo; 08-20-2009 at 07:30 PM.

08-22-2009, 07:59 AM	#11
nrapallo GuteBook/Mobi2IMP Creator Posts: 2,958 Karma: 2530691 Join Date: Dec 2007 Location: Toronto, Canada Device: REB1200 EBW1150 Device: T1 NSTG iLiad_v2 NC Device: Asus_TF Next1 WPDN	In an effort to reduce the ultimate ebook filesize, I have pre-reduced the colours in the images used by the 2006 Wikipedia to only 8 colours/grayscale and uploaded those ebooks here. I only did this with the ebook (NEW 2006 Wikipedia CD Selection_letter A_images150) with its images shrunk down to a max. 150x??? or ???x150, but saw a noticable 20% filesize reduction. Since almost all of the current readers only display 4/8 grayscale (iRex offerings excepted), this may be quite tolerable. Using just the 'Letter A' entries, I noted the following filesize increases, when including images of different # of max. colours: - NEW 2006 Wikipedia CD Selection_letter A (original with max. 150x150 16 colour images) - NEW 2006 Wikipedia CD Selection_letter A_images150-8colors (about 10% incr.) - NEW 2006 Wikipedia CD Selection_letter A_images150 (about 40% increase) - NEW 2006 Wikipedia CD Selection_letter A_images450 (about 85% increase) On a related note, I also uploaded here, a TEST 45MB .epub that contains only 8 colour/grayscale images (whereas all the other ones there have 16 colours) and includes ALL Wikipedia ENTRIES even entries for 'Letter B' to 'Letter Z'. This ebook may not render properly as the .html source included there has not yet been "cleaned up" and may have "glitches" that will be fixed in any released/final ebook version! It looks like, in the end, a .epub with almost original-sized images (450x???) will be about 2/3rds 5/6ths greater than this 45 MB test .epub, namely could be about 75 MB 83 MB. Well, I'm off to create this test .epub (~83MB) with these larger sized images.... and then later upload to my Gargantuan ebook server. EDIT: it actually is 83 MB in size Last edited by nrapallo; 09-15-2009 at 10:27 PM. Reason: updated true size of text ebook with all entries and larger images. It's awesome!!!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Huge discounts on Harlequin eBooks	Rogier	Deals and Resources (No Self-Promotion or Affiliate Links)	1	07-12-2009 09:00 PM
Paragraph spacing when creating eBooks?	gwynevans	Workshop	21	04-24-2009 11:01 AM
Posting Ebooks,link to Wikipedia author	ProDigit	Feedback	1	12-31-2008 01:39 PM
Reference Wikipedia: SOS Children 2006 Wikipedia CD	hn_88	BBeB/LRF Books	0	01-29-2008 12:23 PM
Creating eBooks Isn't So Hard After All	Antoine of MMM	News	4	06-01-2005 04:25 PM

04-08-2009, 12:00 PM	#5
nrapallo GuteBook/Mobi2IMP Creator Posts: 2,958 Karma: 2530691 Join Date: Dec 2007 Location: Toronto, Canada Device: REB1200 EBW1150 Device: T1 NSTG iLiad_v2 NC Device: Asus_TF Next1 WPDN	Using 4-bit .gif improved the filesize of the final ebook a lot. The .imp now is 50 MB, .epub now 43 MB and .prc (stnd) is 44 MB & .prc (high compression) is 37 MB. There was very little image quality loss reducing from 16M colors to 16 colors. Still too big to be useful. If I exclude images, then the resulting filesizes are: 23 MB .imp, 14 MB .epub and 16 MB .prc (stnd). much more useable... I'll experiment a bit more and see if using the Wikipedia 2006 CD source directly instead of the plucker extracted .html/images works just as good or better. Stay tuned...

08-21-2009, 08:01 AM	#10
cbell Frenetic Posts: 590 Karma: 8181 Join Date: Apr 2008 Location: Australia Device: iLiad V2	Gargantuan ebooks... Bring them on! I tried the prc and epub versions on for size. All loaded surprisingly quickly on my reader. I had varying success with the links. I like having images in an ebook and they looked good. Nick, I'll send you a more detailed PM.

08-22-2009, 10:43 AM	#12
ahi Wizard Posts: 1,790 Karma: 507333 Join Date: May 2009 Device: none	I'd be curious to find out initial loading, page loading, and other response times for an ePub that contains the Summa Theologica five times over, ideally structured... though for the sake of simplicity, it might be easier not to nest the structuring fully.

07-14-2010, 01:46 PM	#15
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	I'm rather a fan of seriously big books myself, Nick . Keep up the good work!

Advert

Advert