Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Formats > Kindle Formats

Notices

Reply
 
Thread Tools Search this Thread
Old 11-17-2019, 04:17 PM   #1
Sekkei
Junior Member
Sekkei began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2019
Device: Kindle Voyage
Change AZW3 compression algorithm?

Afaik AZW3 is based on mobi, wich uses probably deflate (?) to compress. Would it be possible to recompress/change the algorithm to PPMd?
The memory usage is one mb higher, but the compression is way better. (e.g 314KB only text in html > 179KB)
Sekkei is offline   Reply With Quote
Old 11-17-2019, 05:10 PM   #2
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 2,523
Karma: 40693499
Join Date: Apr 2011
Device: pb360
kindlegen (which writes mobi and azw3) has 3 compression options:
-c0: no compression
-c1: standard DOC compression
-c2: Kindle huffdic compression

What would be the point of using a compression that reading software does not handle?
j.p.s is offline   Reply With Quote
Old 11-17-2019, 05:40 PM   #3
Sekkei
Junior Member
Sekkei began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2019
Device: Kindle Voyage
Well, I didn't know there are two option to compress. Which one does calibre use to create azw3 files; Would it be possible to handle kindleGen a folder with azw3 structure to create an azw3, or can I only convert mobi and epub?

Quote:
What would be the point of using a compression that reading software does not handle?
This depends on the reader and how it is coded, if it uses a library with more than one supported decompression method it MIGHT simple work out of box

I just want smaller files, I think 300kb for one book is quite alot.
Sekkei is offline   Reply With Quote
Old 11-18-2019, 05:06 AM   #4
Doitsu
Wizard
Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.
 
Doitsu's Avatar
 
Posts: 4,730
Karma: 15356729
Join Date: Dec 2010
Device: Kindle PW2
Quote:
Originally Posted by j.p.s View Post
kindlegen (which writes mobi and azw3) has 3 compression options:
-c0: no compression
-c1: standard DOC compression
-c2: Kindle huffdic compression
By default, KindleGen will also append the source files, unless the -dont_append_source option is specified.

Quote:
Originally Posted by Sekkei View Post
I just want smaller files, I think 300kb for one book is quite alot.
IMHO, you won't be able to create much smaller ebooks with KindleGen or Calibre.
Doitsu is offline   Reply With Quote
Old 11-18-2019, 05:39 AM   #5
stumped
Wizard
stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.stumped ought to be getting tired of karma fortunes by now.
 
Posts: 1,785
Karma: 1668436
Join Date: May 2016
Device: lLenovo Yoya tab plus , fire HD10 HDX8.9,
Quote:
Originally Posted by Sekkei View Post
I just want smaller files, I think 300kb for one book is quite alot.
ridiculous

if you cant live with 300kb, buy shorter books

you do realise that an e ink reader has enough storage for a few thousand 300Kb books ?

lets say you improve the compression by 10% and save an amazing 30kb per book
what could you possibly use that for ?
stumped is offline   Reply With Quote
Old 11-18-2019, 12:18 PM   #6
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 2,523
Karma: 40693499
Join Date: Apr 2011
Device: pb360
Quote:
Originally Posted by Sekkei View Post
Well, I didn't know there are two option to compress. Which one does calibre use to create azw3 files; Would it be possible to handle kindleGen a folder with azw3 structure to create an azw3, or can I only convert mobi and epub?
Your best bet is to ask calibre questions in a calibre forum.

kindlegen only converts to kindle formats, not from them.
j.p.s is offline   Reply With Quote
Old 11-18-2019, 12:22 PM   #7
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 2,523
Karma: 40693499
Join Date: Apr 2011
Device: pb360
Quote:
Originally Posted by Doitsu View Post
By default, KindleGen will also append the source files, unless the -dont_append_source option is specified.
My point was that amazon written readers are unlikely to support decompression of types not provided by amazon written producers.
j.p.s is offline   Reply With Quote
Old 11-18-2019, 12:25 PM   #8
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 54,414
Karma: 51270872
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 4, iPhone 5
One thing you should be aware of is that with higher compression comes more processor usage and more battery usage to decompress. It's not worth it for the pittance in space you'd save.
JSWolf is offline   Reply With Quote
Old 11-18-2019, 12:42 PM   #9
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 2,523
Karma: 40693499
Join Date: Apr 2011
Device: pb360
Quote:
Originally Posted by JSWolf View Post
One thing you should be aware of is that with higher compression comes more processor usage and more battery usage to decompress. It's not worth it for the pittance in space you'd save.
That is not true in all cases.

A compression algorithm specialized for particular input data might produce smaller output while using less CPU than a general compression algorithm.

It is very common for compression algorithms that squeeze out a bit more compression at the cost of a lot of CPU to have decompressors that use little or no additional CPU.
j.p.s is offline   Reply With Quote
Old 11-18-2019, 09:58 PM   #10
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 54,414
Karma: 51270872
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 4, iPhone 5
Quote:
Originally Posted by j.p.s View Post
That is not true in all cases.

A compression algorithm specialized for particular input data might produce smaller output while using less CPU than a general compression algorithm.

It is very common for compression algorithms that squeeze out a bit more compression at the cost of a lot of CPU to have decompressors that use little or no additional CPU.
I don't know about KF8, but I do know that when Mobi gets compressed to maximum, it does use more CPU to decompress.
JSWolf is offline   Reply With Quote
Old 11-18-2019, 10:15 PM   #11
Sekkei
Junior Member
Sekkei began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2019
Device: Kindle Voyage
Thank you all for responding

Quote:
Originally Posted by Doitsu View Post
By default, KindleGen will also append the source files, unless the -dont_append_source option is specified.
So this helps that unsupported files are not striped away? Would this also help with content inside a e.g HTML that would not be supported by kindleGen?

Quote:
Originally Posted by Doitsu View Post
IMHO, you won't be able to create much smaller ebooks with KindleGen or Calibre.
Sad but propably true, I really hoped they slaped a default zip lib on it and call it a day

Quote:
Originally Posted by stumped View Post
ridiculous, if you cant live with 300kb, buy shorter books
you do realise that an e ink reader has enough storage for a few thousand 300Kb books ? lets say you improve the compression by 10% and save an amazing 30kb per book
what could you possibly use that for ?
I would not even write ebooks in xml if I could, so much wasted space, literally 30kb! haha Sadly my Voyage has only 4GB and I like manga, I even recompress them manually to a per page ~180KB High-res low-color PNG, but as you can imagine, there is not space left if you want to carry your favorite manga saga on the go :/
And saving atleast 30kb on 10 books is one additional book
(If we could use PPMd this would cut the size in half, allowing the doubble amount of books you could have without it)


Quote:
Originally Posted by j.p.s View Post
kindlegen only converts to kindle formats, not from them.
So is there any way to create myself a clean azw3 file, that atleast doesn't attempt to "fix" everything I do, or adding unused meta data every two steps.
I would much prefer to reformate my files in vsCode and push it to phyton lib (or other valid opensource converter)
Sekkei is offline   Reply With Quote
Old 11-19-2019, 05:27 AM   #12
pdurrant
Jolly Christmouse 高貴的老鼠
pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.pdurrant ought to be getting tired of karma fortunes by now.
 
pdurrant's Avatar
 
Posts: 59,715
Karma: 235929799
Join Date: Jul 2007
Location: Norfolk, England
Device: Kindle Voyage
Quote:
Originally Posted by Sekkei View Post
Sadly my Voyage has only 4GB and I like manga, I even recompress them manually to a per page ~180KB High-res low-color PNG, but as you can imagine, there is not space left if you want to carry your favorite manga saga on the go :/
And saving atleast 30kb on 10 books is one additional book
(If we could use PPMd this would cut the size in half, allowing the doubble amount of books you could have without it)
If we're talking about images, your best bet is to look into compressing the images further before turning them into an ebook.
pdurrant is offline   Reply With Quote
Old 11-19-2019, 09:46 AM   #13
Hitch
Bookmaker & Cat Slave
Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.Hitch ought to be getting tired of karma fortunes by now.
 
Hitch's Avatar
 
Posts: 7,981
Karma: 80333121
Join Date: Apr 2010
Location: Phoenix, AZ
Device: K2, iPad, KFire, PPW, Voyage, and NookColor. 2 Droid, 1 Win8 ePUB rdrs
Quote:
Originally Posted by pdurrant View Post
If we're talking about images, your best bet is to look into compressing the images further before turning them into an ebook.
Yes, this, ^ exactly.

At my firm we take a few extra steps, for every book with images, using proprietary tools that we've developed and some tools that are available for anyone (like TinyPNG, which is awesome) to eliminate empty pixels and other wasted space in images.

Amazon actually does a pretty incredible job of compression. But the better job you do beforehand, the easier their job is.

Manga and other comic formats (yeah, yeah, I know, but let me lump them together for now) andif you want to save space, man, that's the place to start. IME and FWIW.

Hitch
Hitch is offline   Reply With Quote
Old 11-19-2019, 07:08 PM   #14
Tex2002ans
Wizard
Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.Tex2002ans ought to be getting tired of karma fortunes by now.
 
Posts: 1,384
Karma: 6863233
Join Date: Jul 2012
Device: Kobo Forma, Nook
Quote:
Originally Posted by stumped View Post
lets say you improve the compression by 10% and save an amazing 30kb per book
what could you possibly use that for ?
10% more books.

Quote:
Originally Posted by j.p.s View Post
It is very common for compression algorithms that squeeze out a bit more compression at the cost of a lot of CPU to have decompressors that use little or no additional CPU.
Yep, or just being much more intelligent with the parameters it's choosing.

Like with PNG compression, there are a million valid ways to compress.

Zopfli is an alternative very CPU-heavy algorithm Google came up with, but resulted in a smaller PNG (~3-8%) by selecting better parameters.

Takes the same exact amount of resources to decompress it though.

I gave some examples way back in 2013:

https://www.mobileread.com/forums/sh...45#post2499645

(Since then, ScriptPNG has been superseded by pingo.)

Quote:
Originally Posted by Sekkei View Post
Sadly my Voyage has only 4GB and I like manga, I even recompress them manually to a per page ~180KB High-res low-color PNG, but as you can imagine, there is not space left if you want to carry your favorite manga saga on the go :/
Manga? PNGs?!!? Then definitely run it through pingo and you'll probably save a hell of a lot of space. At least 3-8%, but depending on the complexity and if the publisher compressed previously, you may even save more.

If they're B&W/grayscale/<256-colors, you could also convert it to Indexed images for you and save A TON of space. (In the example linked above, my chart went from 143.1 KB -> 44.5 KB.)
Tex2002ans is offline   Reply With Quote
Reply

Tags
azw3, compression, ppmd

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Author Sort Name Algorithm Tweak phossler Calibre 3 07-05-2018 01:02 PM
Mobi to Azw3 Conversion - Unable to add/change covers zaqxsw Conversion 6 12-15-2016 07:33 PM
sorting algorithm author Archimed Library Management 6 04-07-2014 04:45 PM
Seriously thoughtful Dekker's Algorithm help. Catire Lounge 13 03-19-2010 11:03 AM


All times are GMT -4. The time now is 04:55 PM.


MobileRead.com is a privately owned, operated and funded community.