PNGout, improve PNG optimization

[un_pogaz]

I've just discovered an absurdly powerful PNG optimization tool: PNGout (Original page, Linux port)
I clearly don't have the level to implement it inside calibre, but damn it's really powerful, so I'll share the info if you want to use it.

Also, I think you could add an optimize_images command in the "setup.py" that will run PNGout and then OptiPNG on all PNG, so this optimization can be done entirely autonomously, without having to use the ImgBot service.
(note: optimize_images instead a optimize_png because we could extend it later to other images optimization, like SVG... but later... when there's the motivation to it)

[JSWolf]

The website hosting PNGout is terrible. The background plus the yellow color for the text makes it hard to read.

[Dsharma]

Thanks for the share! I'd never heard of PNGout before, but it does sound like a powerful tool. You bring up a great point about integrating it into the setup.py as an optimize_images command, that'd really streamline the process. As for extending it to other image types like SVG, it's definitely an exciting prospect. We'll just need someone with the right skills and a dash of motivation, right? ��

And @JSWolf, you've got a point about the website's design. Navigating it is a bit of a chore with that color combo.

[kovidgoyal]

Its best claimed improvement is 10% over optipng for some images. I doubt it would actually achieve that for most images. But feel free to run it on some PNGs in the repo and see if you feel like it. If it is significantly better, then I might look at it.

As for integrating in setup.py, why bother, since imgbot does it for us automatically.

[un_pogaz]

Oh, but what are you saying @JSWolf? In the 2000s, this type of website was the pinnacle of quality and innovation, you millennial with no taste for beauty.
(OK, I'll close the subject right now: I was born in 1994, the first site I saw was in 2008 and it was already much better. Just, the joke was too easy to not do it.)

---

Originally Posted by kovidgoyal

Its best claimed improvement is 10% over optipng for some images.

Hmm, according to some initial tests I did, it was more than that. 10% was the final file size... but these were ultra-specific cases, so the result will probably be different on calibre images.

*testing done, that was fun*
*Hmm, I've had some interesting results*

Preamble:
I didn't know too deep into how they work, but from what little I understood:
OptiPNG works on reducing the redundancy of certain data within the PNG file, without touching its content too deeply.
PNGout, if it does that too, has a deeper analysis capacity and can decide to completely rewrite the file, using the "color palettes" mode if possible. And in those moments, it becomes truly monstrous efficiency.
(And no idea how ImgBot works)

Parameter used:
OptiPNG -o7 <file>
PNGout <file> /s0

Result:
Note: I ran the test with d28319b1 as the parent commit, before the last ImgBot run, to get the cleanest comparison with ImgBot (the results will therefore be significantly different from the current commit)
You can download the test results in the attached ZIP file. The '--filtred' txt only contains files that have also been processed by ImgBot for easier file-to-file comparison.

OptiPNG, on the whole, only managed to save around ten bytes on average, with the occasional hundred or so on the lucky few. Nevertheless, when compared with ImgBot, it's surprising: ImgBot modified 37 PNGs (16027 bytes saved) and OptiPNG 52 (6998 bytes saved) BUT, there are only 2 files in common, and in both cases OptiPNG performed significantly better by ~100b.
The result was a mixed bag, with the two software appearing to have different strategies that don't overlap.

PNGout was clearly more efficient for him, modifying 730 files, which leads to a result 124896 bytes saved, which is ~100Kb more than ImgBot. With each file in common, PNGout outperforms by ~100b or more. However, most of the results are on older files, which ImgBot processed a long time ago but hasn't touched since. The improvements are in the same order of magnitude, 10b to 100b and somtime more, but there are many. With some surprising ones like 'recipes/icons/thenewcriterion.png': last optimized by ImgBot 11 months ago and was ignored by 5 other passages; PNGout? 1531b -> 120b.
There are only 2 images that PNGout didn't handle compared to ImgBot... for a reduction of ~40b each. More than negligible view the size saved elsewhere.
Note that in some very minor cases, PNGout performs less well than OptiPNG (but still better than ImgBot).

And this if you use PNGout alone. If you make a second pass behind with OptiPNG, you gain even more for a total of 133745 bytes saved (+10Kb that only PNGout)
> A make a pull request with this, but with the most recent commit.

Conclusion:
PNGout is clearly much more powerful than OptiPNG, well over the 10% announced above.
And, if you use the right tools in the right order (PNGout+OptiPNG), you can get a better results than ImgBot (2%). Thin, but sensitive when you consider that most of the images are already optimized by ImgBot which is already very efficient.
So, I don't know whether to include it in the calibre workflow, nevertheless, I think it's a nice solution to know about.
But beyond the image optimization of the calibre software itself, PNGout could be a welcome addition to the image optimization toolbox of ebooks, because right now I think caliber only implements OptiPNG, but PNGout is much more powerful.
But detail. The information is transmitted.

[kovidgoyal]

PNGout is not open source so there is no chance of it being used in calibre, I refuse to bundle closed source binary blobs with calibre.

[un_pogaz]

Point.
I'd forgotten about that.
Have a nice day.

[JSWolf]

Originally Posted by un_pogaz

Oh, but what are you saying @JSWolf? In the 2000s, this type of website was the pinnacle of quality and innovation, you millennial with no taste for beauty.

Do you really believe this website is quality and innovation? It's hard to read and very ugly. This is the type of website people used to build using Microsoft Frontpage. It was awful then and it's awful now.

[un_pogaz]

Okay. So my autistic brain decided that was this kind of moment.

I did a little research, and found some new toys. After many tests, I came to the conclusion that the best tool to use is advpng (open-source GPL)

Command:
advpng -z -4 -i 10 <file>

Result:
Note: Like the previous, this test was done at commit d28319b1. And like the previous, your find the results in the attached ZIP file.

The results of advpng on were very interesting. Alone, it is slightly more efficient than PNGout, while covering more files, 1156 files for 172976 bytes saved. However, once you make a pass with OptiPNG afterwards, the results are far more impressive: 1297 files for 1621349 bytes saved.

In the end, on the test commit, advpng+OptiPNG achieved a 3.7% gain in total PNGs size (less with the most recent commit which is already optimized, but more than 1%).

Conclusion:
I think I can bring this topic to a close with the final suggestion of using advpng+OptiPNG, and therefore possibly adding advpng to the calibre toolbox.

I also made a Pull request with this new tool set.

[JSWolf]

With all of this extra compression of PNG files, is this going to take longer to display PNG? Is this going to take more battery to display PNG? This is using a Reader.

[kovidgoyal]

advpng says in its manpage that it is not meant for generic PNG files only those produced advancedmame.