View Full Version : manga2pdf - Linux bash scripts for the Cybook, Sony Reader and iLiad


IceHand
08-10-2007, 04:23 PM
Scripts updated on 28 February 2009
You may (re-)use these scripts under the terms of the CC-BY License (http://creativecommons.org/licenses/by/3.0/de/deed.en_GB).

This script will scan for ZIP- and RAR-files (including CBZ- and CBR-files) inside the current folder and inside subfolders (but not subsubfolders), optimize the images found in the archives and put them into PDFs inside a "CYBOOK Converted Manga", "SONY Converted Manga" or "ILIAD Converted Manga" folder (the title and content of the PDF will be the same as the archive).
If there already is a PDF inside the Converted Manga folder with the same name as the archive, the file will be skipped, e.g. if the file "CYBOOK Converted Manga/Megatokyo/MT_Chapter01.pdf" exists, the file "Megatokyo/MT_Chapter01.zip" will be skipped.
The script has the ability to halve double pages (two pages of a manga in one image). By default it's disabled, to activate it open the script and set "HalveDoublePages" to "1". The default is to use the standard manga orientation right-to-left, if you rename a folder to have the characters LTR at the very end (e.g. "Megatokyo LTR"), left-to-right orientation will be used. Of course the script will only halve images if the width is greater than the height and the size is greater than the size of the final resolution.
The script has also the ability to rotate double pages. By default it's disabled, to activate it open the script and set "RotateDoublePages" to "1" for 90 counter-clockwise rotation or "2" for 90 clockwise rotation. This does not work if halving of double pages is activated. Of course the script will only rotate images if the width is greater than the height and the size is greater than the size of the final resolution.

Requirements:
Bash compatible shell, ImageMagick, GIMP, libtiff (aka libtiff-tools), unzip, unrar.
ImageMagick version 6.3.2 or newer is needed for the -extent option to work properly, see posts at the end of the first page of this thread for further informations.

How to install it:
Move the colors4.gpl, colors8.gpl and colors16.gpl files into your GIMP palettes folder (usually "~/.gimp-2.4/palettes").
Move the batch-unsharp-mask-4.scm, batch-unsharp-mask-8.scm and batch-unsharp-mask-16.scm into your GIMP scripts folder (usually "~/.gimp-2.4/scripts")
Move the bash scripts that you need somewhere meaningful (e.g. "/usr/bin" or your Nautilus scripts folder "~/.gnome2/nautilus-scripts").

Note:
As you might have guessed there are three versions of this script, optimised for the Cybook (600x800, 4 colors), Sony Reader (570x760, 8 colors) and the Iliad (768x1024, 16 colors).
For the image optimization trimming will be used, if you don't want trimming, search for and remove "-fuzz "15%" -trim +repage".
The images will be sharpened and dithered using a GIMP script, as this gives better results than the ImageMagick functions.

Known issues:
- Will not work with sub-sub-sub-directories inside an archive (you'll get an error message)
- Only works with *.png, *.jpg and *.gif images (can be changed easily)

Changelog:
4.5.2 Support for CBZ- and CBR-files added.
Changed the order of the tiff2pdf command as it gave errors on a Mac. [link (http://www.mobileread.com/forums/showpost.php?p=367518&postcount=45)]
Removed some unnecessary lines of code.

4.5.1 Fixed a small bug that produced errors with filenames that begin with "-".

4.5 Added the option to rotate double pages 90 counter-clockwise or clockwise

4.4 Changed the the default dpi to 83 on the Cybook to allow zooming with the new firmware.
Added the option to change the dpi setting (at the top of the script), the default setting for Sony Reader is set to 166 and for the iLiad to 160. This should conform to 100% zoom. Please inform me if a different default setting would be better.

4.3 The script will write to the file error.log if it encounters errors.
Added the option to stop the script from exiting on errors.
Check if all necessary programs are installed before running the script.

4.2 Set the image density to 166dpi (Cybook, Reader) or 160dpi (iLiad). Has probably no effect on viewing with the Cybook and Sony Reader, but now the PDFs will correctly view with Evince and Xpdf too (haven't tested them with Adobe Reader).

4.1 Fixed the internal name of the 8 color palette

IceHand
08-11-2007, 07:21 AM
Is there a better format to convert to than PDF? Simple ZIP-files should work on the Cybook and Iliad (I think), but not on the Reader. Is there an easy way to convert a bunch of images to LRF?

For Windows there seem to be better programs for this whole Manga2Ebook task, but unfortunatly not for Linux (don't know about the final image quality compared to this script though).

As I said, some feedback would be nice. Especially the 600x800 resolution for the Reader might not be optimal, because the device has a small bar at the bottom of the screen with the page number and stuff.

Steven Lyle Jordan
08-11-2007, 10:43 AM
Book Designer allows you to turn images into LRF, and you can use as many as you want. If the image is sized to close to 600x800, the next image will simply jump to the next page. And as BD will generate page numbers for the LRF, you might not need the ones embedded into MegaTokyo.

To get the images into BD, you could try embedding them into a Word or HTML file, then importing that into BD. If I were you, I'd lower the resolution as much as possible, to bring down the file size.

JSWolf
08-11-2007, 10:49 AM
Also, PDF is a bad idea. Please, no PDF.

IceHand
08-11-2007, 11:27 AM
Book Designer is a Windows program, isn't it? And even if it would work with Linux, I would prefer a command line tool for batch processing. But thanks for the suggestion anyway.
And when you say "lower the resolution as much as possible" how big is the viewable resolution of the Reader?

kovidgoyal
08-11-2007, 12:49 PM
comic2lrf

athlonkmf
08-12-2007, 02:10 PM
Also, PDF is a bad idea. Please, no PDF.

PDF is a good idea if you take the target-device into consideration.

I made Manga2Ebook (the windows version) to create PDF files because it can then be converted to other formats with the appropiate tools. Not to mention, a rasterized PDF-page is smaller than compressing an image directly into lrf.

athlonkmf
08-12-2007, 02:12 PM
Is there a better format to convert to than PDF? Simple ZIP-files should work on the Cybook and Iliad (I think), but not on the Reader. Is there an easy way to convert a bunch of images to LRF?

For Windows there seem to be better programs for this whole Manga2Ebook task, but unfortunatly not for Linux (don't know about the final image quality compared to this script though).

As I said, some feedback would be nice. Especially the 600x800 resolution for the Reader might not be optimal, because the device has a small bar at the bottom of the screen with the page number and stuff.


Use 570x760. Be sure that the PDF is in that format too.

IceHand
08-12-2007, 02:48 PM
Thanks for the input, I will change that in the script. One question though: Why the width of 570px? I can understand the height of 760px (because of the bar at the bottom), but why can't the whole width of the screen be used?
Anyway, I will make some more adjustments as soon as I have a Cybook. Btw, is there any chance that you will release a Linux version of Manga2Ebook?

athlonkmf
08-12-2007, 06:03 PM
Thanks for the input, I will change that in the script. One question though: Why the width of 570px? I can understand the height of 760px (because of the bar at the bottom), but why can't the whole width of the screen be used?
Anyway, I will make some more adjustments as soon as I have a Cybook. Btw, is there any chance that you will release a Linux version of Manga2Ebook?

The width is to make sure the page is in a 4:3-ratio, otherwise the Reader will use ugly big margins to display things.

As for a linux version, i'm not sure. If the initial version was succesful enough, i might have considered going for a java-based solution. Unfortunately there wasn't much response. So I just made something that suits my needs. I managed to write the .NET-version in a week, so I did that.

sealbeater
04-03-2008, 01:35 AM
Wow, this is exactly what I've been looking for. Quick question or two, how can I specify the 8 scale option, and what does the -extent option to convert do, my version didn't have it but it may be old so I'll try again with a newer version, but thanks a lot for this, I really appriciate it.

IceHand
04-03-2008, 06:43 AM
Wow, this is exactly what I've been looking for. Quick question or two, how can I specify the 8 scale option, and what does the -extent option to convert do, my version didn't have it but it may be old so I'll try again with a newer version, but thanks a lot for this, I really appriciate it.Oh yeah, I forgot to update the scripts. Download the new version and you'll find an optimised version for the new Sony Reader (8 colors). Be sure to read the new description as I've changed some things.

The -extent option extends the image to a specified size with a white border, e.g. after resizing an image the size might be 526x760, but it will be extended with a white border to 570x760 so that every image has exactly the same size. If you don't do that the PDF might get screwed up.

EDIT: The 8 color palette was named wrong. Please download the new 4.1 version to get the Sony Reader script working.

sealbeater
04-04-2008, 04:09 PM
Awesome, awesome, thank you!!

IceHand
04-05-2008, 06:14 AM
You're welcome.
Please let me know, if the created PDFs give any problems on the Sony Reader. I could only test the PDFs on my Cybook.

sealbeater
04-18-2008, 03:01 PM
I've tested it out and it works perfectly! This is what it's supposed to be about, the *nix way, many thanks again, that java app was killing me.

knewbie
07-23-2008, 06:47 PM
I have been using JE-Comics for some time and the lack of batch was just too daunting. hours at a time...

I also never got comiclrf to do its job on zip and rars and subdirectories at the same time, and the cbr2pdf script was killing a 4*2.4GHz PCfor some reason...(incessant IO ? all 10 manga volumes decompressing to memory at the same time and swap actually being full ? ..urghh )

After a few successfull tests I just launched your script on 389 mangas for the coming long holidays and now I can go to sleep (almost) certain everything will be done when I wake up...

Thank you 8)
:thanks:

(Sony PRS-505, Ubuntu Hardy Heron amd64, Q6600 4*2.4 GHz + 4 Go Ram)

IceHand
07-24-2008, 07:21 AM
You're welcome. I hope the script worked without flaw with all your mangas.

knewbie
07-24-2008, 06:40 PM
It worked almost flawlessy. I had a few rars containing other rars that I discovered in the morning, and some sub-subdirectories in the rars, but I am now done fixing it.

3.5 Gb mangas, the Hugo prize winner books of the last 30 years + many others... I won't hope it rains and I have to stay inside, but I see the prospect bothering me less and less 8)

Also I won't be installing the new firmware before AFTER the holidays. With my luck I'd brick it 8p

nekokami
07-24-2008, 09:19 PM
Great script! I've got it churning through some archives now. How about an option to rotate double-wide pages, rather than halving them?

nekokami
07-24-2008, 09:36 PM
It would also be helpful to include some command-line tests for dependencies. I had to restart twice as the script exited since I didn't have ImageMagick installed, then libtiff. Having to restart after installing libtiff was especially annoying because the script decided it had to re-optimize all the images again. :( Even if the tests aren't included in the script itself, it might be a good idea to post them here. Different distros of Linux come with different packages, so it can be easy to forget whether one has them on a specific machine.

IceHand
07-25-2008, 05:54 AM
Sure, no problem. I'll see to these changes as soon as I have the time, which will be starting next week.

nekokami
07-25-2008, 03:24 PM
Also, more graceful error handling would be appreciated. I left it running overnight on a batch of directories, and it got stuck on one (after optimizing the images, it couldn't open one of them when it was time to create the PDF) so it quit altogether. It would have been nice if it could just have spit out an error message (or saved it for the end of the run, or dumped it to an error log), and continued with processing the other directories.

Not complaining-- it's still a very useful script. Just offering constructive suggestions.

IceHand
07-25-2008, 04:38 PM
I'll see to that too. You can change this already by removing "set -e" from the script (it's right at the beginning). Actually I didn't want to remove this yet, because this way the script is safer. I don't know what could go wrong with the script, but if something goes wrong, the script will safely exit. Especially in some stages of development I was very glad I had this option activated.

But as soon as the dependencies will be checked, I will make this option optional ;)

And thanks for all the suggestions, I don't take them as complaints.

stevy
07-28-2008, 11:29 AM
Hi,
could you put in the description this element :
for ubuntu, install libtiff with : sudo aptitude install lidtiff-tools

For the next users ;)
Thanks for this great script

nekokami
07-29-2008, 05:08 PM
I did comment out set -e in the script and most of my zips were processed correctly. (The others were skipped.) Thanks!

IceHand
07-31-2008, 12:30 PM
I did comment out set -e in the script and most of my zips were processed correctly. (The others were skipped.) Thanks!
What errors did you encounter by the way? Did you have some corrupted archives or something like that?

And I'll probably release a new version this weekend.

nekokami
07-31-2008, 03:14 PM
I don't know what caused the errors. The images all got processed-- it quit on building the PDF. I haven't had time to go in and process the errant archives by hand to see what's different about them.

IceHand
08-05-2008, 10:21 AM
I just released a new version, which includes better error handling and a check to see if all necessary programs are installed.

I haven't added the rotate double pages option yet. Would you want the images to rotate left or right?

nekokami
08-05-2008, 10:59 AM
I'd prefer counter-clockwise, myself.

stevy
09-06-2008, 06:19 AM
hello,
since the update of the cybook
http://www.mobileread.com/forums/showthread.php?t=28805

i forgot to mention that it seems to me that the visual quality of the comics that i have converted with your script, icehand, are reduced.
Any idea ?

IceHand
09-06-2008, 09:04 AM
hello,
since the update of the cybook
http://www.mobileread.com/forums/showthread.php?t=28805

i forgot to mention that it seems to me that the visual quality of the comics that i have converted with your script, icehand, are reduced.
Any idea ?Not yet, I'll look into it + the zoom issue mentioned in the other thread.

IceHand
09-06-2008, 12:34 PM
I've fixed the zooming issue (see changelog), but I'm unable to see a reduced image quality. Could you describe how the quality got worse and/or show a screenshot?

By the way, I forgot to add the rotate option, I'll do that later today.

EDIT: To explain the default 83dpi setting: it's half of the real resolution of the Cybook, so setting it to 83dpi will give you the real image size in pixels at 50% zoom. That means the whole page when the image is resized to 600x800 (default), as the Cybook has a resolution of 600x800 at 166dpi. You're then able to use zoom.
Notice however that when zooming in on an image with a 600x800 resolution, you'll notice pixellation as the Cybook needs to resize up. You could set the resolution to e.g. 768x1024, when you know beforehand that you want zooming. However if you do that, the image sharpness when viewing the whole page is a bit worse, because the Cybook needs to resize down.

IceHand
09-06-2008, 03:50 PM
Finally added the option to rotate double pages 90 counter-clockwise or clockwise.

stevy
09-07-2008, 06:33 AM
I converted a picture with the 2 versions of your script and the quality is the same. that's all my fault. i'm reading a serie of comics books and i just realize that i start one number which the scan is darker than usual.
Thanks again for the script and your great quickness :)

IceHand
09-07-2008, 06:58 AM
Ah, I see. So that explains it :D

serpentium
10-26-2008, 12:47 PM
i love the idea of this piece of software but i cannot use with mac os x because i dont know why the line calling gimp doesnt work. I tried installing Gimp.app but there is no command line...
any mac os x users solved?

IceHand
10-26-2008, 03:48 PM
i love the idea of this piece of software but i cannot use with mac os x because i dont know why the line calling gimp doesnt work. I tried installing Gimp.app but there is no command line...
any mac os x users solved?Hm, I don't have access to a Mac so I don't know if this will help: try searching for the Gimp executable file and replace in line 216 of the script the word "gimp" with "/PathToExecutable/gimp" (on Linux usually "/usr/bin/gimp").

If that doesn't help: what error message do you get, when you try to run the script?

serpentium
10-28-2008, 06:57 PM
Hm, I don't have access to a Mac so I don't know if this will help: try searching for the Gimp executable file and replace in line 216 of the script the word "gimp" with "/PathToExecutable/gimp" (on Linux usually "/usr/bin/gimp").

If that doesn't help: what error message do you get, when you try to run the script?

Like you said i found gimp and i changed the script but still some problem with gimp. I have 2.4 in osx and when i run it from command line i obtain
------------------------------------
(gimp-2.4:7356): LibGimpBase-WARNING **: gimp-2.4: gimp_wire_read(): error
Traceback (most recent call last):
File "/Applications/Multimedia/Grafica/Gimp.app/Contents/Resources/lib/gimp/2.0/plug-ins/foggify.py", line 20, in <module>
from gimpfu import *
File "/Applications/Multimedia/Grafica/Gimp.app/Contents/Resources/lib/gimp/2.0/python/gimpfu.py", line 74, in <module>
import gimp
ImportError: could not import pygtk
------------------------------------
so im trying to install and compile from scratch. I dont know why cannot find pygtk because inside the Gimp.app there is.

IceHand
10-29-2008, 08:26 AM
Huh, I'm sorry, I don't know what's wrong there. Maybe you could try asking in a Mac forum or something like that?

serpentium
10-29-2008, 06:25 PM
Huh, I'm sorry, I don't know what's wrong there. Maybe you could try asking in a Mac forum or something like that?

I found the first problem was the gimp!!!
So u can say to other user your program works also in a mac!!! They just need a real command line gimp working!
1)Just install mac port and in terminal do port install gimp... it will do everything (maybe something missing u have to do by yourself, but macports will explain how)
2)modify a little bit the script of yours because the convert version of mac imagemagick work sligthly different, so it creates all the time two files when you crop.

I substituted the convert command with just
convert "$i" -gravity West -crop "50%"x"100%" +repage "$PngName.png" 2>>"$ConvMang a/error.log"
and later
convert "$PngName-0.png" -colorspace GRAY -fuzz "15%" -trim +repage -resize "$FinalWidth"x"$FinalHeight" -background "rgb(255,255,255)" -gravity center -extent "$FinalWidth"x"$FinalHeight" "$PngName-0.png" 2>>"$ConvManga/error.log"
convert "$PngName-1.png" -colorspace GRAY -fuzz "15%" -trim +repage -resize "$FinalWidth"x"$FinalHeight" -background "rgb(255,255,255)" -gravity center -extent "$FinalWidth"x"$FinalHeight" "$PngName-1.png" 2>>"$ConvManga/error.log"
for the LTR case or for RTL case
convert "$PngName-0.png" -colorspace GRAY -fuzz "15%" -trim +repage -resize "$FinalWidth"x"$FinalHeight" -background "rgb(255,255,255)" -gravity center -extent "$FinalWidth"x"$FinalHeight" "$PngName-1.png" 2>>"$ConvManga/error.log"
convert "$PngName-1.png" -colorspace GRAY -fuzz "15%" -trim +repage -resize "$FinalWidth"x"$FinalHeight" -background "rgb(255,255,255)" -gravity center -extent "$FinalWidth"x"$FinalHeight" "$PngName-0.png" 2>>"$ConvManga/error.log"
same for jpg, and so on...
if someone needs i can upload the script or whatever u like to do with Your script!

thankyou again for the sw... i m running it now and i love the RESULTS :-)

IceHand
10-30-2008, 07:07 AM
Glad to hear that it's working now :)

Please feel free to upload a Mac version of the script, I would then add it to the first post.

IceHand
11-07-2008, 05:30 AM
New version 4.5.1: Fixed a small bug that produced errors with filenames that begin with "-"

sealbeater
02-13-2009, 01:36 AM
New version 4.5.1: Fixed a small bug that produced errors with filenames that begin with "-"

Just wanted to let you know that you rock yet again. I know some don't get it but this is a huge plus for *nix users. Thanks again.:thumbsup:

IceHand
02-15-2009, 01:52 PM
Just wanted to let you know that you rock yet again. I know some don't get it but this is a huge plus for *nix users. Thanks again.:thumbsup:You're welcome :)

xOrphenochx
02-24-2009, 08:33 PM
Thanks ALOT IceHand. I too was using JE-Comics and batch jobs SUCKED. I did find one error in your latest version. I am using a mac so I don't know if libtiff might have changed since then, but the -o for output needs to be before the .tif file name or else it won't convert to PDF.

IceHand
02-25-2009, 03:16 PM
Thanks ALOT IceHand. I too was using JE-Comics and batch jobs SUCKED. I did find one error in your latest version. I am using a mac so I don't know if libtiff might have changed since then, but the -o for output needs to be before the .tif file name or else it won't convert to PDF.
You mean the tiff2pdf command? Strange, the order shouldn't really matter, but sure I can change it if it produces errors. Which version of libtiff are you using? (I have 3.8.2 installed)

By the way, did you need to change the script as serpentium did (http://www.mobileread.com/forums/showpost.php?p=279781&postcount=40) or did it work aside from the libtiff error?

xOrphenochx
02-26-2009, 06:13 PM
You mean the tiff2pdf command? Strange, the order shouldn't really matter, but sure I can change it if it produces errors. Which version of libtiff are you using? (I have 3.8.2 installed)

By the way, did you need to change the script as serpentium did (http://www.mobileread.com/forums/showpost.php?p=279781&postcount=40) or did it work aside from the libtiff error?

Yes, sorry, that is where it had failed. With the latest Macports installed, 1.7 i think, I have LIBTIFF, Version 3.8.2 just as you do.

I reviewed the script when I got an error that it did not support multiple files and saw the tiffcp command which lead me to believe that where it was failing, I had did a manual merge with the tiffcp command which worked(I hadn't noticed the merged file in the hidden directory previously).

I tried to manually do a conversion and tiff2pdf failed with a simple tiff2pdf test.tff -o test.pdf command. Which produced that multiple files not supported error message along with "tiff2pdf [options] input.tiff" which since -o is an option I reversed it with success.

As for serpentium's edit, there was no need for me to do that. I might suggest that any errors that come about might be related to an outdated macports install, I know for one I never do an upgrade but every few months as it takes hours to compile all the new packages.

I would also commend you for the excellent output of the filters applied to the images before conversion. You will notice a huge difference between such manga as Berserk where it is lightly drawn. JE-Comics files were too light and yours came out at just the right darkness.

You sir, deserve a nice cold beer.

IceHand
02-28-2009, 11:29 AM
Yes, sorry, that is where it had failed. With the latest Macports installed, 1.7 i think, I have LIBTIFF, Version 3.8.2 just as you do.
Right, I changed the order of the command, did some other small changes (see changelog) and uploaded a new version so that should fix the issue. Strange though that the order apparently doesn't matter on Linux, but does on Mac.

As for serpentium's edit, there was no need for me to do that. I might suggest that any errors that come about might be related to an outdated macports install, I know for one I never do an upgrade but every few months as it takes hours to compile all the new packages.
I'm not really sure how this stuff works on a Mac this Macports downloads the sources of the new versions of all your installed packages and compiles them?

I would also commend you for the excellent output of the filters applied to the images before conversion. You will notice a huge difference between such manga as Berserk where it is lightly drawn. JE-Comics files were too light and yours came out at just the right darkness.

You sir, deserve a nice cold beer.
Thanks, I did a lot of tests with different programs and algorithms and went for best visual quality of the output. The downside of this is that the script is fairly slow and depends on (among others) the not very small Gimp.

xOrphenochx
02-28-2009, 08:01 PM
Right, I changed the order of the command, did some other small changes (see changelog) and uploaded a new version so that should fix the issue. Strange though that the order apparently doesn't matter on Linux, but does on Mac.


I'm not really sure how this stuff works on a Mac this Macports downloads the sources of the new versions of all your installed packages and compiles them?


Thanks, I did a lot of tests with different programs and algorithms and went for best visual quality of the output. The downside of this is that the script is fairly slow and depends on (among others) the not very small Gimp.

Macports is a shot off of Gentoo's port package system where it downloads the code and compiles it. Fink is also another package system that is along the lines of Debian's APT system.

I'll try out the new scripts and see how they go.

xOrphenochx
03-03-2009, 06:47 PM
I've noticed another bug, if there are rar and zips in the same directory the rar files will be skipped when running, but they are not listed as skipped.

IceHand
03-04-2009, 09:48 AM
I've noticed another bug, if there are rar and zips in the same directory the rar files will be skipped when running, but they are not listed as skipped.
That's strange, I've had quite a few directories where there were both rar- and zip-files, but the script never skipped a file. Are you sure the archives are not corrupt? If not, could you upload the directory somewhere so I can check it out myself?

xOrphenochx
03-04-2009, 10:55 AM
That's strange, I've had quite a few directories where there were both rar- and zip-files, but the script never skipped a file. Are you sure the archives are not corrupt? If not, could you upload the directory somewhere so I can check it out myself?

I'll have to do that when I get home. I can run just the rar files through the script fine.

xOrphenochx
03-09-2009, 06:36 PM
Well I tried it again and no problem. Must have been a fluke.