My Manga Program

[lilman]

Originally Posted by jozicka

Thank you very much for help, so the auto-split option to single frames from one page is setup in canti.properties?

Yes. You enable/disable it there, and you can set other options related to auto splitting landscape scans there as well. All of the options are explained within Canti.properties.

[lilman]

Originally Posted by badbob001

The component that deals with zip/cbz seems to be somehow faulty if the file contains folders within. resulting with:
java.io.FileNotFoundException.
(The system cannot find the path specified)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(Unknown Source)
at java.io.FileOutputStream.<init>(Unknown Source)
at common.utils.FileUtils.unzipFile(FileUtils.java:51 9)
at manga.ConvertManga.main(ConvertManga.java:1314)

If I recompress the pages into a rar or a zip with no subfolders, then it works fine. It looks like when canti is processing a file, it extracts the content to a temp folder in the same place as the original file. When it processes a rar file with a subfolder, I see the subfolder within the temp folder. But when it processes a zip file with a subfolder, the temp folder contains a file with the same name as the subfolder, making me think the program is incorrectly trying to extract the subfolder as a file instead of a folder. Perhaps a quick workaround is to force the use of unrar to proccess zips.

I created a test case and it correctly handled both zipping a folder with subfolders and unzipping an archive with subfolders. I also read through the code for unzipping and it appears to be correct. Can you upload the archive you are having trouble with? You can use a free upload service like megaupload and pm me the download link. Do not reply to this thread with the download link... the comic is probably copyrighted. I will delete the comic once I am done testing with it.

[badbob001]

Originally Posted by lilman

Can you post the names of the zip and rar files you used that resulted in a single pdf. I'm guessing they had the same auto sensed suffix.

It was something simple like:
"manga name z.zip"
"manga name r.rar"
I think the issue is that your app is really expecting "manga name 102" since you only use the "102" part. Since the files didn't have a number, I'm guessing your app dumps all numberless archives together. I'm guessing the use of the original file name option should resolve this.

Originally Posted by lilman

max_dpi is the maximum allowable dpi for output images. If an image has a dpi greater than max_dpi, it is processed to max_dpi; if an image has a dpi lower than max_dpi, no dpi processing is performed (since increasing an image's dpi uses pixel estimation, quality would be no better). max_dpi reflects the maximum dpi of the particular ereader you are processing manga for. Placing an image on an ereader with a dpi greater than the ereader screen can support does not make the image look any better, and at the same time dpi directly relates to file size so you are wasting file space without any gain in quality.

How does a pixel image have dpi? It's just pixels and they have no physical size until it is displayed on a device. DPI is only relevant for vector elements, such as font text.

But if you're saying that PDF generation requires to know the physical aspects of the document (ie: height x width or dpi) so that 100% displays correctly, then I am fine with that. It's just too bad that this probably means 1 to 1 pixel display is near impossible when using PDFs. I'm guessing you're using the -density option in ImageMagick.

Originally Posted by lilman

ImageMagick processes each image individually, except for pdf creation. With pdf creation, ImageMagick has to deal with all of the images that will become part of the pdf at the same time. For larger pdfs (greater than 150 images) this can be a real resource hog, and can possibly even fail.

Maybe I just need to get more ram... I looked up seeing if ImageMagick supports combining several smaller PDFs together (convert *.pdf big.pdf) but reprocessing a PDF may degrade image quality. I suppose you could add an option to break up the volume every x number of pages.

Originally Posted by lilman

Um, I'll have to think about that. Somebody else requested an option for adding borders to an image that is smaller than the specified size, I will probably go with that.

For undersized images, you should skip the cropping step for opposite sides (top/bottom, left/right) that are under the target resolution and just add borders. It doesn't make sense to crop and then add back the borders, which can shift the location of the page elements to the middle.

For oversized images, yes it is tricky since you can't really know what are the true geometry of the original page. Maybe something like this can be done:

1. Have a property file option to set trim limit in percentage. 100% would be the current setting of no limit. Maybe 5%?
2. Translate percentage limit to pixel based on the original image dimensions.
3. Trim background pixels from all sides *equally* until a non-background is reached or pixel trim limit is reached or target resolution reached. This will help preserve the aspect ratio and keep image elements from shifting to the middle.
4. Determine which sides (top/bottom, left/right) has the smaller border (width of background pixels to closest page element)
5. Resample oversized image so the smallest border sides side is at the target resolution.
6. If the larger border sides is still over the target resolution, trim background until it is.
7. If image is still oversized, resample to the target resolution.
8. Fill in border.

I hope that made sense. It seems to work (in my mind) .

[jozicka]

strange, I have message :


Testing rar handler...
Can unrar: false
Can rar: false
WARNING: Canti was unable to find a compatible program for extracting rar archives; therefore, rar archives cannot be used as input files. Please see readme.txt for instructions on setting up a compatible unrar handler.


But I have a winrar3 corporate install (c:\program files\winrar\unrar.exe )and I can manually extract the file using unrar command.


Can I use my own batch to have extracted all jpg for processing in one folder? something like:

java -jar Canti.jar -parent_folder "D:\\image1\\ast\\*.jpg\\" -title "Asterix1" -output_folder D:\\Image2\\


edit: If I run

java -jar Canti.jar -parent_folder "D:\\image1\\ast\\" -title "Asterix1" -output_folder D:\\Image2\\

I got this:


Converting all manga scans using ImageMagick...
Processing folder: Asterix1...
Exception in thread "main" java.lang.NumberFormatException: empty String
at sun.misc.FloatingDecimal.readJavaFormatString(Unkn own Source)
at java.lang.Double.parseDouble(Unknown Source)
at common.image.ImageMagick.getImageData(ImageMagick. java:176)
at common.image.ImageMagick.processImage(ImageMagick. java:113)
at manga.ConvertManga.main(ConvertManga.java:1625)


and files are actually copied to image2\%title% without any change...

[lilman]

Originally Posted by badbob001

It was something simple like:
"manga name z.zip"
"manga name r.rar"
I think the issue is that your app is really expecting "manga name 102" since you only use the "102" part. Since the files didn't have a number, I'm guessing your app dumps all numberless archives together. I'm guessing the use of the original file name option should resolve this.

What you have said is mostly correct. When using auto sense suffix on files that don't end with a number, it's not that Canti is dumping all numberless archives together, it just thinks that each of their suffixes is "" (i.e. empty string). Because they all have the same suffix, they all end up copying to the same folder in output_folder... or repeats get discarded, I don't remember how it works in the new brain. In any case, if you use auto sense suffix on files that don't end with a number, the output will probably be messed up; it is better to use use_original_filenames instead.

Btw, Canti can auto sense suffixes of the form [0-9]*[.]+[0-9]*[-]+[0-9]*[.]+[0-9]* (where * means 0 or infinte instances of what's in the brackets, + means 0 or 1 instances)
I think I wrote that regex correctly. The most complex case would be: 093.34-1234.234
Oh, and there can be spaces or underscores in between.
If you want to use auto sense suffix, you can rename your comics like "03a" and "03b" to "03.1" and "03.2". Or use use_original_filenames once it has been fixed.

Originally Posted by badbob001

How does a pixel image have dpi? It's just pixels and they have no physical size until it is displayed on a device. DPI is only relevant for vector elements, such as font text.

Dpi was a really confusing concept for me to understand (I had to read a few docs to get it). Dpi is the density of pixels in an image. An image with a high dpi means that there are many pixels occupying a small amount of space, and because of this, the image quality can be higher (since you are able to fit more color information into the same amount of space). Think about how the new iPhone4 screen is about the same physical size as the previous iPhone, but images on the newer iPhone can look so much better. This is because the newer screen has a higher dpi (well, there's other factors as well that make the new screen better, but the higher dpi is crucial).

Every ereader device has a certain dpi screen; for example, the Kindle DX screen has a dpi of 150. This means that every 150x150 pixel group takes up a square inch when displayed on the Kindle DX. Let's consider the different image dpi scenarios for a Kindle DX:
1) image has dpi < 150 (for example, 72)
If the image has a dpi of 72, then every square inch of the image is made up of 72x72 pixels. However, the Kindle DX screen displays 150x150 pixels per square inch. This means that each pixel of the input image must be interpreted larger than it was originally intended to. Thus, image quality is a bit lower.
2) image has dpi = 150
The image's dpi matches the screen's dpi. Every square inch of the image will correlate perfectly to a square inch on the screen, so image quality is perfect.
3) image has dpi > 150
The screen cannot handle having more than 150x150 pixels per square inch, so it must run an algorithm on the image at runtime to reinterpret each square inch as 150x150 pixels. There shouldn't be much quality loss decreasing dpi, but you are wasting file space since the screen isn't actually displaying all of the original pixels.

Does that make sense?

Originally Posted by badbob001

But if you're saying that PDF generation requires to know the physical aspects of the document (ie: height x width or dpi) so that 100% displays correctly, then I am fine with that. It's just too bad that this probably means 1 to 1 pixel display is near impossible when using PDFs. I'm guessing you're using the -density option in ImageMagick.

I don't have the Canti source code in front of me right now, but I think I use the -density option.

Originally Posted by badbob001

Maybe I just need to get more ram... I looked up seeing if ImageMagick supports combining several smaller PDFs together (convert *.pdf big.pdf) but reprocessing a PDF may degrade image quality. I suppose you could add an option to break up the volume every x number of pages.

I tried writing my own algorithm to break larger pdf projects into smaller pdfs and then join them all together... it didn't work any better than making one large pdf from the start. If the large pdf creation fails, doing it through smaller pdf merges also fails. The ideal solution would be for me to write my own pdf handler (I think the pdf standard is open source), but that's not going to happen since I'm already working on another project.

Originally Posted by badbob001

For undersized images, you should skip the cropping step for opposite sides (top/bottom, left/right) that are under the target resolution and just add borders. It doesn't make sense to crop and then add back the borders, which can shift the location of the page elements to the middle.

I was thinking about that. I still want to crop then add borders since I will include an option for the user to set border color (so you can trim white and then add back black border if you want, or vice versa).

Originally Posted by badbob001

For oversized images, yes it is tricky since you can't really know what are the true geometry of the original page. Maybe something like this can be done:

1. Have a property file option to set trim limit in percentage. 100% would be the current setting of no limit. Maybe 5%?
2. Translate percentage limit to pixel based on the original image dimensions.
3. Trim background pixels from all sides *equally* until a non-background is reached or pixel trim limit is reached or target resolution reached. This will help preserve the aspect ratio and keep image elements from shifting to the middle.
4. Determine which sides (top/bottom, left/right) has the smaller border (width of background pixels to closest page element)
5. Resample oversized image so the smallest border sides side is at the target resolution.
6. If the larger border sides is still over the target resolution, trim background until it is.
7. If image is still oversized, resample to the target resolution.
8. Fill in border.

I hope that made sense. It seems to work (in my mind) .

Using % with images is tricky because of dpi. This is made even more tricky since I think there are some images that you can't determine the dpi of (very annoying problem, it's the reason I use ImageMagick to determine dpi instead of letting Java handle it). I haven't programmed it yet, but I think I will keep the auto trim as it is and include the option to fill border for images smaller than the target size. That is the simplest way to go, and since I need to be focusing more on my new porject I can't spend too much time updating Canti.

[lilman]

Originally Posted by jozicka

strange, I have message :


Testing rar handler...
Can unrar: false
Can rar: false
WARNING: Canti was unable to find a compatible program for extracting rar archives; therefore, rar archives cannot be used as input files. Please see readme.txt for instructions on setting up a compatible unrar handler.


But I have a winrar3 corporate install (c:\program files\winrar\unrar.exe )and I can manually extract the file using unrar command.

I will look at the source code to see why it didn't find your rar handler.

Originally Posted by jozicka

Can I use my own batch to have extracted all jpg for processing in one folder? something like:

java -jar Canti.jar -parent_folder "D:\\image1\\ast\\*.jpg\\" -title "Asterix1" -output_folder D:\\Image2\\

Close, but that won't work. Just specify the folder path, not the *.jpg part. So it would look like:
java -jar Canti.jar -parent_folder D:\\image1\\ast\\ -title "Asterix1" -output_folder D:\\Image2\\

Originally Posted by jozicka

edit: If I run

java -jar Canti.jar -parent_folder "D:\\image1\\ast\\" -title "Asterix1" -output_folder D:\\Image2\\

I got this:

Converting all manga scans using ImageMagick...
Processing folder: Asterix1...
Exception in thread "main" java.lang.NumberFormatException: empty String
at sun.misc.FloatingDecimal.readJavaFormatString(Unkn own Source)
at java.lang.Double.parseDouble(Unknown Source)
at common.image.ImageMagick.getImageData(ImageMagick. java:176)
at common.image.ImageMagick.processImage(ImageMagick. java:113)
at manga.ConvertManga.main(ConvertManga.java:1625)


and files are actually copied to image2\%title% without any change...

Upload that comic to a free upload service like megaupload and pm me the download link. I will have to test it myself to see what is going wrong.

[badbob001]

Originally Posted by lilman

Dpi is the density of pixels in an image.

I understand what DPI is but I just don't understand why we need to worry about it. If I'm making a wallpaper for my PC or a smartphone, all I care about is the resolution (ie: 1280x1024, 480x320)... I don't need to worry about the PC being 72dpi and the Smartphone being 200dpi. But if someone gave me an image and said that it was scanned at 1200dpi and I need to print it at 1200dpi to match the original, then I can set my printer to that dpi. Reader devices have a fixed display dpi so there is no way we can match the original. I bet if the output was a bitmap instead of PDF, you wouldn't need to worry about it either. But alas, I don't believe PDF allows for a pixel as a unit of measurement, and thus you are probably forced to use DPI. This is probably why manga/comics are packaged in a cbz/cbr instead of PDF... since there is no way to force a PDF to display an image with 1:1 pixel correctness and without interpolation. Ok, I think we are off topic now.

Originally Posted by lilman

Using % with images is tricky because of dpi. This is made even more tricky since I think there are some images that you can't determine the dpi of (very annoying problem, it's the reason I use ImageMagick to determine dpi instead of letting Java handle it).

I used percentage because specifying a pixel amount doesn't make sense. For example, 100 pixels for a 1600x1200 page is small, but huge for a 800x600 page. With percentage, you calculate the number of pixels relative to the resolution of the page: ie: width pixels = width resolution x %, height pixels = height resolution x %. Unless your input file is PDF, you are still working with non-physical bitmaps and thus DPI is unimportant until the very end when you need to output to PDF. Some image formats like jpeg allow for metadata to be included, such as dpi, but I have never needed to worry about it... I can only imagine it being relevant if I wanted to print it to paper and don't want different image elements to look different. One day, we may have screens that match the dpi of print... when that day comes, then we probably no longer need to worry about resolution and just care about dpi.

Good luck on your new project. We certainly don't want you to get burned out doing the same thing... but please revisit!

[lilman]

Originally Posted by badbob001

I understand what DPI is but I just don't understand why we need to worry about it. If I'm making a wallpaper for my PC or a smartphone, all I care about is the resolution (ie: 1280x1024, 480x320)... I don't need to worry about the PC being 72dpi and the Smartphone being 200dpi. But if someone gave me an image and said that it was scanned at 1200dpi and I need to print it at 1200dpi to match the original, then I can set my printer to that dpi. Reader devices have a fixed display dpi so there is no way we can match the original. I bet if the output was a bitmap instead of PDF, you wouldn't need to worry about it either. But alas, I don't believe PDF allows for a pixel as a unit of measurement, and thus you are probably forced to use DPI. This is probably why manga/comics are packaged in a cbz/cbr instead of PDF... since there is no way to force a PDF to display an image with 1:1 pixel correctness and without interpolation. Ok, I think we are off topic now.

I used percentage because specifying a pixel amount doesn't make sense. For example, 100 pixels for a 1600x1200 page is small, but huge for a 800x600 page. With percentage, you calculate the number of pixels relative to the resolution of the page: ie: width pixels = width resolution x %, height pixels = height resolution x %. Unless your input file is PDF, you are still working with non-physical bitmaps and thus DPI is unimportant until the very end when you need to output to PDF. Some image formats like jpeg allow for metadata to be included, such as dpi, but I have never needed to worry about it... I can only imagine it being relevant if I wanted to print it to paper and don't want different image elements to look different. One day, we may have screens that match the dpi of print... when that day comes, then we probably no longer need to worry about resolution and just care about dpi.

Dpi relates directly when printing. I think it is still important to screen displays, but the correlation is indirect. To a screen, 100 pixels is 100 pixels, but since every screen displays at a given dpi, the pixel interpretation is virtual.
... This dpi talk is getting confusing. I'll just add an option to disable max_dpi, and that way the user can decide if Canti should mess with dpi.

Originally Posted by badbob001

Good luck on your new project. We certainly don't want you to get burned out doing the same thing... but please revisit!

Thanks. I enjoy working on Canti, and I personally use it a lot (I read manga on my iPad every day), but it is a free program. My new project will be commercial and requires a lot more research and effort than Canti, so I need to cut back on Canti development.

[lilman]

Here's Canti v1.91

Note: There were a lot of requests for features and bug fixes lately, I hope this update addressed most of them.

New to verison 1.91:
-Bug fix: Unzipping didn't work on Windows if the archive contained subfolders
While unzipping a zip archive, the way Canti was determining whether a zip entry was a folder didn't work correctly on Windows machines.
The code has been updated, and now unzipping archives with subfolders should work correctly on all os platforms.
-Bug fix: use_original_filenames caused crashes
This bug was introduced in Canti v1.90 when I rewrote Canti's brain. It was a one line of code fix, problem solved.
-Auto split buffer
This value determines the % of the other side of a split page that should be included after the split (disabled by default).
For example, if the value is set to 0.2 then 20% of the other split page will be included in the current split page.
Acceptable values: A decimal between 0 and 1.0, inclusive (0 will disable the buffer, 1.0 will include the complete other page)
This is set up in the .properties file.
-Add border
Determines whether to add borders to images smaller than the desired size (disabled by default).
Can also specify the color of the added border as either black, white, gray, or a color hex code (like #cc0000).
This is set up in the .properties file.
-Auto trim cancelling
An auto trimmed image will be discarded if x% of the total number of pixels from the original image get trimmed.
For example, if you set @trim_cancel to 0.7, then an auto trimmed image will be discarded if 70% of the pixels from the original image were trimmed.
This is set up in the .properties file.
-Improved rar handling on Windows
Canti is now better able to find a compatible rar handler on Windows os.
-Analyze Manga Collection results saved to file
Before the Analyze Manga Collection feature would only output the results to the command terminal. Now it also records these results in a text file.
-@max_dpi can now be disabled
If you set max_dpi to -1 the feature will be disabled (meaning that the dpi of input images will not be altered during processing).
This is set up in the .properties file

Enjoy and let me know if this works for you

[jozicka]

Hi,

this asterix file...

unrar - the same issue

jpg files - for some time running, but again error.


Copying files to new folders...
Converting all manga scans using ImageMagick...
Processing folder: Asterix1...
Exception in thread "main" java.lang.NumberFormatException: empty String
at sun.misc.FloatingDecimal.readJavaFormatString(Unkn own Source)
at java.lang.Double.parseDouble(Unknown Source)
at common.image.ImageMagick.getImageData(ImageMagick. java:270)
at common.image.ImageMagick.processImage(ImageMagick. java:150)
at manga.ConvertManga.main(ConvertManga.java:1662)




I was thinking - may be if you prepare some directory with all needed tools: rar, java, imagemagic and zip it and put it to same megaupload, so we can just extract to C:\ drive and run it, it should avoid issue with different versions..

something like \convertmanga\
inside folders TOOLS (with all needed tools)
then folder SOURCE
then folder OUTPUT
then several canti properties, so user will rename the one he needs.

and script convertmanga.bat with inside

FOR /f "tokens=*" %%G IN ('DIR /B c:\convertmanga\SOURCE\*.*') DO (SET FILE=%%G) & CALL :CONV1
PAUSE
goto :EOF

:CONV1
java -jar Canti.jar -parent_folder "C:\\convertmanga\\%file%\\" -title "%file%" -output_folder "c:\\convertmanga\\OUTPUT\\"
goto :EOF


So users will only rename canti properties (which they want) and put cbr files to source and run bat script?

Maybe in the canti properties can be a path for rar to this c:\\convertmanga\\TOOLS\\unrar.exe


But great job. Thank you for your work on this.

[lilman]

Originally Posted by jozicka

unrar - the same issue

Can you post the path to rar.exe on your system, and make sure you use exact capitalization. For example, on my Windows XP machine it is:
C:\Program Files\Winrar\Rar.exe

Originally Posted by jozicka

jpg files - for some time running, but again error.

Copying files to new folders...
Converting all manga scans using ImageMagick...
Processing folder: Asterix1...
Exception in thread "main" java.lang.NumberFormatException: empty String
at sun.misc.FloatingDecimal.readJavaFormatString(Unkn own Source)
at java.lang.Double.parseDouble(Unknown Source)
at common.image.ImageMagick.getImageData(ImageMagick. java:270)
at common.image.ImageMagick.processImage(ImageMagick. java:150)
at manga.ConvertManga.main(ConvertManga.java:1662)

Damn, when I tested the Asterix test case you sent me it worked perfectly in Linux. I will have to do testing in Windows XP to see if it is a Windows issue.

Originally Posted by jozicka

I was thinking - may be if you prepare some directory with all needed tools: rar, java, imagemagic and zip it and put it to same megaupload, so we can just extract to C:\ drive and run it, it should avoid issue with different versions..

something like \convertmanga\
inside folders TOOLS (with all needed tools)
then folder SOURCE
then folder OUTPUT
then several canti properties, so user will rename the one he needs.

I don't want to do that. Users should be able to download the latest version of any of Canti's helper programs without compatiblity issues. If I make a package with all the helper programs, the user may be using outdated versions that run slower or produce lower quality output than their newer counterparts. For instance, I just updated my ImageMagick the other day cause there had been some nice updates since the last time I installed it.

Originally Posted by jozicka

and script convertmanga.bat with inside

FOR /f "tokens=*" %%G IN ('DIR /B c:\convertmanga\SOURCE\*.*') DO (SET FILE=%%G) & CALL :CONV1
PAUSE
goto :EOF

:CONV1
java -jar Canti.jar -parent_folder "C:\\convertmanga\\%file%\\" -title "%file%" -output_folder "c:\\convertmanga\\OUTPUT\\"
goto :EOF

So users will only rename canti properties (which they want) and put cbr files to source and run bat script?

The way Canti.properties works now is easier than your suggestion. The user specifies his/her ereader device and Canti makes a .properties file especially for that device (and also finds ImageMagick on the user's system, which can vary among systems). If the user wants more than one .properties file because they have more than one ereader, he/she can give it a unique name and specify -properties [path to .properties file] in the command line at runtime.

Originally Posted by jozicka

Maybe in the canti properties can be a path for rar to this c:\\convertmanga\\TOOLS\\unrar.exe

If I still can't get rar support working for you in the next update, I will add an option to specify its location in the .properties file (similar to ImageMagick).

[badbob001]

Is there a way to upgrade a canti.properties to the latest version with the new options or should I just rename the old one, generate a new one, and manually copy my settings over?

[lilman]

Originally Posted by badbob001

Is there a way to upgrade a canti.properties to the latest version with the new options or should I just rename the old one, generate a new one, and manually copy my settings over?

There's no update function. Rename your current file so it doesn't get overwritten, make a new one, and copy over the settings manually.

Sometimes variable names change, or the meaning of settings change, so keeping the update function backwards compatible would be too much of a hassle :P

[squall_dc]

Thanks lilman for the quick fixes!

The split buffer works perfectly! And it no longer crashes when when I use original filenames.

Nice work!

[badbob001]

The latest version works great! Thanks. For anyone wondering why the occasional landscape page is not split, you have to set auto_sense_landscape_folders = false.

Regarding the new trim_cancel option, which is awesome, if the trim amount exceeds the specified percentage, does it just trim up to the limit or does it leave the page untrimmed?

Is it possible to specify that the add_border_color option use the same color that was trimmed? If nothing was trimmed, then use a specified default border color?

I ran it against a 179 page comic and as usual, convert ate up all the virtual memory but strangely didn't use much real memory. I left it running and churning. In the morning, touching the computer made it bluescreen and reboot. After it's back, I check the comic pdf and it's perfect. I'll have to investigate if convert has parameters to limit its memory usage. Anyway to pass in additional convert parameters during your apps operation?

Thanks!