My Manga Program

[lilman]

Originally Posted by badbob001

It seems like there are some strange bugs when the output is set to GIF. And some minor bugs with landscape pages when border=true and rotate=false.

The following chart may help explain what is happening (I need to use my free time more wisely :


The input are a portrait and a landscape page from the web comic Octopus Pie. If you want to see the actual input and output images and properties files, see the attached zip.

It looks like some sort of extra black border is being applied to the GIF images to cause them to be bigger than the target size.

Another major thing is that the created GIFs are unreadable on my Kindle but are okay on my PC. When I simply ran "convert.exe 0000.png 0000.gif", that gif views fine on the Kindle so it must be some extra steps in Canti causing the problem. Progressive gif?

Holy thorough testing batbob, nicely done! I will look into what's going on with .gif's tomorrow.

-Update-
badbob, going through the amazing test case you uploaded, it looks like the output .gif images are the correct size. What program are you using to get the width and height stats of the images? Both my linux standard image viewer ("Eye of Gnome") and ImageMagick (identify -format [%w or %h] [filename]) are reporting size information different from your chart.

Originally Posted by badbob001

Another thing is that the generated xxxxx.manga_save file always specifies the filename extension as .PNG, despite the output in canti set for .JPG or .GIF. Is this expected? It seems to cause a strange bug on the Kindle where trying to view the gif makes it show the last screen saver image. Buffer overflow?

I haven't touched the Kindle bookmark code in ages, but this is probably an easy fix.

-Update-
Fixed the Kindle bookmark code, it was a silly mistake that has probably been broken for a long time. I don't read manga on my KDX anymore so I never would have caught this. Thanks, the fix will be included in the next update.

[badbob001]

Sorry that I didn't respond sooner since updating your post doesn't actually make the thread appear updated.

I really don't understand why on linux you're seeing the gif sizes as normal. I'm relying on Windows XP's interface to tell me the dimensions and they are confirmed by MS Paint, Picasa Photo Viewer, Internet Explorer, and Firefox. The attached screen capture compared \GIF\01 Scenario+trim+border+rotate\0000.gif and \JPG\01 Scenario+trim+border+rotate\0000.jpg. Notice that the gif has extra large margins to the left and rights sides and a very small margin to the top (and bottom)? On web browsers, those extra margins appear as white. Perhaps it's transparency?

I ran the canti analyze option and it doesn't show the problem, so I don't know what to say about that (see attached txt). At least it shows the 800x800 issue with jpg/png in certain scenarios.

Have you tried viewing one of the gifs on your Kindle? I'm curious if it can be viewed currently without crashing the app.

Thanks!

[lilman]

Originally Posted by badbob001

Sorry that I didn't respond sooner since updating your post doesn't actually make the thread appear updated.

Good point, I'll make fresh replies in the future.

Originally Posted by badbob001

I really don't understand why on linux you're seeing the gif sizes as normal. I'm relying on Windows XP's interface to tell me the dimensions and they are confirmed by MS Paint, Picasa Photo Viewer, Internet Explorer, and Firefox. The attached screen capture compared \GIF\01 Scenario+trim+border+rotate\0000.gif and \JPG\01 Scenario+trim+border+rotate\0000.jpg. Notice that the gif has extra large margins to the left and rights sides and a very small margin to the top (and bottom)? On web browsers, those extra margins appear as white. Perhaps it's transparency?

I ran the canti analyze option and it doesn't show the problem, so I don't know what to say about that (see attached txt).

This is bizarre. I finally got to see the weird sizes by bringing up a .gif in GiMP (kinda like the MS Paint of Ubuntu). It looks like the extra space that ImageMagick isn't reporting is transparent/empty (like the canvas is too big). And only .gif is affected (at least of the file formats tested). I will have to do some research to see why this is happening.

I'm guessing it's a problem with the way trimming works now. In order to force the trimming of a certain color, I have to add a 1px border of that color to the image and then trim. After trimming the algorithm then does a -1-1 repage of the image, but that may not be correctly adjusting the canvas size.

Originally Posted by badbob001

At least it shows the 800x800 issue with jpg/png in certain scenarios.

I see the problem. Look at the control case (#8) where Canti is simply resizing the image. The .jpg landscape image is being resized to 800x512, when you specified that it should be 600x800. This means that when ImageMagick encounters a resize in which the orientation is different (i.e. you have a landscape image, want to give it portrait orientation, but you still want to maintain scale and not rotate) then ImageMagick simply flips your input parameters to match the original orientation. I can probably fix this within Canti by having it do some checking before passing on the job to ImageMagick.

Originally Posted by badbob001

Have you tried viewing one of the gifs on your Kindle? I'm curious if it can be viewed currently without crashing the app.

I haven't turned on my Kindle in ages. If I get a chance I will try it, but if it crashes for you I'm guessing it will crash for me as well. Also, I don't see the benefit of using .gif for comic scans... they're freaking huge compared to .jpg, and personally I can't tell the difference in graphics quality.

[badbob001]

Yes, at this point, I think I'm going to stick with JPG, but I was just curious if GIF would be sharper. Curious how the PNGs created by Mangle are much smaller than even Canti's JPG output.

Any chance the next version will move the unarchiving temp folder from the parent_folder to the output_folder? This would allow me to work with read-only media, read-only network shares, and cut down the back and forth on network shares. Or perhaps have a setting to specify an over-riding temp folder.

Do you know if the .manga and .manga_save extensions are actually officially created by Amazon for the kindle, implying that they had manga support in mind?

Thanks!

[lilman]

Originally Posted by badbob001

Yes, at this point, I think I'm going to stick with JPG, but I was just curious if GIF would be sharper. Curious how the PNGs created by Mangle are much smaller than even Canti's JPG output.

I did some research on why the .gif images are coming out with weird sizes. Apparently when ImageMagick crops a .gif it leaves the canvas size of the original image. A lot of image viewers will ignore the transparent outer borders, but some don't. I don't have a solution for this since I like to leave all image processing to ImageMagick. For now I will probably leave it alone.

Originally Posted by badbob001

Any chance the next version will move the unarchiving temp folder from the parent_folder to the output_folder? This would allow me to work with read-only media, read-only network shares, and cut down the back and forth on network shares. Or perhaps have a setting to specify an over-riding temp folder.

I'm still thinking how to implement this. It's not as simple as saying "hey put parent_folder temp files over there instead of in parent_folder". Well, it would have been that simple in the older versions of Canti, but Canti's new brain makes everything more tricky (faster, but tricky). To be honest it's not on the top of my to-do list since I like having parent_folder writable (I'm constantly adding new comics to my collection, and also a lot of times I let Canti archive the parent_folder image folders, which requires write permission). In fact, a good chunk of code in Canti is dealing with an ever expanding series, which of course implies that you have write access to add new files to the series. But I guess if you have a comic library that you never update (like a collection of older comics) then you could make do with read-only access to parent_folder. I will let you know if I figure out a good way to do this.

Originally Posted by badbob001

Do you know if the .manga and .manga_save extensions are actually officially created by Amazon for the kindle, implying that they had manga support in mind?

I forgot where I learned about .manga and .manga_save... I think someone stumbled upon on it while playing with a Kindle. I don't think Amazon has publicized manga on the Kindle before. There was an article recently on how Yen Press and Dark Horse are going to be opening up digital manga stores in the next few months. And another company was opening a digital manga storefront as well... oh yeah, Square Enix: http://www.square-enix.com/na/manga/. I'm guessing Amazon thought that the digital manga market would eventually catch on and so they implemented the bookmark system early.

[badbob001]

Does the conversion process involve multiple convert commands or just a single long convert command? If the former, then perhaps it would be best to pick a lossless image format to hold the intermediate results (png?). This may fix the gif issue since you will only need to output to gif at the very end, similar to pdf. And maybe the files will be smaller and less blurry since you're not re-compressing over and over again.

Does the @auto_split_buffer parameter limit itself so that the final split pages are not wider than the target resolution? So if I set it to the maximum (1.0), it'll only include as much of the other page as the available space. This would help eliminate seeing the fill border on split pages.

Thanks!

[lilman]

Originally Posted by badbob001

Does the conversion process involve multiple convert commands or just a single long convert command? If the former, then perhaps it would be best to pick a lossless image format to hold the intermediate results (png?). This may fix the gif issue since you will only need to output to gif at the very end, similar to pdf. And maybe the files will be smaller and less blurry since you're not re-compressing over and over again.

Depends on the options you set. The main image processing (resize, adjust dpi, change image format, etc.) is done in one command. If you enabled trimming then that is done in a separate command right before the main processing, but the trimming is performed using the same image format as the input image. Since trimming is merely an intelligent crop, I think there should be no quality loss even for lossy formats like .jpg. Auto splitting landscape pages (which is done before trimming) also boils down to an intelligent crop command, so again no quality loss. There is also a preprocessing step that uses the same image format as the original image. It overwrites the original image, and merely attempts to fix any erroneous data in the image... no real processing is done, so there shouldn't be any quality loss.

So it all comes down to what algorithms ImageMagick uses for cropping. I know there are lossless crop algorithms for .jpg, and I assume that since ImageMagick is such an amazing image processing program that they would use the best available algorithms.

Originally Posted by badbob001

Does the @auto_split_buffer parameter limit itself so that the final split pages are not wider than the target resolution? So if I set it to the maximum (1.0), it'll only include as much of the other page as the available space. This would help eliminate seeing the fill border on split pages.

@auto_split_buffer is the % of the other split page to include with the current split page. So if you set it to 1.0, then 100% of the other split page will be included in the current split page (i.e. you will get the original unsplit image, but you'll get it twice since each split page will include the entire other split page).

If you use fill border, you would still want it for split pages since the purpose of fill border is to make the page fit the screen perfectly (so the need for fill border is independent of the contents of the page). The purpose of splitting pages is to that the page can better fill up the screen, making it easier to read. Fill border occurs after image resizing so it won't affect how much of the screen the real image data takes up (it just fills in what would have been empty space with whatever border color you specify).

[badbob001]

Ok, but I just didn't want an arbitrary auto_split_buffer percentage to push the final image beyond the target dimensions. It would be nicer if the overlap buffer was dynamically determined based on how much leftover space there will be after the split.

Say the target device is 600x800 and there is a 1000x800 page. A non-buffered split would be 2 x 500x800 pages. With bordering, it'll add 50 pixels to the left/right sides of each page. It would look better to only add the border to the sides away from dividing edge:
[border][left page] [right page][border]
Now it is very clear visually which side of each page is the divider.

Another idea would be a dynamic overlap buffer that would calculate how much room is left (600-1000/2=100) and add that difference to to the left and right pages so we don't have to deal with borders for splits.
[left page][100 pixel overlap] [pixel overlap][right page]

Just some suggestions. App is still awesome.

[lilman]

Originally Posted by badbob001

Ok, but I just didn't want an arbitrary auto_split_buffer percentage to push the final image beyond the target dimensions. It would be nicer if the overlap buffer was dynamically determined based on how much leftover space there will be after the split.

Say the target device is 600x800 and there is a 1000x800 page. A non-buffered split would be 2 x 500x800 pages. With bordering, it'll add 50 pixels to the left/right sides of each page. It would look better to only add the border to the sides away from dividing edge:
[border][left page] [right page][border]
Now it is very clear visually which side of each page is the divider.

Sorry, I don't see the point of [border][left page] [right page][border]. The auto split algorithm should only be splitting pages which are independent from each other. If it decides to split them then, it doesn't matter which page was left or right before the split (since they are independent, they could have easily been on opposite sides of the same piece of paper, know what I mean?). All that matters is maintaining proper page order during a split.

Btw, the split doesn't necessarily occur in the dead center of the image. The split algorithm starts in the center and moves outwards to handle cases where the two-page scan isn't perfectly centered. The split buffer then takes the specified percentage of the other split page. So if a page with a width of 1000 is split at x=600 and the split buffer is set to 0.1, then the left page (with a width of 600) will get a 40px buffer from the right page (width of 400), and the right page gets a 60px buffer from the left page. This all happens prior to resizing.

Originally Posted by badbob001

Another idea would be a dynamic overlap buffer that would calculate how much room is left (600-1000/2=100) and add that difference to to the left and right pages so we don't have to deal with borders for splits.
[left page][100 pixel overlap] [pixel overlap][right page]

I like that idea, I'll try to implement it.

[badbob001]

My point regarding borders only applies to splits for landscape spreads. In print, it's normal for single pages to have margins on both sides. But for two-page spreads, the margins are only located on the opposite far sides. If you take apart the two-page spread and add in the inside margins, then the pages would look like stand-alone pages and be confusing visually. By having the margin on one side gives one a visual clue to the fact that the current page is part of a spread and the next section continues on the side where there is no margin.

It's just a visual design thing and not worth worrying about.

[lilman]

Originally Posted by badbob001

My point regarding borders only applies to splits for landscape spreads. In print, it's normal for single pages to have margins on both sides. But for two-page spreads, the margins are only located on the opposite far sides. If you take apart the two-page spread and add in the inside margins, then the pages would look like stand-alone pages and be confusing visually. By having the margin on one side gives one a visual clue to the fact that the current page is part of a spread and the next section continues on the side where there is no margin.

It's just a visual design thing and not worth worrying about.

The auto split algorithm was designed for only splitting landscape scans where the pages are independent, but I'm guessing you use it to split all landscape scans (setting the strictness values to 0). I can see in that case you would like a visual cue of what side each page is from. If I have time I will see if I can add an option for that.

[lilman]

Here's an update on what I've been working on for the next release of Canti:

-[code done] Contrast adjustment (user requested)
Ability to adjust the contrast of images. Personally I don't like this feature... contrast is one of those things you shouldn't leave up to a batch job, but instead should be done manually on a per image basis. But someone requested it, and it is a feature of ImageMagick so it wasn't hard to add to Canti. The same user also requested implementing ImageMagick's auto-gamma feature, but I tried it out and it really doesn't work well on manga scans so it will not be included.
-[code done] Handle read-only parent_folder (user requested)
Canti normally uses parent_folder for some temporary files (namely unarchiving zips/rars in parent_folder, or converting a pdf in parent_folder to an image folder), but this isn't possible if parent_folder is read-only. I was afraid this would be difficult to implement, but it turned out to be like a 30 minute job.
-[dropped] When using @auto_split_landscape_scans and @add_border, use split_buffer as horizontal border (user requested)
I am not sure this can be implemented since adding borders occurs after landscape splitting and major image processing. In other words, by the time you want to use the split_buffer as a border it may be no longer possible to do so. I will think more about it, there maybe a clever way to pull this off.
Update: This really isn't possible short of reprocessing from the unsplit scan. This feature has been dropped.
-[code not started] Directional add_border (user requested)
Instead of centering the scan when adding a border, add horizontal border to simulate the original physical page side (left or right) of the scan. This of course could only be used on landscape split scans since it is impossible to determine the original page side of a scan otherwise.
-[code not started] Sharpen image
I'm interested in exploring ImageMagick's sharpening abilities to see if they should be included in manga processing.

Well that's about it. Also some random bug fixes (like the Kindle bookmark bug, which has already been fixed).

[badbob001]

Originally Posted by lilman

-[code done] Contrast adjustment (user requested)
Ability to adjust the contrast of images. Personally I don't like this feature... contrast is one of those things you shouldn't leave up to a batch job, but instead should be done manually on a per image basis. But someone requested it, and it is a feature of ImageMagick so it wasn't hard to add to Canti. The same user also requested implementing ImageMagick's auto-gamma feature, but I tried it out and it really doesn't work well on manga scans so it will not be included.

Does this have anything to do with Color Balancing? In Windows app CDisplay, the Color Balance feature is described as such:

The Colour Balance option applies a histographic colour balance to the image. In other words, it analyzes the image and spreads out the colours so that the lightest is white (255, 255, 255) and the darkest is black (0, 0, 0).

There is also a discards per 1000 value. This allows the specification of a number of 'pixel discards' per 1000 when producing the colour table. This allows you to stop the phenomenon of the odd bright white or jet black pixel from skewing the averages - these extreme valued pixels are simply discarded from the average calculation. Put simply, it allows a glint of white or fleck of black which is not part of the image to be discarded.

Originally Posted by lilman

-[dropped] When using @auto_split_landscape_scans and @add_border, use split_buffer as horizontal border (user requested)
I think this cannot be implemented since adding borders occurs after landscape splitting and major image processing. In other words, by the time you want to use the split_buffer as a border it may be no longer possible to do so.

I'm not sure how auto_split_buffer size relates with solid color borders. But it is a brain twister since you calculate the auto_split_buffer and split BEFORE your resize BUT only after you resize do you realize if the auto_split_buffer amount is too much (ie: top/bottom borders introduced due to split image being too wide). Would the following logic work to calculate how much buffer is needed?

/* Resizing an image to a target size while maintaining the correct aspect ratio involves calculating two ratios and deciding which ratio gets us to the target size without going over: */
ratio1 = target_width/source_width
ratio2 = target_height/source_height
ratio1width = source_width * ratio1
ratio1height = source_height * ratio1
ratio2width = source_width * ratio2
ratio2height = source_height * ratio2

/* I'm ASSuming other test cases are not needed to be checked... */
if (ratio1width <= target_width) and (ratio1height <= target_height) {
ratioToUse = ratio1
} else {
ratioToUse = ratio2
}

/* With the correct ratio found, we can calculate the width the source will be reduced to and determine the leftoverSpace of the target, which we can use to fill with background color. But instead, we will divide that number with the ratio to calculate the leftoverSpace of the source image. And that number is the auto_split_buffer amount to use (in pixels). */
leftoverSpace_target = target_width - (ratioToUse * source_width)
leftoverSpace_source = leftoverSpace_target / ratioToUse

bufferwidth = leftoverSpace_source

Example time:


I hope that made sense.

[lilman]

Originally Posted by badbob001

Does this have anything to do with Color Balancing?

I think color balancing is more related to gamma than contrast. Contrast is how distinct things in the image look in comparison to other things. For example, after increasing contrast what was once a mere speck looks a lot more noisy. I don't like auto balancing color or contrast since I think it is something that should be left to the human eye to decide, but it was requested so I included it.

Originally Posted by badbob001

I'm not sure how auto_split_buffer size relates with solid color borders. But it is a brain twister since you calculate the auto_split_buffer and split BEFORE your resize BUT only after you resize do you realize if the auto_split_buffer amount is too much (ie: top/bottom borders introduced due to split image being too wide).

Yes, the auto_split_buffer is calculated and used before resizing; however, once you say that you want to use the split_buffer, it gets used. There is no later processing to say "the split buffer was too much, let's crop it down". The split_buffer is not treated as a border, it becomes part of the image.

To sum up the logic since it can be a bit confusing:
1) Find the x coordinate for a page split
2) Split the image. If the user specified a split buffer, move the x coordinate a bit for each page side before performing the split (so each page may have a different x split coordinate when using a split buffer)
3) Process the split pages like normal (trim, resize, etc.)
4) Add borders if the user requested them and they are needed

That should explain why I couldn't implement using split_buffer as a border. Once a landscape scan has been split in two, each page is treated independent of the other. Each page may be trimmed, resized, etc. differently than the other. You can't go back later and say "I want to use some of you as a border for this other page" since there is no guarantee that they will still be properly aligned. The alternative is to know ahead of time how big the image is going to be after resizing, and this would be possible if ImageMagick didn't handle the trimming (i.e. Canti could predict a simple image resize, but not including an unknown amount of trimming).

I hope that explains it.

[hupple]

Originally Posted by lilman

Here's an update on what I've been working on for the next release of Canti:

-[code done] Contrast adjustment (user requested)
Ability to adjust the contrast of images. Personally I don't like this feature... contrast is one of those things you shouldn't leave up to a batch job, but instead should be done manually on a per image basis. But someone requested it, and it is a feature of ImageMagick so it wasn't hard to add to Canti. The same user also requested implementing ImageMagick's auto-gamma feature, but I tried it out and it really doesn't work well on manga scans so it will not be included.

I have tried -auto-gamma. -auto-level and -contrast. I agree that auto-gamma doesn't work well. But auto-level is better.