RasterFarian 3 discussion + feature suggestions

[alex_d]

I'd like to make a new release of RasterFarian, written in .NET 3.5 with a nice gui. There's opportunity for new functionality.

What do people do with pdfs and ebooks? What would people like rasterfarian to do?

Something I always wanted to implement was manual cropping, with a fancy interface for zooming in on the pages. Another programmer I remember implemented an autocropping feature that removed space between lines. A lot of people said they worked with scanned two-page book images, and wanted to clean them up and split them vertically. Something I've fantasized about is a cpu-sharing network for doing conversions, with people earning points they could redeem by tapping the grid. Some stuff isn't actually as hard to implement with modern programming technology as it used to be. Be creative, and we'll sort out what can be done.

(P.S. if you know of any image-processing steps that could improve RasterFarian's core transformation code, that would be welcome too.)

[athlonkmf]

Ah cool. You're picking things up again?

I've added these to 2.5:
Chose between 4/8 bits.
Batch working.


What would really help.
Drag and drop complete folders. So you only need to edit title (which should come from filename) and author.
Book thumbnail.
Parallel processing (while one lrf is starting, the images for another can be processed

[ndrake]

It would be cool if it worked on more than just Windows!

[Valloric]

Originally Posted by alex_d

...written in .NET 3.5...

Please target it to .NET 2.0. I seriously doubt you need more than that. Always target the lowest numbered version that has all the features you want. With 3.5, your users will most definitely have to download and install the framework installer (even with Vista), whereas most computers today have .NET 2.0 by default.

Targeting it to 3.5 and using only features from 2.0 and lower has absolutely no advantages, but has several disadvantages.

[alex_d]

Originally Posted by Valloric

Please target it to .NET 2.0. I seriously doubt you need more than that. Always target the lowest numbered version that has all the features you want. With 3.5, your users will most definitely have to download and install the framework installer (even with Vista), whereas most computers today have .NET 2.0 by default.

Targeting it to 3.5 and using only features from 2.0 and lower has absolutely no advantages, but has several disadvantages.

Why do you think i'd target it at 3.5 and not use its features? Thing is I definately want to learn/use WPF. It's a beautiful framework that's half a decade ahead of winforms or anything else.

I've previously thought of doing a 2.0 class library, and then wrapping it in a 2.0 command-line tool along with the 3.5 gui. I would be careful to make the 2.0 parts portable with posix filenames, etc. I was going to use the same pipeline as 2.5, going out to the shell and and running the same command-line utilities.

What I found out recently, though, is that 3.5 SP1 is introducing a new feature that basically lets you use pixel shaders in WPF. If I offload the computations to the video card, I could really speed up rasterfarian's sorry performance. I have a bunch of experience writing pixel shaders, and it shouldn't be too much work if I can find what equations my tools use. Also, I found out that going out to the shell is a fairly dirty and/or inflexible business from C#.

What I might be able to do is to rewrite 2.5 into a cross-platform .net executable. Is there a demand for that?

[alex_d]

Originally Posted by athlonkmf

Ah cool. You're picking things up again?

I've added these to 2.5:
Chose between 4/8 bits.
Batch working.


What would really help.
Drag and drop complete folders. So you only need to edit title (which should come from filename) and author.
Book thumbnail.
Parallel processing (while one lrf is starting, the images for another can be processed

Thanks for helping maintain rasterfarian.

Doing complete folders and queuing up conversions would definately be features. You say you specifically want to drag folders and files? I actually forgot all about dragging and dropping.

What do you mean by book thumbnails?

Parallel processing: I think rasterfarian 2.5 is one of the only multi-threaded batch files on earth. It runs on dual-core cpus out of the box, and you can configure it to launch more threads as well (the threads normally use a lot of ram, so the default's low). the next version would also be multithreaded. I was also toying with the idea of making rasterfarian distributed, so you could have all the pcs in your house help you process. But that probably won't be in the next version.

[lovebeta]

Originally Posted by alex_d

Why do you think i'd target it at 3.5 and not use its features? Thing is I definately want to learn/use WPF. It's a beautiful framework that's half a decade ahead of winforms or anything else.

I've previously thought of doing a 2.0 class library, and then wrapping it in a 2.0 command-line tool along with the 3.5 gui. I would be careful to make the 2.0 parts portable with posix filenames, etc. I was going to use the same pipeline as 2.5, going out to the shell and and running the same command-line utilities.

What I found out recently, though, is that 3.5 SP1 is introducing a new feature that basically lets you use pixel shaders in WPF. If I offload the computations to the video card, I could really speed up rasterfarian's sorry performance. I have a bunch of experience writing pixel shaders, and it shouldn't be too much work if I can find what equations my tools use. Also, I found out that going out to the shell is a fairly dirty and/or inflexible business from C#.

What I might be able to do is to rewrite 2.5 into a cross-platform .net executable. Is there a demand for that?

I support you. I think anyone who visits this forum and use your software will not be bothered by installing an api library.

Featurewise, I would like see the support of more formats. At lease mobipocket since I am a Kindle user too. This is not hard to implement and the key, I think, is still finding a high quality raster algorithm that is custom-made to the E-ink display. The current thickening is still as crisp as the native rendering. Are there any subpixel smoothing tricks?

[athlonkmf]

Originally Posted by alex_d

Thanks for helping maintain rasterfarian.

Nah, I just did what I needed.

Doing complete folders and queuing up conversions would definately be features. You say you specifically want to drag folders and files? I actually forgot all about dragging and dropping.

When I wanted to make a .NET version, I've made an interface where I could just drop folders and files. The program would list the files, and I could edit the metainfo I want to put in the lrf in a spreadsheetway.
(never went further than a halfworking prototype though, didn't had the time)

What do you mean by book thumbnails?

The LRF-format has an option to use one thumbnail-image. Bookdesigner can set something like that.
It won't be viewable in the sony reader, but it's available in the readersoftware. And who knows, maybe in the future a firmware would be out which would display the coverthumbnails for each book.

Parallel processing: I think rasterfarian 2.5 is one of the only multi-threaded batch files on earth. It runs on dual-core cpus out of the box, and you can configure it to launch more threads as well (the threads normally use a lot of ram, so the default's low). the next version would also be multithreaded. I was also toying with the idea of making rasterfarian distributed, so you could have all the pcs in your house help you process. But that probably won't be in the next version.

I've set my version to run at 8 cores, but img-processing isn't even the real slowdown. It makes optimal use of the CPU, but eventually creating an LRF could take hours and another queued item can't start before that lrf is created.

[alex_d]

Originally Posted by lovebeta

I support you. I think anyone who visits this forum and use your software will not be bothered by installing an api library.

well really, microsoft's automated that pretty well. the rasterfarian setup file will just download and install the right version of .net if it has to.

Featurewise, I would like see the support of more formats. At lease mobipocket since I am a Kindle user too. This is not hard to implement and the key, I think, is still finding a high quality raster algorithm that is custom-made to the E-ink display. The current thickening is still as crisp as the native rendering. Are there any subpixel smoothing tricks?

More formats is something I was planning on. In terms of input, the ones most relevant to myself are PDF, CHM*, DJVU, and picture formats like JPG or TIFF. People should suggest what formats they'd like. The trick to support any input format is to have an easy way to render it to a picture. (Ie a command-line program or a library.) For output, I would obviously support the Reader and as many other devices as I could. The file formats are a question. Since each device usually supports several (the Kindle does too, right?) it might make sense to just offer a generic format (tuned for the device) instead of the proprietary one. This isn't too clear an issue, because not all formats will work as well. On the Reader, for example, PDF, even if it's just used to bundle images, is slower and crops the picture vs BBeB. If anyone is knowledgeable about their device please share what you know.

For the Kindle, the currect post-processing should work just fine. It's just using the same one as the Reader, right? For an LCD device, I'd definately have to tweak it. eInk is a bit blurry and low-contrast, so I found that an especially harsh filter looks best. (This is why I warn, in the preview, not to judge the picture by how it looks on your monitor.)

Subpixel smoothing is only for color screens. I do use anti-aliasing.

*(ms help files... it's weird, but a lot of books come in this format since it's basically a compressed single-file website with a simple ui)

Originally Posted by athlonkmf

The LRF-format has an option to use one thumbnail-image. Bookdesigner can set something like that.
It won't be viewable in the sony reader, but it's available in the readersoftware. And who knows, maybe in the future a firmware would be out which would display the coverthumbnails for each book.

Oh. Yeah, i've wondered what that was in the lrf sources. Never realized what it does since I don't use Connect. Ok, I'll see if I can put that in.

I've set my version to run at 8 cores, but img-processing isn't even the real slowdown. It makes optimal use of the CPU, but eventually creating an LRF could take hours and another queued item can't start before that lrf is created.

Right, the dumb lrs2lrf bug. Actually, I've already got MSH_LRSparser working and it shouldn't have this problem.

[wallcraft]

Originally Posted by alex_d

For output, I would obviously support the Reader and as many other devices as I could. The file formats are a question. Since each device usually supports several (the Kindle does too, right?)

The only DRM-free ebook format with images the Kindle supports is MOBI. This is also available on iLiad and Cybook, but the image size would need to be customized for each. See All on the Mac: PDFs with Formatting!. In that thread I concluded that 520x640 was the maximum image size in MOBIs on the Kindle, but see also Optimizing images for use in Kindle where 450x550 is suggested as a maximum.

[jakeluck]

Alex, how about an interactive way of cropping and previewing the rendered output. e.g. for a given page, let the user to preview what does boldness 4,5,6 and 7 look like side by side, ditto on edge enhancement, mockup attached.
what do you think?

oh and djvu support please.

[jakeluck]

Originally Posted by alex_d

A lot of people said they worked with scanned two-page book images, and wanted to clean them up and split them vertically.

thought to share as I have figured out, a way to accomplish that with the existing version 2

1) rotate the 2 page pdf 90 degrees clockwise
2) render using half page mode
3) when viewing, set the reader in landscape mode

this essentially simulates the book spliting, the page forward
and backword all work as expected, with only the status bar
beging vertical.

[athlonkmf]

Originally Posted by alex_d

Right, the dumb lrs2lrf bug. Actually, I've already got MSH_LRSparser working and it shouldn't have this problem.

I believe MSH_LRSparser had a problem with TOC. It doesn't support the title and author-info from the LRF when I last tried it (>1 years ago or so)

[alex_d]

Yes, finding the right resolution will need a bit of work. I did it for the Reader using a test pattern and patience. I'll need a volunteer to do this for the Kindle and others, but that's for later.

Interactive cropping is something I'm really hoping for. We'll see how much .net skillz it'll take.


Ok, some news: I start playing around writing code, and tried that GPU-offloading I was talking about. Doing it frankly sounded a bit crazy at first, but I've actually got it working (at least, I've got the dilate filter working). This was way easier than I expected. And when the rendering happens on the GPU, it runs orders of magnitude faster. BUT, as soon I want to save the image instead of just display it onscreen, it automatically switches to software rendering.

Software rendering, despite the whole circuitous route, is actually no slower than calling netpbm to handle the dilate, but neither is it faster. So much for my hopes. Still, the question now is do I continue with this path of using bleeding-edge tech instead just a command-line tool or plain C code? I think so. First of all, this shader business is actually easy. Going out to the shell is dirty, and writing manual code also takes more work. Even if the pixel shaders are software rendered, I don't have to worry about shuffling data or even having a for loop since it's all being automated beneath the surface of WPF. Second of all, at some point in the future GPU acceleration might get turned on. Third, using .net 3.5 tech means it's easier to plug it back into the 3.5 UI to have interactive previewing and cropping.


Tomorrow I'll try to package it in an install program.

[alex_d]

Additionally, I need a new PDF rendering/metadata-reading library. I think I'm going to say goodbye to ghostscript because of its slow performance and incorrect rendering of some pdfs. I will try using the Acrobat SDK, which should be able to do what I need and with a pretty good interface. Foxit has its own SDK, but it's not free. Anyone know of any other contenders?