View Full Version : impmake: automated creation of IMP books via the command line


ashkulz
08-26-2008, 05:08 AM
NOTE: impmake has now been renamed to ebookutils, and is hosted at http://gitorious.org/projects/ebookutils using Git, which should easily allow other developers to make changes. I plan to merge impserve, impmake, 2 new python utilities which replace ebw1150+rebcomm and pdfread under this umbrella.

This is impmake, a utility for creating ebooks for the 1100/1150/1200 from HTML source files. It requires the installation of eBook Publisher (http://ebooktechnologies.com/support_publisher_download.htm). In case this is not installed, you may get an error:


Execution failed at src/impmake.c:159 while attempting to execute
dhCreateObject(L"SBPublisher.Project", NULL, &impProject)

To compile the code, you will need MinGW with MSys (http://mingw.org). Just switch to the directory where you extracted the files, and type "make" from Msys command prompt (There's a precompiled version available in the download).

Usage: impmake [-OPTIONS] FILES [...]

-h, --help Show this help message.
-s, --save Specify the project to save as.
-a, --author Specify the book's author.
-t, --title Specify the book's title.
-c, --category Specify the book's category.
-d, --out-dir Specify the output directory.
-n, --name Specify the book name.
--isbn Specify the book's ISBN
--pubdate Specify the book's publishing date
--publisher Specify the book's publisher
--language Specify the language for the book.

--1100, --1150, --1200, --oeb
Specify the target device to use for book creation.

--zoom-small, --zoom-large, --zoom-both
Specify the zoom states to be supported in the book

--no-underline Specify that links should not be underlined.
--no-img-conv Specify that image auto conversion to JPG should be disabled.
--no-img-scale Specify that image pre scaling should be disabled.
--compress Specify that the compression is to be used.
--keep-anchors Specify that link anchors should be kept in output.

I plan to host this later at SourceForge or something similiar, after trying to coordinate a super-archive of all the tools currently available for the REB 1100/1150/1200 eg. libreb, rbmake, rebcomm, eb1150, and the planned-for browsing proxy.

Changes in 0.2:
- convert relative to absolute paths when referring to files/directories
- add a --log option which generate a log from the publisher itself (can be useful for debugging errors)

Changes in ebookutils-0.3:
- impmake: add an -f option to specify the files to be included, with one file per line
- impserve: handle paging of booklist requests
- impserve: handle downloading of IMP books which are sent as an attachment (direct download of IMP from mobileread works)
- impserve: implement a plugin system and provide 2 default plugins to change the URL/content before being sent to the device.
- impserve: refactored the code heavily, please run impserve/run.py from python.

To get it working in WINE, you'll need to perform the following:

Remove or backup your existing wine folder (e.g. using rm -fR ~/.wine)
Download winetricks (http://wiki.winehq.org/winetricks) and run "sh winetricks vcrun6"
Install eBook Publisher (http://ebooktechnologies.com/support_publisher_download.htm)
run "sh winetricks dcom98" and click Yes twice.
Now you can try running wine impmake.exe <other-params>

nrapallo
08-26-2008, 09:22 AM
Great effort!

I used to use rbmake with my REB1100 all the time because of its simplicity and power. You seem to have duplicated that with impmake.

I tried compiling it under Cygwin (what I have installed right now in WinXP) and got an error since I don't have "tchar.h".

I know it's in "/usr/include/mingw" and it's not a good idea to mix these two installations, BUT is there a workaround to using tchar.h for the moment?

I don't feel like installing mingw right now as I finally have my Cygwin implementation setup again. Do you know the full Cygwin installation downloaded over a GIG of stuff and took several hours to complete? :angry:

Let me play with this a bit later today before asking any more questions... :grin2:

ashkulz
08-26-2008, 11:08 AM
I know it's in "/usr/include/mingw" and it's not a good idea to mix these two installations, BUT is there a workaround to using tchar.h for the moment?

I don't feel like installing mingw right now as I finally have my Cygwin implementation setup again. Do you know the full Cygwin installation downloaded over a GIG of stuff and took several hours to complete? :angry:

You might want to use the -mno-cygwin (http://gcc.gnu.org/ml/gcc/2001-01/msg00205.html) flag for getting it to compile. MinGW is just a ~12Mb download, so I'd recommend you download it anyway -- no dependency on cygwin1.dll. Also, I do hope you noticed that the compiled EXE is in the archive anyway so you don't need to compile it again :-)

I plan to work on PDFRead in the next few weeks, and one of the first things I will do is use this instead of directly using COM from Python -- which should reduce the installation size quite a bit. I think you can take a similiar approach for the html2mobi which is done in Perl, I think...

DixieGal
08-26-2008, 11:12 AM
Thank you! I haven't the foggiest idea what to do with it, but I'm grateful for everything you and Nick are doing to keep IMP format alive so that we can continue using our beloved EB=1150's.

:thanks:

DaleDe
08-26-2008, 11:14 AM
This is impmake, a utility for creating ebooks for the 1100/1150/1200 from HTML source files. It requires the installation of eBook Publisher (http://ebooktechnologies.com/support_publisher_download.htm). In case this is not installed, you may get an error:

I plan to host this later at SourceForge or something similiar, after trying to coordinate a super-archive of all the tools currently available for the REB 1100/1150/1200 eg. libreb, rbmake, rebcomm, eb1150, and the planned-for browsing proxy.

Also, this should work in Wine as-is so that one can have an all-linux solution for your 1150/1200.

You might want to use the development environment provided here at mobileread. Click the DEV HUB entry at the top of the screen. Sounds great by the way.

Dale

DixieGal
08-26-2008, 11:17 AM
Thank you! I haven't the foggiest idea what to do with it, but I'm grateful for everything you and Nick are doing to keep IMP format alive so that we can continue using our beloved EB=1150's.

:thanks:


And Dale, thank you also!!

nrapallo
08-26-2008, 11:44 AM
You might want to use the -mno-cygwin (http://gcc.gnu.org/ml/gcc/2001-01/msg00205.html) flag for getting it to compile. MinGW is just a ~12Mb download, so I'd recommend you download it anyway -- no dependency on cygwin1.dll.Thanks for the heads up! It will be the next install on my list. I really hate having to also use that cygwin1.dll. Also, I do hope you noticed that the compiled EXE is in the archive anyway so you don't need to compile it again :-)First thing I did after unzipping your file was to rename your executable so that I could get it compiled. It was almost a knee-jerk reaction since I've done this a thousand times before; download someone else's code and compile it! :grin2: I still haven't used your .exe because I'm too busy solving my (unimportant) compile problem. Boy do I have my priorites all screwed up! :rofl:

I plan to work on PDFRead in the next few weeks, and one of the first things I will do is use this instead of directly using COM from Python -- which should reduce the installation size quite a bit.Great News! Will you host it on SourceForge as I think the DEV HUB here doesn't allow co-developers (hint-hint) as suggested by DaleDe. I think you can take a similiar approach for the html2mobi which is done in Perl, I think...

I think I'll wait for you to "pioneer" that method, and then I will follow (copy) with something similar in Mobi2IMP!

We (actually Ret) just got the EBW1150 proxy browser working (check here (http://www.mobileread.com/forums/showthread.php?p=241188#post241188)) so I hope this will aid in your proxy browser/librarian effort!

We may need a "Developer's Corner" sub-forum in the Fictionwise forum now...

Thanks for all your help keeping the .imp format alive!!!

ashkulz
08-27-2008, 04:25 AM
Made a new release (impmake-0.2) with the following changes:

- convert relative to absolute paths when referring to files/directories
- add a --log option which generate a log from the publisher itself (can be useful for debugging errors)

Note that 0.2 doesn't work in Wine yet, it fails at ValidateManifest. I am still investigating that.

derrell
08-27-2008, 07:52 AM
I am getting the same sort of error with wine but with version .01 I think. Downloaded it yesterday.

Execution failed at src/impmake.c:191 while attempting to execute
dhCallMethod(impBuilder, L".ValidateManifest(%o)", impProject)


As the versions go up you might want to add a --version switch as I'm not sure which one I have, but I think it is .01.

Ok looked back at the first post where the files are and it is .01 that have as .02 is labeled .02. Brain isn't working today.

GeneS
08-27-2008, 09:19 AM
Wonderful.
It works like a charm as far as I have been able to go -- and FAST, too. It could be that I'm not crossing my fingers as I proceed, but images don't show up in the resulting .imp file regardless of whether they're in the directory with the html file or not. Apparently there's something I've overlooked. The log file indicates:

Error: A link to a non-existent anchor ("\H:\HTML2imp\images\Illus_1.jpg") was found. Ignoring link ...

Could it be the beginning backslash that it inserts into the search path?

Thanks for your work on this.

Gene

ashkulz
08-27-2008, 11:30 AM
The log file indicates:

Error: A link to a non-existent anchor ("\H:\HTML2imp\images\Illus_1.jpg") was found. Ignoring link ...

Could it be the beginning backslash that it inserts into the search path? Can you post a zip of your HTML here (or PM me with a link), so that I can debug it?

And yes, both versions don't work in WINE (but eBook Publisher does). Apparently, it is something to do with the path-translation WINE does which causes the COM functions to return an E_FAIL (generic error) -- at least, that's what I think for now.

DaleDe
08-27-2008, 11:45 AM
I asked about co-author in the DEV HUB and it is already implemented according to Alex.

The person who creates the project (the "owner") can give other people (who must be registered on the dev hub) full access to the project's repository.

He needs to go to Admin -> Subversion Access.

nrapallo
08-27-2008, 11:46 AM
I asked about co-author in the DEV HUB and it is already implemented according to Alex.

The person who creates the project (the "owner") can give other people (who must be registered on the dev hub) full access to the project's repository.

He needs to go to Admin -> Subversion Access.

Wow, where did that upgrade slip in. Good to know, Dale!

DaleDe
08-27-2008, 11:46 AM
Wonderful.
It works like a charm as far as I have been able to go -- and FAST, too. It could be that I'm not crossing my fingers as I proceed, but images don't show up in the resulting .imp file regardless of whether they're in the directory with the html file or not. Apparently there's something I've overlooked. The log file indicates:

Error: A link to a non-existent anchor ("\H:\HTML2imp\images\Illus_1.jpg") was found. Ignoring link ...

Could it be the beginning backslash that it inserts into the search path?

Thanks for your work on this.

Gene

Certainly, backslashes should never be used in html.

Dale

ashkulz
08-27-2008, 02:19 PM
I am getting the same sort of error with wine but with version .01 I think. Downloaded it yesterday.

Execution failed at src/impmake.c:191 while attempting to execute
dhCallMethod(impBuilder, L".ValidateManifest(%o)", impProject)


As the versions go up you might want to add a --version switch as I'm not sure which one I have, but I think it is .01.

Ok looked back at the first post where the files are and it is .01 that have as .02 is labeled .02. Brain isn't working today.

Ok, here's a quick way to get it working in WINE.

Remove or backup your existing wine folder (e.g. using rm -fR ~/.wine)
Download winetricks (http://wiki.winehq.org/winetricks) and run "sh winetricks vcrun6"
Install eBook Publisher (http://ebooktechnologies.com/support_publisher_download.htm)
run "sh winetricks dcom98" and click Yes twice.


Now you can try running wine impmake.exe <other-params>

GeneS
08-27-2008, 03:27 PM
Can you post a zip of your HTML here (or PM me with a link), so that I can debug it?

And yes, both versions don't work in WINE (but eBook Publisher does). Apparently, it is something to do with the path-translation WINE does which causes the COM functions to return an E_FAIL (generic error) -- at least, that's what I think for now.

Thanks, ashkulz:
Here's a copy of the zip file.

Gene

ashkulz
08-27-2008, 04:23 PM
Thanks, ashkulz:
Here's a copy of the zip file. Hmm, this looks like a bug in eBook Publisher itself. If I manually add the file to the publisher, the images were missing in that case too. If there are any bugs in the automation code, impmake will inherit them too :(

Looks like it works if you put the images in the same directory, but not different directories. I'll do some more testing tomorrow, but if you are able to produce a OPF file which creates images which impmake doesn't please post just the project file here. (You can use the -s option to auto-create a project file when building the IMP).

nrapallo
08-27-2008, 05:06 PM
Thanks, ashkulz:
Here's a copy of the zip file.

Gene

OK I found your html problems, yes problems.

You need to change the lines around each <img src> tag to:
-eliminate the <a href> & </a> tags surrrounding each <img src> tag .
-delete the width=40% attributed within the <img src> tag.

Now it will convert properly with eBook Publisher. Haven't tried it yet with impmake, but there should be no html issues now.

GeneS
08-27-2008, 05:56 PM
OK I found your html problems, yes problems.

You need to change the lines around each <img src> tag to:
-eliminate the <a href> & </a> tags surrrounding each <img src> tag .
-delete the width=40% attributed within the <img src> tag.

Now it will convert properly with eBook Publisher. Haven't tried it yet with impmake, but there should be no html issues now.


Thanks. Making the changes worked like a charm. Now the question is: Why? since I was taught to surround internal image links with the <a href></a> tags.
Or is it just a peculiarity of eBook Publisher?

In any case, I (and I hope other EBookwise users) am very grateful that you and others continue to make the reader eminently useable.

Gene :thanks:

DaleDe
08-27-2008, 07:13 PM
Thanks. Making the changes worked like a charm. Now the question is: Why? since I was taught to surround internal image links with the <a href></a> tags.
Or is it just a peculiarity of eBook Publisher?

In any case, I (and I hope other EBookwise users) am very grateful that you and others continue to make the reader eminently useable.

Gene :thanks:

the href is for the case where you want the image to be clickable. That is the only reason to add it although it will place a border around the image this can be done with different methods.

Of course eBook Publisher does not support clickable images. There are lots of html constructs that are not intended for eBook use.

Dale

nrapallo
08-27-2008, 08:44 PM
Thanks. Making the changes worked like a charm. Now the question is: Why? since I was taught to surround internal image links with the <a href></a> tags.
Or is it just a peculiarity of eBook Publisher?

In any case, I (and I hope other EBookwise users) am very grateful that you and others continue to make the reader eminently useable.

Gene :thanks:

Surrounding the <img> tag is often used to display a thumbnail picture inline and then once clicked, to display a full-size/larger picture. It has a purpose, but not for .imp ebooks.

As Dale mentioned, eBook Publisher is not so accommodating and chokes on those constructs, as you have subtly found out!

GeneS
08-27-2008, 09:26 PM
As Dale mentioned, eBook Publisher is not so accommodating and chokes on those constructs, as you have subtly found out!

My thanks to Ashkulz, Nick & Dale. My education continues.
Gene

ashkulz
08-30-2008, 08:01 PM
NOTE: impmake has now been renamed to ebookutils, and is hosted at http://gitorious.org/projects/ebookutils using Git, which should easily allow other developers to make changes. I plan to merge impserve, impmake, 2 new python utilities which replace ebw1150+rebcomm and pdfread under this umbrella.

Changes in ebookutils-0.3:
- impmake: add an -f option to specify the files to be included, with one file per line
- impserve: handle paging of booklist requests
- impserve: handle downloading of IMP books which are sent as an attachment (direct download of IMP from mobileread works)
- impserve: implement a plugin system and provide 2 default plugins to change the URL/content before being sent to the device.
- impserve: refactored the code heavily, please run impserve/run.py from python.

derrell
08-31-2008, 10:19 AM
After setting up wine to work with impmake, ie using winetricks to add vcrun6 and dcom98 does the gui preview of imp books quit working? I've tried every combination I can think of including which version of windows is being emulated and the best I've been able to get is that the eBook Publisher program will start and work but the preview application crashes. Thats the best case the worst is that eBook Publisher won't even start. Strangely using impmake still works and creates a valid imp format book.

For now I've just set up two different directories for wine. One with the wine version of dcom98 so eBook Publisher works and the other with the microsoft dcom98 so that impmake will work.

Seems like its an odd problem. I was just wondering if anyone else was having it as well?

ashkulz
08-31-2008, 12:41 PM
After setting up wine to work with impmake, ie using winetricks to add vcrun6 and dcom98 does the gui preview of imp books quit working? I've tried every combination I can think of including which version of windows is being emulated and the best I've been able to get is that the eBook Publisher program will start and work but the preview application crashes. Thats the best case the worst is that eBook Publisher won't even start. Strangely using impmake still works and creates a valid imp format book.

For now I've just set up two different directories for wine. One with the wine version of dcom98 so eBook Publisher works and the other with the microsoft dcom98 so that impmake will work.

Seems like its an odd problem. I was just wondering if anyone else was having it as well? I too have the same problem, but I don't use the viewer application much so I manage without it. The real problem is, the ebook publisher links directly to the PubUtil22.dll *without* COM, but to use it via impmake I have to use COM (no direct interface provided to PubUtil22.dll) hence dcom98 needed. Also, you can't start impmake + ebook publisher at the same time -- it crashes, for some weird reason.

But the main thing for me, it works and in an automated manner without having to start a Windows VM or do lots of configuration. If I do ever get the time, I plan to investigate it further but that is rather unlikely since it works with the workarounds so I don't complain.

The problem is, eBook Publisher is quite buggy at times and chokes on invalid HTML and crashes or fails with some GomiIzer (?!) error. I may have to write a script which just sanitizes the input HTML for impmake later...

derrell
08-31-2008, 03:15 PM
Thanks just wanted to make sure it wasn't just me. There are times when it seems that very odd things go wrong and no one else can replicate them.

It does work and I'll keep using it in the fashion I mentioned above. Thanks for all the effort that you are putting into the software for these readers.