Automatic convert webpage to epub?

[bthoven]

I have one favorite webblog which I would like to automatically download and convert it to epub.

http://uncle-rods.blogspot.com/

This is how I do it manually:

1. goto website web2fb2.com (give excellent result!)
2. paste the above website address, fillin title and author
3. press a "do it" button to start converting it to epub
4. download the output epub file to my local drive
5. either copy the epub file to my Nook or add it to Calibre before sending it to my Nook.

Is there a way to automate step 1-4 by using any builtin Calibre commands? If so, I'll create a batch file containing those command with proper parameters and schedule it to run everyday by using Windows Task Scheduler function.

Thanks

[kovidgoyal]

web2disk is what you're looking for.

[bthoven]

Thanks Kovid..could you elaborate more? It seems this web2disk is just downloading the web content into my local disk. How about a command to convert html to epub?

[kovidgoyal]

ebook-convert

[bthoven]

Thanks Kovid.

I tried web2disk with the above website and it creates the following files:

index.xhtml (79kb but it can't be opened with webbrowser; it seems to have unreadable content)
\links\ (no content in this folder)
\images\ (no content in this folder)
\stylesheet\ (no content in this folder)

Any reason why?

[Starson17]

Originally Posted by bthoven

Thanks Kovid.

I tried web2disk with the above website and it creates the following files:

index.xhtml (79kb but it can't be opened with webbrowser; it seems to have unreadable content)
\links\ (no content in this folder)
\images\ (no content in this folder)
\stylesheet\ (no content in this folder)

Any reason why?

Lots of possible reasons - could be cookies, scripts, referer (sic), useragent, etc. They probably don't want robots to grab content from this site.

[chaley]

I just tried fetching it using wget on windows 7. I got the correct index.html and was able to convert it.

Get wget at http://gnuwin32.sourceforge.net/packages/wget.htm

After installing it, and assuming you took the defaults, in a command box, run

"C:\Program Files\GnuWin32\bin\wget" -P c:\unclerod http://uncle-rods.blogspot.com/
cd c:\unclerod
ebook-convert index.html unclerod.epub

Change c:\unclerod to the directory you want the index.html to appear.

[bthoven]

Hi..thanks everyone

It was my mistake, Google Chrome can't display index.xhtml; but it displays fine in IE.

When I create a batch file having both web2disk and ebook-convert, the ebook-convert won't work! But if the batch file has ebook-convert command alone, it works fine. I'm not sure whether because ebook-convert start running when index.xhtml has still not been created. How can I set the delay time between web2disk and ebook-convert command?

Another question, how can I put the Title and Author into the output epub file automatically?

[Terisa de morgan]

Originally Posted by bthoven

Another question, how can I put the Title and Author into the output epub file automatically?

You can do it directly when you convert it with ebook-convert, with options --authors and --title. It's all described at calibre user-manual (very good manual, BTW )

[bthoven]

Originally Posted by Terisa de morgan

You can do it directly when you convert it with ebook-convert, with options --authors and --title. It's all described at calibre user-manual (very good manual, BTW )

Thanks a lot Terisa.

I can now add author and cover into the epub when doing the ebook-convert.

Here is my post how I did it:

https://www.mobileread.com/forums/sho...043#post844043

Great software!

[Starson17]

Originally Posted by chaley

I just tried fetching it using wget

BTW, nothing to do with Calibre, but wget is a fantastic tool for automatically grabbing content from the web. I've been using it for years for my wife's computerized embroidery hobby to grab files from sites that provide them free, but only if you come back once every 3 days, or only if you hunt through 20 pages to find the page with the free file.

[bthoven]

Hi,

I create a batch file containing the following command lines:


web2disk http://uncle-rods.blogspot.com -r 0
ping -n 60 127.0.0.1 >NUL
ebook-convert index.xhtml unclerod.epub --authors "Rod Mollise" --author-sort "Mollise, Rod" --cover

"http://skywatch.brainiac.com/astroland/rodnow.jpg"

The problem is after web2disk command was completed, the batch file just quit itself and never execute the next command lines.

I also tried this in another batch file:

ebook-convert index.xhtml unclerod.epub
ping -n 60 127.0.0.1 >NUL (to delay 60 seconds)

Same problem, ping command was never executed.

Any idea what is the cause of problem?

[itimpi]

I seem to remember it coming up elsewhere that the calibre tools do not play well with batch files. The symptom being what you state - that subsequent commands do not get executed because the command processor dies.

I seem to remeber that the best answer that anyone came up with was to that when you wanted to execute a new command you did it in a new instance of the command processor (using the CMD /C option).

Another solution was to use alternative command processors.

[chaley]

Hmmm....

On windows 7, I created a batch file containing

web2disk http://uncle-rods.blogspot.com  -r 0
ebook-convert index.xhtml unclerod.epub --authors "Rod Mollise" --author-sort "Mollise, Rod" --cover "http://skywatch.brainiac.com/astroland/rodnow.jpg"

Running that batch file from a command box, I get

C:\Users\charles\Desktop\foo>run.bat

C:\Users\charles\Desktop\foo>web2disk http://uncle-rods.blogspot.com  -r 0
.

C:\Users\charles\Desktop\foo>ebook-convert index.xhtml unclerod.epub --authors "Rod Mollise" --author-sort "Mollise, Rod" --cover
"http://skywatch.brainiac.com/astroland/rodnow.jpg"
Downloading cover from u'http://skywatch.brainiac.com/astroland/rodnow.jpg'
1% Converting input to HTML...
InputFormatPlugin: HTML Input running
on C:\Users\charles\Desktop\foo\index.xhtml
Language not specified
Creator not specified
Building file list...
IgnoreFile(u'Could not read from file: c:\\"http:\\uncle-rods.blogspot.com\\" with error: (22, "invalid mode (\'rb\') or filename"
)',)
Initial parse failed:
Parsing file 'index.xhtml' as HTML
Forcing index.xhtml into XHTML namespace
Stripping comments and meta tags from index.xhtml
Merging multiple <head> and <body> sections
[lots of random warnings deleted]
Creating EPUB Output...
67% Creating EPUB Output
        Looking for large trees in index.xhtml...
        No large trees found
        Split into 6 parts
EPUB output written to C:\Users\charles\Desktop\foo\unclerod.epub
Output saved to   C:\Users\charles\Desktop\foo\unclerod.epub
C:\Users\charles\Desktop\foo>

As it works for me, you must be having some environment-specific problem. Some questions:

1. What version of windows are you running?
2. If run the commands one at a time in a foreground command box, does it work?
3. If the answer to #2 is yes, does running your batch file in the same foreground command box (not with the task scheduler) work? Be sure that your working directory is the same as in #2.

My suspicion is that the problem comes from the current directory being something other than what you expect, especially if you are using a shortcut to the batch file or if the batch file lives in a normally-readonly folder. One thing you can try is to add as the first line of the batch file a cd to the folder where the output files are expected to end up.

[slm]

on another thread, the problem of running several lines of batch code for Calibre was discussed. There someone who seemed to know said that the batch file doesn't wait for the first line call to finish but that you can do multiple lines by making each line begin with a "call" statement.

I have no idea what I'm talking about.