local hosting on e-readers

[STEM]

This is my first attempt - so please laugh politely

I have a very large free web site of text material - 10s of 1000s of books, booklets and articles many 10s of 1,000,000s of words all in html and xhtml. This collection could be zipped up to several 100MBs - so is OK for e-reader storage capacity.

I would like this to be available on e-readers (and phones, tablets etc.).
At the moment it is available as a web site to any device with web access.
I am aware of a large purist e-book group who do not want web access, however what I have offered since 1995 is books on digital devices - I don't care what the devices are - but I think e-ink is very exciting!
I believe that there is a large number of people with poor internet access who would like to use this collection in daylight with long battery life.
Could some one point me to places where I could get advice about local hosting on any e-reader device? . . . .
because I see bulk re-formatting as a terrible task when there is an archive of the size of mine.
With my site there is a built-in navigation structure (and searching function - Google)
What (reasonably pain-free) ways are there to build such structure if local-hosting is not possible?
I have been waiting for this feature to arrive - should I just keep waiting?
Thank you in anticipation - one for you readers - chrestomathy

[HarryT]

If you're asking how to bulk-convert HTML into a format suitable for eBook readers, the best place to ask advice would be the "Workshop" forum.

[STEM]

If I can find my way to it I will transfer the request, but local hosting possibilities on e-readers was my interest - in order to read free-flowing text and avoid huge re-formatting work.
I rejected pdf more than 15 years ago because it was page-based (IMO many of the problems with Adobe products over the years is that they never left their DTP roots behind), and I considered that for digital use a screen-based format should be free scrolling, free-flowing to window screen width(s).
Programs need to have a clean page turn/scroll to give a satisfactory experience - many do not! I wonder if any one has thought to put a temporary (changing each time the page-down command was given) marker line on a scrolling page to show where the previous page/window end-point was?
Thanks again.

[STEM]

PS I have read 'The Courts of the Morning', by John Buchan, a teenage adventure story which works surprisingly well considering JB had to invent a country to do it.
Last week I sold a set of JB on Ebay

[Valmore Daniels]

welcome

[DMcCunney]

Originally Posted by STEM

If I can find my way to it I will transfer the request, but local hosting possibilities on e-readers was my interest - in order to read free-flowing text and avoid huge re-formatting work.

I looked at your site. You're a specialist in Christian related literature, with Bibles, bible commentary, and writing on Christian topics. For those interested, it looks like a valuable resource.

But I'm not clear on what you mean by "local hosting". It sounds like you want to create a copy of your site, possibly as a zip archive, that interested readers could download and browse locally without requiring Internet access.

That's fine, if they have a PC/laptop/notebook/netbook with sufficient storage, and a capable browser to view the material. (I could create a local copy now using HTTrack). It's much less fine if they have a dedicated ebook reader, or ebook viewer application on a PDA, smartphone, or tablet. Storage and browser may be issues.

A larger issue is reader expectations. Folks using readers tend to want individual books, in the native format (MobiPocket/ePub/etc.) supported by their device, created to display optimally on their devices, with things like working hyper-linked tables of contents.

If you really want to provide your content to ebook viewer users, I fear you are looking at conversion if you want people to actually use the resources you want to offer. Places like Project Gutenberg, ManyBooks.net, Munseys.com and others offer a variety of ebook formats, handled by scripted conversion from HTML source. While it's a large effort, it can be automated, and is probably what you will need to do.

I rejected pdf more than 15 years ago because it was page-based (IMO many of the problems with Adobe products over the years is that they never left their DTP roots behind), and I considered that for digital use a screen-based format should be free scrolling, free-flowing to window screen width(s).

It's possible with current versions of Acrobat to create PDFs with tagging that let the viewer reflow the text for the device screen. Most PDFs are not so created, and some viewer devices (like the Kindle DX) that can display PDFs don't support the feature.

Programs need to have a clean page turn/scroll to give a satisfactory experience - many do not! I wonder if any one has thought to put a temporary (changing each time the page-down command was given) marker line on a scrolling page to show where the previous page/window end-point was?

I haven't seen that. But being able to specify a scroll overlap (IE: show the last line or two from the previous page/screen on the next screen when a page down operation is requested) is common enough.
______
Dennis

[STEM]

Thanks!
I very much appreciate your detailed response.
What I don't appreciate is having to do all this work over again.

Originally Posted by DMcCunney

But I'm not clear on what you mean by "local hosting". It sounds like you want to create a copy of your site, possibly as a zip archive, that interested readers could download and browse locally without requiring Internet access.

Yes something like that. All e-readers work on an OS and some have browsers: e.g. the Kindle. This Linux OS & browser seems to be more than a bit crippled. It is possible to have an image of a web site on a local machine - PC etc., so that the browser seamlessly recognises the web site though it is off-line. With Windows you can do this without a second party program, and set a site up which the OS and browser recognise as being located locally at http://localhost/ - IP address of 127.0.0.1

I don't care how it is achieved, the easier for users the better for me!
I don't suppose that there are many folk who would enjoy rebuilding 1,000's of files into a mobi or other format. Only to find that another format pops up . . . . and another . . .
If the device has a browser (Kindle), surely it should be possible to 'force' the OS & browser to recognise a locally hosted website. Then users can access the files/pages/books in exactly the same way they would on a web site - no learning curve! Free-flowing text, choice of fonts/sizes and all the other benefits.
I may be naive but this seems so obvious to me. I can see that Amazon want to sell books and their primary commercial reason for a browser on the Kindle is to enable people so to do, any other browsing they give is a bonus selling point. They have no real interest in enabling folk to get books elsewhere. I feel that the e-readers which will have long-term success will have a decent browser, and not be tied into an Amazon/Apple/Microsoft/Adobe proprietary solution. But mebbe I am VERY naive

(I like the Sony e-reader touch screen, I do not like the clunky Kindle navigation, I would be rather surprised if the Kindle does not feature touch soon, I think colour should not take over but could be an optional model)

[GeoffC]

Welcome to Mobileread ....

[DMcCunney]

Originally Posted by STEM

Thanks!
I very much appreciate your detailed response.

You're welcome.

What I don't appreciate is having to do all this work over again.

I don't blame you, but I fear you're stuck with it if you want ereader users to actually use the material you offer.

Yes something like that. All e-readers work on an OS and some have browsers: e.g. the Kindle. This Linux OS & browser seems to be more than a bit crippled. It is possible to have an image of a web site on a local machine - PC etc., so that the browser seamlessly recognises the web site though it is off-line. With Windows you can do this without a second party program, and set a site up which the OS and browser recognise as being located locally at http://localhost/ - IP address of 127.0.0.1

On Windows, I don't need to go through that. All I need is a bookmark to the local resource using file:// instead of http://. And I have large quantities of HTML content stored locally which I can simply click to open, as my default browser is specified as the handler for that type of file in Windows file associations. You can pass any browser a file spec or a url on the command line and expect the browser to try to open it.

But don't limit the consideration to just dedicated ereader devices. Any awful lot of ebooks are consumed using viewer applications on tablets, smartphones, and the like.

And the reader is a dedicated appliance, designed to do a particular thing, so it will be crippled compared to a general purpose device using the same OS. Things not related to what the device is intended to do will be absent, even if the OS can technically support them.

I don't care how it is achieved, the easier for users the better for me!

*cough* conversion *cough*

I don't suppose that there are many folk who would enjoy rebuilding 1,000's of files into a mobi or other format. Only to find that another format pops up . . . . and another . . .

Which is why sites like Project Gutenberg provide more than one format. We're stuck with that, too.

If the device has a browser (Kindle), surely it should be possible to 'force' the OS & browser to recognise a locally hosted website. Then users can access the files/pages/books in exactly the same way they would on a web site - no learning curve! Free-flowing text, choice of fonts/sizes and all the other benefits.

The device may not. Or it might have a different idea of a file system than you are used to. Or it may have a browser that can't open local content. (I have a device with a browser like that.)

I may be naive but this seems so obvious to me. I can see that Amazon want to sell books and their primary commercial reason for a browser on the Kindle is to enable people so to do, any other browsing they give is a bonus selling point. They have no real interest in enabling folk to get books elsewhere. I feel that the e-readers which will have long-term success will have a decent browser, and not be tied into an Amazon/Apple/Microsoft/Adobe proprietary solution. But mebbe I am VERY naive

It's quite possible to side-load content not encumbered by DRM and read it on a Kindle, as long as it's in Mobipocket format (which is an encapsulated subset of HTML.)

For that matter, it's possible to side-load content in ePub or PDF on devices that support those formats and read them using the device.

And the Kindle browser is capable enough that there are a few folks on MR who participate from the Kindle.

I've no idea if the Kindle browser can be told to open and display local HTML content (I don't have or want one, and can't test.), but it isn't what I would want to do if it could. I'd want to use it to read ebooks in the format it was designed to display. The browser would simply be a tool to help me locate and acquire them.

(I like the Sony e-reader touch screen, I do not like the clunky Kindle navigation, I would be rather surprised if the Kindle does not feature touch soon, I think colour should not take over but could be an optional model)

Color is an evolving option. eInk doesn't really support it. (There is a flavor of eInk that supports 12 bit color, but only one manufacturer is sampling devices using it, and the color offered will be insufficient for many applications.)

Full color is easy enough to achieve if you don't use eInk, but you lose eInk's advantage in battery life. There are several technologies that claim to address that, but none are in widespread usage yet.
______
Dennis

[STEM]

Originally Posted by DMcCunney

I fear you're stuck with it if you want ereader users to actually use the material you offer.

Where is the weeping icon

'Other devices'
Yes, I have been concentrating on making lots of good material available in xhtml which seems to be the most universal format so that many systems/devices/programs can use them.
Potentially easier to convert in bulk with their CSS.

I do appreciate the desire to keep reading pure - I have thousands of real books and I work from home so I can read them easily

I am surely a bit thick that I cannot see why html presented, black on white, text in e-ink is not a good reading experience . . . . ?

Just to continue with my (false) humility - please explain:
*cough* conversion *cough*

It looks like I start a lot of work or start a lot of waiting - I am much better at the waiting than the working - decision made! . . . mebbe.
Thanks again!

Off-topic: Isn't it great that paper is still the best! Years ago I came across a news item that the British Library had digitised the Domesday Book or something equally priceless - and within a few years the computers could not read it
(I started using AmiPro3 - remember that? - better than Word2. Then Word, then I got fed up and went non-proprietary to html. Gnashing of teeth that slightly older Word version cannot read docx natively - but OOo doesn't even make a fuss about it. I only use Win for OmniPage)

[DMcCunney]

Originally Posted by STEM

Where is the weeping icon

We don't have one. Pity - it would be useful.

'Other devices'
Yes, I have been concentrating on making lots of good material available in xhtml which seems to be the most universal format so that many systems/devices/programs can use them.
Potentially easier to convert in bulk with their CSS.

It's my preferred storage format, because it is easy to convert. In my case, my normal reader is a Palm OS PDA, and I get content in HTML and convert for reading on the device with Plucker, an open source offline HTML viewer for Palm devices. (It has a web browser, but the default browser supplied with it can't open local content, doesn't support JavaScript, and falls over on most CSS.)

I do appreciate the desire to keep reading pure - I have thousands of real books and I work from home so I can read them easily

So do I. The advantage to ebooks here is the ability to carry a library in my pocket.

I am surely a bit thick that I cannot see why html presented, black on white, text in e-ink is not a good reading experience . . . . ?

In fact, on readers, that normally is what you are seeing. MobiPocket uses an encapsulated subset of HTML. ePub is a container, and XHTML text is one of the things it may contain.

The issue isn't the HTML - it's the file it's in. If I'm using a dedicated reader device that supports a particular ebook format, I'll probably want all of the ebooks I read in that format, so I can use one interface to select and read them.

For instance, I mentioned I use a Palm OS PDA as my primary reader device.

The advantage, aside from the fact that it does a lot of other things besides display ebooks, is that there isn't much I can't read. Aside from Plucker documents, I can read MobiPocket files, eReader files, Adobe PDFs, Word and RTF files, and plain text files. The only format I don't have native support for is ePub, but I can convert that to Mobi if it isn't encumbered by DRM.

The disadvantage is that I have to remember which book is in which format, read by what viewer. There's no way to have the entire library selectable from one interface,.

Just to continue with my (false) humility - please explain:
*cough* conversion *cough*

Sorry - just a way of indicating that you would be stuck doing conversion.

It looks like I start a lot of work or start a lot of waiting - I am much better at the waiting than the working - decision made! . . . mebbe.
Thanks again!

The biggest work would be learning what's involved in doing the conversions. As mentioned, it can be scripted, once you have the tools and have written the scripts, and the computer can do the work. It's getting the tools you need and writing the scripts that will be time consuming.

Off-topic: Isn't it great that paper is still the best! Years ago I came across a news item that the British Library had digitised the Domesday Book or something equally priceless - and within a few years the computers could not read it

More efforts like that are in place. With luck, XML will become the standard storage format. Once in XML, it's much easier to get it into the output format you need.

(I started using AmiPro3 - remember that? - better than Word2. Then Word, then I got fed up and went non-proprietary to html. Gnashing of teeth that slightly older Word version cannot read docx natively - but OOo doesn't even make a fuss about it. I only use Win for OmniPage)

I recall AmiPro. I began in the MS-DOS days when WordStar was the WP of choice. I kept some fluency in it because many text editors either used the WordStar command set or could be told to, and I didn't need to retrain my fingers. (I had Gnu Emacs customized to use WordStar commands at one point. )

I have Office 2007, but have OO3 as well in part because it can deal with DOCX files. Nice to be able to do that under Linux. (And in typical MS fashion, DOCX is an XML based format with proprietary MS stuff tossed in. "Yes, we support standards! Do it our way!" Er, no.)
______
Dennis

[STEM]

A little difficulty for me is 'chicken and egg': do I wait for folk to ask for material in e-book format, or do I do it first . . . . mebbe they don't want it . . .
Most of my material is KISS (keep it simple, stupid), it is only recently that I have even divided books into chapters!

>e(book) is a container
Yes, and for most files it should be easy to script - then there are the exceptions, and checking them all . . . .
A pity that e-book on demand, like pdf on demand from a site is not really helpful.


Off-topic: Emacs - great prog.!
I was spoiled in my early days of making this material available on CDs, by finding a search program called ISYS (only under Win). Its building, indexing, search and display, stats, search-term highlighting, jump to, notes with refs . . . it was so easy and effective that I did not need to do anything sophisticated. Alas it became too successful and thereby expensive, so I have to put up with Google. But they kindly allow its non-commercial use on my site for which I am very thankful.
Millions of hits every month means that I am doing something right.

[STEM]

I have this dream . . . .mebbe some enthusiast will volunteer to do it . . . .

[Latinandgreek]

Welcome, and best of luck with your project.

[DMcCunney]

Originally Posted by STEM

A little difficulty for me is 'chicken and egg': do I wait for folk to ask for material in e-book format, or do I do it first . . . . mebbe they don't want it . . .

Put a note on your front page and ask.

Most of my material is KISS (keep it simple, stupid), it is only recently that I have even divided books into chapters!=

Works well enough on line, where you can expect the browser to remember where you were. Doesn't work locally, where you might want to jump to a specific section, and a ToC is invaluable.

>e(book) is a container
Yes, and for most files it should be easy to script - then there are the exceptions, and checking them all . . . .

If what you have is all valid XHTML, I see few exceptions.

A pity that e-book on demand, like pdf on demand from a site is not really helpful.

It would rather miss the point of what you want to do.

Off-topic: Emacs - great prog.!

Well, great design philosophy. There are lots of things that call themselves emacs. See http://texteditors.org/cgi-bin/wiki.pl?EmacsFamily for a probably incomplete list.

I was spoiled in my early days of making this material available on CDs, by finding a search program called ISYS (only under Win). Its building, indexing, search and display, stats, search-term highlighting, jump to, notes with refs . . . it was so easy and effective that I did not need to do anything sophisticated. Alas it became too successful and thereby expensive, so I have to put up with Google. But they kindly allow its non-commercial use on my site for which I am very thankful.

I'm aware of ISYS. If you have the need, and the $ to fund it, it's very much worth the while.

Millions of hits every month means that I am doing something right.

So it would seem.
______
Dennis