On device site scraper

[knc1]

Originally Posted by Cinisajoy

Heck I am taking my most used recipes from the Internet and putting them on index cards.

K5 + Aries Kindle Menu gives you KindleNotes + Leafpad to edit them with.
I don't know if Collections Manager could organize them for you, but Librarian Sync can (if you organize them as a directory tree).

[Dark-Show]

Well thanks for all the input, as originally thought it will be a personal device, that would add news feeds and such in an easy to read format directly on your kindle, no cost, just an applet to crawl and format your favourite feeds. When and if that new exploit is released I will experiment with it. If it's not feasible, I may make a kindle browser formatted website to send the book to your kindle email. All books would be for personal use only. Downloaded on the fly with no storage on the server, if I go this route.

[Cinisajoy]

Originally Posted by Dark-Show

Well thanks for all the input, as originally thought it will be a personal device, that would add news feeds and such in an easy to read format directly on your kindle, no cost, just an applet to crawl and format your favourite feeds. When and if that new exploit is released I will experiment with it. If it's not feasible, I may make a kindle browser formatted website to send the book to you kindle email. All books would be for personal use only. Downloaded on the fly with no storage on the server, if I go this route.

This last part confuses me. Will it be you or the user creating the books?
First one could cause problems. Second one might be ok.

[Dark-Show]

Originally Posted by Cinisajoy

This last part confuses me. Will it be you or the user creating the books?
First one could cause problems. Second one might be ok.

The user would click on feeds that the parser can read, or input links to posts/threads that interest the user. And the script would format it on the fly, into a book then send it to the kindle through kindle email. The hardest part would be making a parser for each site.

[Dark-Show]

See this is an evolving idea, I was just looking for tips on MOBI creation on the kindle, and possibly user interest. Sorry I'm so confusing lots of ideas flowing through my head to bypass different problems.

[Cinisajoy]

Now I see what you are trying to do. I am not sure it is feasible. Let me see if I have your idea right.
You want to make a feed that can be read on the kindle.
The user gets to pick what is in the feed.
So if I was interested in human interest stories, this would get the content from various websites and put it in an easy to read format.
How would we find the content to begin with?
Note this one would change every day.

Or let's say I wanted to turn several online recipes into an easy to read format. Would your idea do this?

[Dark-Show]

Originally Posted by Cinisajoy

Or let's say I wanted to turn several online recipes into an easy to read format. Would your idea do this?

Well depending on if the parser is supported (I'd program multiple depending on what people wanted)

You would simply enter the links to each recipe, and click the format & send to kindle button. This would take each link and parse, format. then send it to your kindle as a book. If the website has a most popular page or similar you could also just click something (gotta think of names for this stuff) and it would make a book will all the most popular recipes on the site) I'll make sure for each link, the link to the page, and possibly to the author of the recipe is included.

[shamanNS]

And you seriously think that people would like browsing to some website and input looooong feed links using Kindle's experimental web browser, eInk display and laggy on-screen keyboard? Rather than using a PC?

Also how is any of this "on-device scrapping"?

[Cinisajoy]

What do you mean by parse and would I have to click the link in said book?

[Dark-Show]

Originally Posted by Cinisajoy

What do you mean by parse and would I have to click the link in said book?

No it is just there as a reference to where the data came from, the whole page would be stored in the book.

[Dark-Show]

Originally Posted by shamanNS

And you seriously think that people would like browsing to some website and input looooong feed links using Kindle's experimental web browser, eInk display and laggy on-screen keyboard? Rather than using a PC?

Also how is any of this "on-device scrapping"?

Well there are other way of going about it then links as well. But ill save those ideas for the site. And if you read this thread the website will only be created if its not feasible to do it on the device.

[eschwartz]

There is no need for people to come here poo-pooing the mere idea that someone would want to do this.

@Dark-Show,
I would seriously think about utilizing calibre's conversion, and possibly, recipes, infrastructure.
Again, there isn't a lot of open-source -->AZW3 conversion code around, and calibre is most of that, so why not reuse it? Otherwise, it should serve as documentation for re-implementing your own conversion code.
Also see our Wiki: MOBI or KindleUnpack
calibre's News recipes are pretty popular with some people -- I don't know how many would rather use your hack vs. calibre, but there is probably some call for on-device usage.

calibre's recipes are also an easy way to get well-written support for many websites out-of-the-box. You could fallback on some sort of generic screen-scraping for anything else.

It's not reasonable to ask people to install third-party ARM builds of all of calibre but some kind of stripped-down lightweight alternative would definitely be interesting.

calibre is all GPLv3.

@shamanNS,
Two words. "configuration file"
Thank you for your concern.