On device site scraper

Dark-Show · 12-08-2015, 03:58 PM

Hello, Kinda new here, been lurking for a while.

At the moment my current project has been a website scraper, that scrapes lets say CNN news stories and turns them into a EPUB, then using kindlegen into MOBI. My idea is to have a script for the kindle to select a website, then it would scrape and form a book. Is this possible, I have yet to hack my PW2 (was waiting for the software jailbreak) but my only question would be if we can convert ebooks on the device or if there is a way to make a mobi directly.

Also would people be interested in this project?

If no on device conversion is possible I could host a script that would convert the EPUB to MOBI, and output the new book. But i would much rather the conversion be done on the device.

Cinisajoy · 12-08-2015, 04:00 PM

Can we spell copyright infringement and possibly plagiarism?

Dark-Show · 12-08-2015, 04:04 PM

Never really thought about that. I mean it would be just a tool to format content that's hosted on sites for your kindle. I wanted to have the news on my kindle for a while, kind of like a newspaper.

EDIT:
And if all links and sources are given, wouldn't it just be a reference ebook?

cvkemp · 12-08-2015, 04:06 PM

Check the news content available on amazon. Or look at using calibre and see what feeds it can give you.

HarryT · 12-08-2015, 04:07 PM

Probably OK for personal use. But you obviously couldn't give the output to anyone.

Dark-Show · 12-08-2015, 04:07 PM

But that's what I mean, take feeds and turn them into book with just your kindle. Same as what calibre would do. Just without the need of a computer.

Dark-Show · 12-08-2015, 04:08 PM

Quote:

Originally Posted by HarryT

Probably OK for personal use. But you obviously couldn't give the output to anyone.

Well it would be personal, I mean if possible use maybe a simple DRM to lock it to the kindle it was generated on?

knc1 · 12-08-2015, 04:08 PM

Quote:

Originally Posted by Dark-Show

Never really thought about that. I mean it would be just a tool to format content that's hosted on sites for your kindle. I wanted to have the news on my kindle for a while, kind of like a newspaper.

EDIT:
And if all links and sources are given, wouldn't it just be a reference ebook?

It is much more like what most news services sell you.
You can get (often by purchase) news feeds, I think even from Amazon.

Dark-Show · 12-08-2015, 04:11 PM

Ahh so it would be an illegal project. Well I mean the initial idea was for news sites, but would It be more legal if it scraped say User made stories sites? Have references for the site and the author. Or would this still be copyright infringement and possibly plagiarism?

Cinisajoy · 12-08-2015, 04:15 PM

Quote:

Originally Posted by Dark-Show

Ahh so it would be an illegal project. Well I mean the initial idea was for news sites, but would It be more legal if it scraped say User made stories sites? Have references for the site and the author. Or would this still be copyright infringement and possibly plagiarism?

What do you mean by User site?
You may not be aware but anytime a person posts something on a website, it is automatically copyrighted.

Be very careful with your scraping.

Dark-Show · 12-08-2015, 04:20 PM

Well I suppose I'll keep this a private project then. Ahaha I just like to share the love.

eschwartz · 12-08-2015, 05:06 PM

Scaremongering aside, HarryT already made it clear that it is OK.
The tool is functionally similar to calibre's News Recipes feature.

Obviously you cannot redistribute them without violating e.g. CNN's copyright... but that has nothing to do with the tool.

Now to actually answer your question

The only two EPUB --> MOBI/AZW3 converters are Amazon's kindlegen and the conversion code in calibre.

kindlegen is closed-source and Amazon doesn't offer ARM builds (how many people write and publish books from their Kindle

).
calibre can be built under ARM but it has large dependencies.

I would say your best bet is to use calibre's source to write your own lightweight converter, possibly just regluing calibre's source code.
NiLuJe has published python builds for the Kindle (see the screensavers thread).

Cinisajoy · 12-08-2015, 05:09 PM

I will admit that I misread his original post. I read it as making ebooks and selling them.

As to his intent, that is ok if the contents are personal use only.

knc1 · 12-08-2015, 06:51 PM

Thanks!
I must have mis-read the post also. My bad.

For your own use, it is just a fancy way to organize your viewing.

Cinisajoy · 12-08-2015, 07:28 PM

Heck I am taking my most used recipes from the Internet and putting them on index cards.

12-08-2015, 03:58 PM	#1
Dark-Show Member Posts: 12 Karma: 10 Join Date: Dec 2015 Device: Kindle PW2	On device site scraper Hello, Kinda new here, been lurking for a while. At the moment my current project has been a website scraper, that scrapes lets say CNN news stories and turns them into a EPUB, then using kindlegen into MOBI. My idea is to have a script for the kindle to select a website, then it would scrape and form a book. Is this possible, I have yet to hack my PW2 (was waiting for the software jailbreak) but my only question would be if we can convert ebooks on the device or if there is a way to make a mobi directly. Also would people be interested in this project? If no on device conversion is possible I could host a script that would convert the EPUB to MOBI, and output the new book. But i would much rather the conversion be done on the device.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Leafmarks: New Book Site/series-tracking social book site	BearMountainBooks	General Discussions	16	05-24-2015 06:30 PM
Metadata scraper plugin api	kiwidude	Development	5	03-06-2011 11:58 AM
news scraper alternatives to feedbooks	bZkindle	Deals and Resources (No Self-Promotion or Affiliate Links)	3	01-22-2011 08:59 AM
PRS-500 device bricked even after trying everything from site please help	hammerfall82	Sony Reader	1	09-21-2010 08:26 AM
The Times Labs: Book Scraper	TadW	News	0	01-27-2009 03:12 PM

12-08-2015, 04:00 PM	#2
Cinisajoy Just a Yellow Smiley. Posts: 19,161 Karma: 83862859 Join Date: Jul 2015 Location: Texas Device: K4, K5, fire, kobo, galaxy	Can we spell copyright infringement and possibly plagiarism?

12-08-2015, 04:04 PM	#3
Dark-Show Member Posts: 12 Karma: 10 Join Date: Dec 2015 Device: Kindle PW2	Never really thought about that. I mean it would be just a tool to format content that's hosted on sites for your kindle. I wanted to have the news on my kindle for a while, kind of like a newspaper. EDIT: And if all links and sources are given, wouldn't it just be a reference ebook?

12-08-2015, 04:06 PM	#4
cvkemp Loving life Posts: 1,412 Karma: 7991496 Join Date: Mar 2009 Location: Hot Springs Village, Arkansas Device: PaperWhite 5,iPhone 13, IPad, MacBook Air	Check the news content available on amazon. Or look at using calibre and see what feeds it can give you.

12-08-2015, 04:07 PM	#5
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	Probably OK for personal use. But you obviously couldn't give the output to anyone.

12-08-2015, 04:11 PM	#9
Dark-Show Member Posts: 12 Karma: 10 Join Date: Dec 2015 Device: Kindle PW2	Ahh so it would be an illegal project. Well I mean the initial idea was for news sites, but would It be more legal if it scraped say User made stories sites? Have references for the site and the author. Or would this still be copyright infringement and possibly plagiarism?

12-08-2015, 04:20 PM	#11
Dark-Show Member Posts: 12 Karma: 10 Join Date: Dec 2015 Device: Kindle PW2	Well I suppose I'll keep this a private project then. Ahaha I just like to share the love.

12-08-2015, 05:06 PM	#12
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Scaremongering aside, HarryT already made it clear that it is OK. The tool is functionally similar to calibre's News Recipes feature. Obviously you cannot redistribute them without violating e.g. CNN's copyright... but that has nothing to do with the tool. Now to actually answer your question The only two EPUB --> MOBI/AZW3 converters are Amazon's kindlegen and the conversion code in calibre. kindlegen is closed-source and Amazon doesn't offer ARM builds (how many people write and publish books from their Kindle ). calibre can be built under ARM but it has large dependencies. I would say your best bet is to use calibre's source to write your own lightweight converter, possibly just regluing calibre's source code. NiLuJe has published python builds for the Kindle (see the screensavers thread).

12-08-2015, 05:09 PM	#13
Cinisajoy Just a Yellow Smiley. Posts: 19,161 Karma: 83862859 Join Date: Jul 2015 Location: Texas Device: K4, K5, fire, kobo, galaxy	I will admit that I misread his original post. I read it as making ebooks and selling them. As to his intent, that is ok if the contents are personal use only.

12-08-2015, 06:51 PM	#14
knc1 Going Viral Posts: 17,212 Karma: 18210809 Join Date: Feb 2012 Location: Central Texas Device: No K1, PW2, KV, KOA	Thanks! I must have mis-read the post also. My bad. For your own use, it is just a fancy way to organize your viewing.

12-08-2015, 07:28 PM	#15
Cinisajoy Just a Yellow Smiley. Posts: 19,161 Karma: 83862859 Join Date: Jul 2015 Location: Texas Device: K4, K5, fire, kobo, galaxy	Heck I am taking my most used recipes from the Internet and putting them on index cards.

Advert

Advert