Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Formats > Workshop

Notices

Reply
 
Thread Tools Search this Thread
Old 04-05-2009, 06:36 AM   #76
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Whoa.... this thread seems to have taken on a life of its own! Apologies for not having consolidated everyone's changes yet into the main build - (paid) work has been getting in the way a bit of late. Rest assured I haven't forgotten about it, although with the number of changes that seem to have taken place since I last looked it seems there may be a bit of work involved! Thank you to everyone who's added their own bits and pieces - it sounds like it's a bit of a frankenstein at the moment, but a frankenstein that seems to be filling people's needs :-).

If anyone would prefer that I *not* integrate their changes into my main build, please sing out - the next release of this is intended to be GPLv2, but I'm happy to change that if there are objectors.
erayd is offline   Reply With Quote
Old 04-05-2009, 10:24 AM   #77
araevin
Member
araevin began at the beginning.
 
Posts: 13
Karma: 10
Join Date: Aug 2005
Location: Miami, Fl
Device: iPhone
Ok does anybody know how to get this working on OSX 10.5.6 I run mamp pro and I've cant tell this script from the top to bottom anybody tell how to get started?
araevin is offline   Reply With Quote
Old 04-05-2009, 12:54 PM   #78
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
erayd! Yay! You're back!

By the way, flag.erayd.net seems to be down the past few weeks.

FanFiction.Net changed their pages quite a bit (added graphics, changed handling of crossover fics, etc). Below are the changes I made to get FF.Net working again, albeit there's probably a few changes in there for additional information (I was working on an already modified version ).

Code:
function ffnet_get_meta($page) {
	$matches = array();

	//category and title
	if (preg_match('/<td>.*<a href.+>.+Crossover<\/a> »/Ui', $page, $matches)) {
		preg_match('/<td>.*<a href.+>(.+) Crossover<\/a> » <b>(.+)<\/b><\/td>/Ui', $page, $matches);
		$matches[1] = str_ireplace(',', '', $matches[1]);
		$tag = str_ireplace(' and ', ',', $matches[1]);
	}
	else preg_match('/<td>.+ » <a href=.+>(.+)<\/a> » <b>(.+)<\/b><\/td>/Ui', $page, $matches);

	//author
	preg_match("/<td>Author: <a href='\/u\/[0-9]+\/.+'>(.+)<\/a><\/td>/Ui", $page, $author);
	$meta['title'] = $matches[2];
	$meta['author'] = $author[1];
	$meta['category'] = $matches[1];
	$meta['crosscat'] = $tag;

	//rating, lang, category
	preg_match("/<TR><TD><a href='http:\/\/www.fictionratings.com\/guide.php' target='rating'>Fiction Rated: (.+)<\/a> - (.+) - +(.+) - Reviews: <a href='.+'>[0-9]+<\/a> - Published: ([0-9-]+) - Updated: ([0-9-]+) (- (Complete)+ )?- id:([0-9]+) <\/td><\/TD><\/TR>/Ui", $page, $matches);
	$meta['rating'] = $matches[1];
	$meta['language'] = $matches[2];
	$meta['genre'] = $matches[3];
	$meta['date_pub'] = $matches[4];
	$meta['date_update'] = $matches[5];
	$meta['complete'] = $matches[7] == 'Complete' ? true : false;
	$meta['ficstatus'] = $matches[7] == 'Complete' ? 'Complete' : 'In-Progress';
	$meta['id'] = $matches[8];

	//description
	preg_match('/<meta name="description" content="(.+), (.+),  (.+)">/Ui', $page, $matches);
	$meta['summary'] = $matches[3];
	$meta['summary'] = iconv('UTF-8', 'ISO-8859-15//TRANSLIT//IGNORE', $meta['summary']);

	//chapters
	if(preg_match("/<SELECT title='chapter navigation' Name=chapter onChange=\".+\">(<option.+)<\/select>/Ui", $page, $matches)) {
		preg_match_all("/<option  value=[0-9]+ (selected)?>([0-9]+). (.+)(?=<option|$)/Ui", $matches[1], $matches);
		foreach($matches[2] as $key => $chapnum) $chaparr = $chapnum;
		$meta['chapters'] = $chaparr;
	}

	return $meta;
}
Here are some changes I made to epub.codec.php so it will display additional meta information in Calibre. This works with the Windows CLI utilities. I think it should be the same for Linux. As far as I'm aware, the only changes rgibbo made in the Windows port is changing the handling of temp folder and removal of posix functions.
Code:
	//Fix summary
	$story['meta']['summary'] = str_replace('"', '\"', $story['meta']['summary']);

	//XPath for chapter detection
	$cli_chapter = "";
	$cli_chapter .= " --chapter-mark \"pagebreak\"";

	//metadata
	$cli_metadata = "";
	$cli_metadata .= " -t \"{$story['meta']['title']}\"";
	$cli_metadata .= " -a \"{$story['meta']['author']}\"";
	if(!($story['meta']['crosscat'] == '')) $cli_metadata .= " --subjects \"{$story['meta']['crosscat']}\"";
	else if(!($story['meta']['category'] == '')) $cli_metadata .= " --subjects \"{$story['meta']['category']}\"";
	if(!($story['source'] == '')) $cli_metadata .= " --publisher \"{$story['source']}\"";
	$cli_metadata .= " --comment \"<b>Story id:</b> {$config['storyid']}";
	if(!($story['meta']['category'] == '')) $cli_metadata .= "<br><b>Category:</b> {$story['meta']['category']}";
	if(!($story['meta']['genre'] == '')) $cli_metadata .= "<br><b>Genre:</b> {$story['meta']['genre']}";
	if(!($story['meta']['rating'] == '')) $cli_metadata .= "<br><b>Rating:</b> {$story['meta']['rating']}";
	if(!($story['meta']['ficstatus'] == '')) $cli_metadata .= "<br><b>Status:</b> {$story['meta']['ficstatus']}";
	if(!($story['meta']['summary'] == '')) $cli_metadata .= "<br><b>Summary:</b> {$story['meta']['summary']}";
	$cli_metadata .= '"';

	//Create the epub content and store it in a temp file
	exec("html2epub $cli_chapter $cli_metadata \"$temp_fname.html\"");
	$result = file_get_contents("$temp_fname.epub");
Anyway, it's good to have you back!
ilovejedd is offline   Reply With Quote
Old 04-05-2009, 02:47 PM   #79
EowynCarter
Wizard
EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.EowynCarter ought to be getting tired of karma fortunes by now.
 
Posts: 4,084
Karma: 1604164
Join Date: Oct 2008
Location: Paris
Device: Cybooks; Sony PRS-T1
Whoo that rocks !

I was thinking of making someting like this.
Calibre does fine in reading mobi files. as well as converting them from epub
EowynCarter is offline   Reply With Quote
Old 04-05-2009, 03:42 PM   #80
Mnementh
Groupie
Mnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the end
 
Mnementh's Avatar
 
Posts: 154
Karma: 31496
Join Date: Mar 2009
Location: UK
Device: Kindle 4 NT & Kobo Aura
ilovejedd,

Any chance of posting the whole code listing or maybe the updated files? When i try making the changes you've posted to the two versions of ffnet.source.php and epub.codec.php I have I get errors when trying to use them.

TIA

Mnementh
Mnementh is offline   Reply With Quote
Old 04-05-2009, 05:41 PM   #81
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
Unfortunately, the ffnet.source.php that code snippet was taken from is in shambles and contains numerous errors. I just posted the fix relevant to the current "trunk" (basically rgibbo's Windows port of erayd's latest script). Even then, I had to fix up the snippet because it had bits and pieces of extraneous code.

By the way, mobileread seems to have changed & # 1 8 7 ; (remove spaces) to ». Try changing it back and see if that fixes things.

Okay, I re-downloaded and reverted back to rgibbo's version and implemented the changes I made. Here's to hoping it works.
http://www.utterlyinsane.org/fflag/ffnet.source.zip

The epub.codec.php snippet probably works, but if you can't study the source code and see where the problem is, I'd just leave the old version be. It's quite possible that the snippet won't work since it was tested with different versions of the scripts compared to what's been uploaded. Besides, the only relevant fix made between this and the version I uploaded a few days ago (in the custom recipe thread) is how it handles crossover fanfics.

Previous behavior for crossover fanfics is to create a new category/subject/tag Category 1 and Category 2 as its own separate category. With the fix, it will tag the fanfics as being part of each individual category, e.g. it will be tagged as Category 1 and Category 2 separately.

Last edited by ilovejedd; 04-05-2009 at 06:12 PM.
ilovejedd is offline   Reply With Quote
Old 04-05-2009, 07:44 PM   #82
Mnementh
Groupie
Mnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the end
 
Mnementh's Avatar
 
Posts: 154
Karma: 31496
Join Date: Mar 2009
Location: UK
Device: Kindle 4 NT & Kobo Aura
Looks like the problem was just with the ffnet.source.php, with the new version everything is working nicely again, thank you.

I really appreciate the time and effort everybody has put into this, just makes me wish I had something of my own to contribute.

/me goes to read up on php as there are quite a few sites that I would like to see a script for to do the same job as this one.
Mnementh is offline   Reply With Quote
Old 04-05-2009, 08:37 PM   #83
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
I actually have working sources for TheSugarQuill and AdultFanfiction.net in case you need any for those sites. They should be attached somewhere in this thread.
ilovejedd is offline   Reply With Quote
Old 04-05-2009, 10:19 PM   #84
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by ilovejedd View Post
erayd! Yay! You're back!

By the way, flag.erayd.net seems to be down the past few weeks.
Oops - moved servers a little while ago and forgot about flag - the web service is now back up and running where it was before. DNS has a 20min TTL, so shouldn't take too long to update. Webservice version is based on r29 svn, so it probably needs an upgrade. Doesn't seem to be broken though - whatever changes ffnet have made to their site don't seem to have troubled it.

Edit: although it does seem to be a tad slow.... I'll need to figure out why.
erayd is offline   Reply With Quote
Old 04-05-2009, 10:27 PM   #85
araevin
Member
araevin began at the beginning.
 
Posts: 13
Karma: 10
Join Date: Aug 2005
Location: Miami, Fl
Device: iPhone
nah Im getting some error with the meta data the story authors names come out proper but the story names come out as flag and a bunch of numbers anybody else having this problem?

Also does anybody have any instructions as to get this working localy on a mac using mamp and OS X 10.5. I'm really unsure where to go from here with my host as Im unsure what its using but I bleve its linux. I also dont know about the dependancies (tidy & cli)

Last edited by araevin; 04-05-2009 at 10:30 PM.
araevin is offline   Reply With Quote
Old 04-05-2009, 10:28 PM   #86
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Hmm, I'll have another look - can you give me a failing storyid?
erayd is offline   Reply With Quote
Old 04-05-2009, 11:38 PM   #87
araevin
Member
araevin began at the beginning.
 
Posts: 13
Karma: 10
Join Date: Aug 2005
Location: Miami, Fl
Device: iPhone
3928537
araevin is offline   Reply With Quote
Old 04-06-2009, 02:34 AM   #88
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
If you only need the html version, you can use this temporarily:
http://localhost/fflag/getstory.php?...toryid=3928537

At least until erayd gets his server all fixed up (it's down again for me). It's easy enough to convert from html to epub using Calibre.

Unfortunately, can't help you with the Mac problem as I'm unfamiliar with both Mac and Linux.

@erayd
FanFiction.Net changed handling of crossover fanfics. The code snippet above (for the ffnet_get_meta function) should fix it, except, mobileread automatically substituted the >> character instead of showing the html entity. Are you sure svn r29 still works for you? Because right now, there's an extra <img src='/static/ficons/script.png' width=16 height=16 border=0 align=absmiddle> in between <td> and <a href.
ilovejedd is offline   Reply With Quote
Old 04-06-2009, 03:35 AM   #89
ruskie
Shade
ruskie will become famous soon enoughruskie will become famous soon enoughruskie will become famous soon enoughruskie will become famous soon enoughruskie will become famous soon enoughruskie will become famous soon enough
 
ruskie's Avatar
 
Posts: 100
Karma: 546
Join Date: Mar 2009
Location: U.LC.MW.Sol.Earth.EE.SI.LJ
Device: Hanlin Jinke V3ext running OpenInkPot
Hmm note to self: next time check first to see before writing your own.... Anyway my own is shell(bash) script based and outputs xhtml only. I'm not posting it anywhere yet as I'm abstracting some stuff as I want to cover other sites that might be unrelated to fanfic... And reading their ToS(yes I did this) I have it preserve the original link, copyright, author etc and as much data as I could possibly grab. I also limit requests to one per N seconds as their ToS states(paraphrased) you are allowed to use automatic tools as long as those tools do not exceed what a normal human being would do.

Name of my tool fanfic-dl
ruskie is offline   Reply With Quote
Old 04-06-2009, 12:10 PM   #90
Mnementh
Groupie
Mnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the endMnementh knows the complete value of PI to the end
 
Mnementh's Avatar
 
Posts: 154
Karma: 31496
Join Date: Mar 2009
Location: UK
Device: Kindle 4 NT & Kobo Aura
Quote:
Originally Posted by ilovejedd View Post
I actually have working sources for TheSugarQuill and AdultFanfiction.net in case you need any for those sites. They should be attached somewhere in this thread.

Yeah I've got those thanks. Not interested in TheSugarQuill but I've just had a look at adultfanfiction, I can now honestly say I've seen my first warning for Rimming <shudder> , the format of the site is pretty poor but looks like there might be some good stuff on there.
Mnementh is offline   Reply With Quote
Reply

Tags
converter, fanfiction, fanfiction.net, grabber, lrf

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Fanfiction.net on Kindle forkyfork Amazon Kindle 26 08-07-2011 08:42 AM
bookmarks/notes grabber Reader2 Android Developer's Corner 0 10-02-2010 09:24 AM
EASY fanfiction grabber? sherryg Workshop 19 01-08-2010 03:13 AM
FLAG (Fanfiction.net Lightweight Automated Grabber) and Calibre? malkie13 Calibre 1 02-10-2009 05:43 PM


All times are GMT -4. The time now is 02:45 AM.


MobileRead.com is a privately owned, operated and funded community.