10-01-2010, 05:16 PM | #211 | |||||
Addict
Posts: 250
Karma: 20386
Join Date: Sep 2010
Location: France
Device: Bookeen Diva, Kobo Clara BW
|
Quote:
PHP Code:
Actually, id'ing the other meta fields would be a good idea. It would also allow the import of an .html, then convert to another format without having to reparse the story on the source site. Quote:
Quote:
Quote:
Quote:
That would allow the configuration options to be removed from the code and put in a .conf. |
|||||
10-01-2010, 11:53 PM | #212 |
Zealot
Posts: 134
Karma: 146
Join Date: Apr 2008
Device: Onyx Boox Poke 2
|
OK, I've created a repo on github (http://github.com/erayd/flag), and am currently writing a basic OO skeleton for it - this will be uploaded either today or tomorrow, and then provided you're all happy with it we can start merging stuff.
Aleyx - you wanted to refactor the existing code into OO - feel free - but please hold fire until I've uploaded the skeleton; that way we can make sure nobody's duplicating work. Re logging - I'll add this as part of the base, it makes more sense to be using the same thing from the start rather than trying to achieve the same thing in several different ways. Ditto for config. Last edited by erayd; 10-01-2010 at 11:57 PM. |
10-02-2010, 02:01 AM | #213 | |||||
Member
Posts: 14
Karma: 10
Join Date: Sep 2010
Device: psp, htc g1
|
Quote:
Quote:
Quote:
Quote:
Quote:
log($level,$str): level would be an int with 0=normal output 1=warn 2=error 3=debugging. This could be configured via $CONFIG['loglevel']=array(0=>'php://stderr',1=>array('php://stderr','/var/log/whatever'),2=>'/var/log/whatever',3=>NULL); (syntax error in that, likely). --loglevel 3 on commandline would add php://stderr to everything. One could also add functionality to adjust the level per namespace but that's probably going overboard. Last edited by AtomicDryad; 10-02-2010 at 02:06 AM. |
|||||
10-02-2010, 04:08 AM | #214 | ||
Addict
Posts: 250
Karma: 20386
Join Date: Sep 2010
Location: France
Device: Bookeen Diva, Kobo Clara BW
|
Quote:
Quote:
N. |
||
10-02-2010, 04:16 AM | #215 | |
Member
Posts: 14
Karma: 10
Join Date: Sep 2010
Device: psp, htc g1
|
Quote:
I'm eying the htmlout module and am considering a few things; the css file should be selectable and optional (like --css /home/user/book.css or --nocss). If the latter, it would use 'oldschool' html. This also makes me wonder about commandline (and www) option selection in general. --css and --nocss would only apply for -f html and output modules that use htmlout as a base. Should module-specific options like that be a suboption, so there isn't a flood of options to maintain? Instead of: fflag -i 65535 -f html --css /tmp/test.css --noindex you would have: fflag -i 65535 -f html --formatopt css=/tmp/test.css,noindex or fflag -i 65535 -F nocss,noindex,simple,onepage or fflag -i 65535 -f pdf -F font=/path/to/font.ttf,keephtml,reversecolor etc. The modules themselves can handle their own options and provide a means for --help to list them. |
|
10-02-2010, 04:22 AM | #216 | ||||
Addict
Posts: 250
Karma: 20386
Join Date: Sep 2010
Location: France
Device: Bookeen Diva, Kobo Clara BW
|
Quote:
Quote:
Quote:
Quote:
N. |
||||
10-02-2010, 04:43 AM | #217 | |||
Addict
Posts: 250
Karma: 20386
Join Date: Sep 2010
Location: France
Device: Bookeen Diva, Kobo Clara BW
|
Quote:
Was the bookreader using straight HTML, or an ePub? Quote:
Quote:
fflag -i 65535 -f pdf:font=/path/to/font.ttf,keephtml,reversecolor -f html:css=stuff.css,noindex And now that I think about it, the nocss option would be useful in such a scheme: fflag -i 65535 --css stuff.css -f epub -f rtf -f html:nocss,noindex The epub and rtf outputs would use the 'global' css option, but not the html output. Which... doesn't really make sense in a real-world use case, but the concept is sound, I think. Of course, the generation itself would be a little awkward as the code stands now, what with having to reset the config arrays for each format, but once we have individual instances for formats it'll be easy as pie. N. |
|||
10-02-2010, 09:22 AM | #218 |
Member
Posts: 14
Karma: 10
Join Date: Sep 2010
Device: psp, htc g1
|
Regarding modules and commandline, there's also the question about what to do with url arguments. As is, for a new source plugin one must edit arg.inc.php and add a regexp. The following way of doing it would remove that need...the idea is real modularity; being able to drop in any valid .php in the source directory without editing the base program.
args.inc would process things like this: parse $argc/$argv/parameters, setting non module specific options (which can include disabling a module etc) stuff everything else into $arr['extra'] or ['moduleopts'] determine what modules to load, do so, and call the modules 'getinfo' function. One of the things that would do would be to add to an $argspec (or somesuch) array, which is a nested hash of regex strings and other info like this: Spoiler:
Next, any ambiguous non-options like urls or files are processed to populate a 'downloads' nested hash. I've got a working example of this. fflag -D http://archiveofourown.org/works/00000000 http://www.fanfiction.net/s/3748479/1/A_Very_Big_Weapon imjustaguy_-_A_Very_Big_Weapon.html http://fanfiction.net/u/12345 results in: Spoiler:
When fflag starts to process it's downloads: * It first checks $opt['downloads']['file'] to determine what the file's site/story is, then: * $opt['downloads']['list'], calling the appropriate source module's _get_storylist function, adding what it finds to: * $opt['downloads']['story'], which is processed last, fflag selecting sites on a per entry basis, and output filenames where specified. Last edited by AtomicDryad; 10-02-2010 at 09:29 AM. |
10-02-2010, 09:44 AM | #219 |
Member
Posts: 14
Karma: 10
Join Date: Sep 2010
Device: psp, htc g1
|
On a slightly related note, what -is- a good alternative to fanfiction.net? I'm discovering the site I used for testing is mostly pr0ns.
...stargate atlantis featuring rodney mckay pr0ns x.x |
10-02-2010, 10:38 AM | #220 | |
Addict
Posts: 250
Karma: 20386
Join Date: Sep 2010
Location: France
Device: Bookeen Diva, Kobo Clara BW
|
Quote:
PHP Code:
If we have an URL but no source in the arguments, we'll be able to invoke Sources::getSource($url); and get back an object Source that will tell us everything we need to know about the URL: how to extract the story, its meta, whether the URL contains a story or a list of stories, etc. However, not all inputs will be URLs, so I think it'd be better to allow the use of the same argument format for sources as for codecs: Code:
./fflag --some-option --other-option=/some/file -o /some/dir -i ffml:id1,id2 -i siye:id3,id4 -i ficpress:id5,id6 -f html:option1,option2 -f epub:option3,option4 url1 url2 Spoiler:
Processing the array: Spoiler:
The getStory() function will know the difference between an ID and an URL, and go from there. Idem for the export() function, which will be able to get from the opts what are the output formats and options, and call Codec instances accordingly. Erayd, would it be possible to set up a bugzilla or something? We're getting technical on many different issues, and I don't know if a forum thread is the best place? We'll need to document the code, too. How does everyone feel about Doxygen? N. |
|
10-02-2010, 10:56 AM | #221 | |
Addict
Posts: 250
Karma: 20386
Join Date: Sep 2010
Location: France
Device: Bookeen Diva, Kobo Clara BW
|
Quote:
Aside from that, it really depends on your fandoms, I guess. I'm partial to SIYE, but that's because I'm a sucker for romance. YMMV. There's a couple sites cited back in this thread, IIRC. We'll have to go back and check any source plugin to update them, anyway. Oh, and McKay pr0n? Ew. N. |
|
10-02-2010, 10:07 PM | #222 |
Zealot
Posts: 134
Karma: 146
Join Date: Apr 2008
Device: Onyx Boox Poke 2
|
Have just uploaded the skeleton so far to github - still working on it, but would be good to have some feedback on what's there so far - anything you guys want changed?
Edit: Commits are announced via flag_dev on twitter, for those who are interested in following the project, but don't have a github account. Last edited by erayd; 10-03-2010 at 12:31 AM. |
10-03-2010, 02:58 AM | #223 |
Addict
Posts: 250
Karma: 20386
Join Date: Sep 2010
Location: France
Device: Bookeen Diva, Kobo Clara BW
|
|
10-03-2010, 04:39 AM | #224 |
Addict
Posts: 250
Karma: 20386
Join Date: Sep 2010
Location: France
Device: Bookeen Diva, Kobo Clara BW
|
Just realized that AtomicDryad wouldn't see my messages to you from github, so here are some thoughts:
Spoiler:
N. |
10-03-2010, 09:29 AM | #225 | ||||||||
Zealot
Posts: 134
Karma: 146
Join Date: Apr 2008
Device: Onyx Boox Poke 2
|
Quote:
Quote:
Spyc is basically there in case the yaml extension is missing - the yaml extension is better, but depends on libyaml, and isn't bundled with PHP (it's in pear), so I figured having a fallback was a wise idea - hence that check at the beginning of flagFactory::init(). Quote:
I've got no problem with changing things around though - if you guys would rather do things a different way, I'm quite happy to change my approach. Quote:
Quote:
Quote:
Quote:
Quote:
If we are going to go down that route though, I'd rather go the whole hog and jump to PHP 6 - but I still think that requiring anything higher than 5.2 is a bad idea. |
||||||||
Tags |
converter, fanfiction, fanfiction.net, grabber, lrf |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Fanfiction.net on Kindle | forkyfork | Amazon Kindle | 26 | 08-07-2011 08:42 AM |
bookmarks/notes grabber | Reader2 | Android Developer's Corner | 0 | 10-02-2010 09:24 AM |
EASY fanfiction grabber? | sherryg | Workshop | 19 | 01-08-2010 03:13 AM |
FLAG (Fanfiction.net Lightweight Automated Grabber) and Calibre? | malkie13 | Calibre | 1 | 02-10-2009 05:43 PM |