[GUI Plugin] FanFictionDownLoader

[Firedancer885]

Jimm,

please check this story: http://www.squidge.org/peja/cgi-bin/....php?sid=53103
The Rating has a few unnecessary extra characters at the end. I think the opening bracket of [Reviews - 0] got included for some reason.

[JimmXinu]

There's a '[' being included in Rating because, from an HTML point of view, that's where it belongs.

<span class="classification">Rated:</span> FRC - child safe   [<a href="reviews.php?sid=49812">Reviews</a>

But clearly that's not correct. So I'll add some special code in the squidge.org/peja adapter to remove it in the next version.

[Firedancer885]

Originally Posted by JimmXinu

There's a '[' being included in Rating because, from an HTML point of view, that's where it belongs.

<span class="classification">Rated:</span> FRC - child safe   [<a href="reviews.php?sid=49812">Reviews</a>

But clearly that's not correct. So I'll add some special code in the squidge.org/peja adapter to remove it in the next version.

Jimm,

please also check these two stories
http://www.squidge.org/peja/cgi-bin/....php?sid=51215
http://www.squidge.org/peja/cgi-bin/....php?sid=51214

The first one has the extra bracket, the second one has not.

[JimmXinu]

I have a new version needing testing again.

This one contains some metadata features that have been requested, plus quite a bit of internal refactoring to allow them.

New Features:

• Allow individual site adapters to add extra metadata entries to the standard ones. (key word: extra_valid_entries)
• Allow custom labels on title/log pages for extra metadata entries. (key words: themes_label, cliches_label, etc)
• Allow those extra metadata to be included in titlepage/logpage, or added to existing metadata entries. (key words: extra_titlepage_entries, extra_logpage_entries, include_in_genre, etc)
• Allow new, purely composite, metadata entries to be made from existing and extra metadata entries. (key words: include_in_*)
• Allow existing metadata entries to be redefined. If one site has what you think should be category and genre reversed, for example, you could swap them. (key words: include_in_category, include_in_genre, etc)
• Allow calibre custom columns to be populated from the new extra metadata entries. (key word: custom_columns_settings)
• New configuration options to add to titlepage_entries, logpage_entries or include_subject_tags for specific sites (instead of replace it). (key words: extra_titlepage_entries, extra_logpage_entries, extra_subject_tags)
• Allow the format for Published, Updated and Packaged dates on titlepage/logpage to be configured in personal.ini. (key words: dateCreated_format, datePublished_format, dateUpdated_format)

For now, I've only added a few site specific metadata that I know have been asked for:

New 'Extra' Metadata:

• dramione.org: themes, hermiones, dracos, timeline, cliches (cliches = hermiones, dracos to demonstrate the composite feature)
• www.fanfiction.net: reviews, favs, follows
• www.fictionpress.com: reviews, favs, follows (fanfiction.net & fictionpress.com share code, but not configuration)

dramione.org is the prototype for this feature; ini code:

Spoiler:

[dramione.org]

## Some adapters collect additional meta information beyond the
## standard ones.  They need to be defined in extra_valid_entries to
## tell the rest of the FFDL system about them.  They can be used in
## include_subject_tags, titlepage_entries, extra_titlepage_entries,
## logpage_entries, extra_logpage_entries, and include_in_* config
## items.  You can also add additional entries here to build up
## composite metadata entries.  dramione.org, for example, adds
## 'cliches' and then defines as the composite of hermiones,dracos in
## include_in_cliches.
extra_valid_entries:themes,hermiones,dracos,timeline,cliches
include_in_cliches:hermiones,dracos

## For another example, you could, by uncommenting this line, include
## themes in with genre metadata.
#include_in_genre:genre, themes

## You can give each new valid entry a specific label for use on
## titlepage and logpage.  If not defined, it will simply be the 
themes_label:Themes
hermiones_label:Hermiones
dracos_label:Dracos
timeline_label:Timeline
cliches_label:Character Cliches

## extra_titlepage_entries (and extra_logpage_entries) *add* to
## titlepage_entries (and logpage_entries) so you can add site
## specific entries to titlepage/logpage without having to copy the
## entire titlepage_entries line.  (But if you want them higher than
## the end, you will need to copy titlepage_entries.)
extra_titlepage_entries: themes,timeline,cliches
extra_logpage_entries: themes,timeline,cliches
extra_subject_tags: themes,timeline,cliches

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.)  There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'.  'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only.
## Make sure to keep at least one space at the start of each
## line.

custom_columns_settings:
 cliches=>#acolumn,r
 themes=>#bcolumn,a
 timeline=>#ccolumn,n

Assuming this works well, I'll take requests on extra metadata to add for other sites. I'm not, however, planning to add new metadata across the board. It's easy to say 'add reviews for all sites', but to actually add it to all 75 adapters is not.

This test version also includes bloodties-fan.com, thehookupzone.net, cover support for dramione.org and some fixes for squidge.org/peja.

Update - fixes so html/mobi/txt work.

Update 9/20/2012 - Fixes for castlefans.org author and AO3 URLs.

[ILB]

I created a custom column which is filled automatically with category - only on fanfiktion.de it does not work.
I tried to replace the metadata in personal.ini:
replace_metadata:
Kategorie=>category

Unfortunately that does not help.
Is there any other possibility?

[JimmXinu]

Originally Posted by ILB

I created a custom column which is filled automatically with category - only on fanfiktion.de it does not work.
I tried to replace the metadata in personal.ini:
replace_metadata:
Kategorie=>category

Unfortunately that does not help.
Is there any other possibility?

"For now, I've only added a few site specific metadata that I know have been asked for." And that's not one of them.

Are you sure "Kategorie" isn't already going into "category"?

[ILB]

I tried with http://www.fanfiktion.de/s/4b5cdeda00011eba06520f58/1, but the column stays empty.
The same is with characters. I let the characters add to tags per personal.ini. But with fanfiktion.de are they not in tags.
personal.ini:
include_subject_tags: extratags, genre, category, status, characters, collections, relationships, relationship, community, extracategories, extragenres, extracharacters, extraships, extrawarnings, configuration parameters, ships, status, warnings
Are there other names for categories etc. on fanfiktion.de?

[JimmXinu]

I reiterate, again:

Originally Posted by JimmXinu

For now, I've only added a few site specific metadata that I know have been asked for:

New 'Extra' Metadata:

• dramione.org: themes, hermiones, dracos, timeline, cliches (cliches = hermiones, dracos to demonstrate the composite feature)
• www.fanfiction.net: reviews, favs, follows
• www.fictionpress.com: reviews, favs, follows (fanfiction.net & fictionpress.com share code, but not configuration)

The fanfiktion.de adapter hasn't had any extra metadata added to it.

The same is with characters. I let the characters add to tags per personal.ini. But with fanfiktion.de are they not in tags.
personal.ini:

Characters aren't currently collected by the fanfiktion.de adapter. I didn't write that one, and I don't read German. Even so, using machine translation I don't even see where that site gives you characters data to add.

[Firedancer885]

Jimm,

Please make FFDL recognize this kind of url as valid.

http://archiveofourown.org/works/517067?view_adult=true

[JimmXinu]

@Firedancer885 - next version. Which may be a while if I don't hear from anyone about whether the new test version works or not.

[JimmXinu]

New test version posted here.

Fixes for castlefans.org author and AO3 URLs.

[Firedancer885]

Originally Posted by JimmXinu

New test version posted here.

Fixes for castlefans.org author and AO3 URLs.

Thanks

[AmberDragon]

A quick (and non-urgent because it's probably pretty obscure) request - if there's a problem with personal.ini that causes FFDL to throw an error, could the error message please point that out? I opened Calibre this afternoon and tried to grab some fics only to have FFDL throw up an "unbalanced parenthesis" error. I fiddled around for quite awhile trying to figure out what was wrong until I realized that I'd edited my personal.ini a few days ago. I checked it over and, sure enough, I'd accidentally dropped an opening parenthesis on an entry. I added it back and everything is back on track, but a more direct error message could have jogged my memory just a little bit faster.

[JimmXinu]

Can you give an example? I don't recall having any parenthesis in FFDL's ini file.

But I'll add it to the wishlist to do some basic format checking on persona.ini save.

[AmberDragon]

Originally Posted by JimmXinu

Can you give an example? I don't recall having any parenthesis in FFDL's ini file.

But I'll add it to the wishlist to do some basic format checking on persona.ini save.

I had this

(Buffy the vampire slayer|BTVS)=>Buffy: The Vampire Slayer

under the "replace metadata" section, and when I added a new entry underneath it I accidentally deleted the leading parenthesis somehow. Thanks for adding it to the wishlist!