[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by ij26

Incidentally, while editing personal.ini, I also noticed:

[www.fimfiction.net]
is_adult:true
#fail_on_password: false

Does uncommenting that allow password-protected stories to be downloaded? Why is it commented by default?

The default is for FanFicFare to stop and prompt you for a password. Setting fail_on_password true will cause password protected stories to fail entirely.

[ij26]

Originally Posted by JimmXinu

The default is for FanFicFare to stop and prompt you for a password. Setting fail_on_password true will cause password protected stories to fail entirely.

So the commenting is redundant in this case; OK.

Originally Posted by ij26

strictly speaking you don't have to log in to Fimfiction to see mature links; you just need the appropriate cookie set.

Thinking further, supporting Fimfiction logins could allow downloading from a user's bookshelves, or from search results within those shelves, without the user's having to make the selves public. (That's probably a bit harder than just displaying mature stories, though.)

[JimmXinu]

Originally Posted by ij26

Thinking further, supporting Fimfiction logins could allow downloading from a user's bookshelves, or from search results within those shelves, without the user's having to make the selves public. (That's probably a bit harder than just displaying mature stories, though.)

Originally Posted by JimmXinu

New test version posted (direct link).

2015-11-24

- Set is_adult cookie for FimF on 'Get URLs from Page' if set in personal.ini.

I didn't remember that FimF uses an adult cookie when I replied the first time. I don't have any plans to add login.

[JimmXinu]

New test version posted.

2015-11-25

- Fix for hpfandom.net change so one-shot stories will work correctly.

[ij26]

Originally Posted by JimmXinu

I didn't remember that FimF uses an adult cookie when I replied the first time. I don't have any plans to add login.

Disappointing, but fair enough; and the is_adult cookie feature for Fimfiction works, so thanks.

Further suggestions/requests:

Adding tags for Fimfiction stories' content ratings (Everyone/Teen/Mature); ideally including the option to update the tags for stories already downloaded. Is there a particular reason that that isn't currently done?

Support for www.fimfetch.net, a site that archives Fimfiction stories (albeit without character tags). (Maybe an option to automatically try there if a Fimfiction link doesn't work? That might be overboard, though.)

Support for Fimfiction series metadata. I've been adding it manually so far. This might be tricky, since Fimfiction just has individual stories linked to each other rather than formally named and numbered series; but it might be possible to develop a set of rules for naming and numbering series. Maybe when you download a story with series metadata, the plugin could automatically check your library for other stories in the series and update them accordingly, so prequels and interquels correctly bump prior stories to later numbers? Could still lead to messy situations; No Rest For The Traitor is listed a sequel to Horn And Wings, but Horn And Wings is password-protected, so you can't see what stories it might or might not be linked to. Would you want No Rest listed as part 2 of the series "Horn And Wings" (even though it's apparently actually fifth in a series starting with "Imbalanced")? I'm not sure.

Option to enter multiple URLs at once into "Get Story URLs from Web Page".

As for other sites to support: this is getting into feature creep territory; but at the same time, there are a lot of places on the Internet where you can find a wall of links to interesting content and wish you could easily bulk-save them to an organized collection to read at your leisure. Maybe a general website-saving feature that saves pages in HTMLZ format, reading title/author/date metadata where available but not bothering with any Readability-style cleanup of comments and sidebars and other junk? Or is there already a plugin that does that?

Specific sites that it'd be nice to be able to bulk-save from:
* Project Gutenberg.
* Project Gutenberg Australia.
* Wikisource.
* MIT Technology Review (example).
* Infinity Plus.
* Reddit (text posts).
* Less Wrong (including tags).
* Voat (text posts).
* Pastebin.
* halo.bungie.org.
* scp-wiki.net (though on the other hand, there's already someone producing regularly updated compilation ebooks).
* bdsmlibrary.com and its various mirrors.
* aryion.com (example collection).
* Google Docs/Drive (examples: The Sun is Tired, Peppy Greyskull's stories).
* Deviantart (example).
* Pony Fiction Vault (pull ebook links from blog posts?).
* Equestria Daily (download the story or stories that the blog post links to, but with the tags that EqD has assigned to it?).
* Equestria After Dark (same suggestion).
* creepypasta.org.
* creepypasta.com.
* Fluffybooru. My hopes aren't high for this one, since it's a pretty obscure and disreputable site. Example of a text post; formatting guide. Ideally, tags would be saved, "author:" tags would be converted into author metadata, and possibly the uploader name would be used as the author if one wasn't otherwise listed (since tagging wasn't very consistently enforced there last I checked).
* Speaking of disreputable: Gurochan. "Stories" are posted in threads mixed with discussion; but you could download all the posts in a thread posted with the same username and email as the OP.
* Archive.org backups of all of the above; including the old [orz.]gurochan.org.

[JimmXinu]

Originally Posted by ij26

Adding tags for Fimfiction stories' content ratings (Everyone/Teen/Mature); ideally including the option to update the tags for stories already downloaded. Is there a particular reason that that isn't currently done?

Rating is already collected. You can modify what's included in calibre's tags with the include_subject_tags ini parameter.

Originally Posted by ij26

Support for www.fimfetch.net, a site that archives Fimfiction stories (albeit without character tags). (Maybe an option to automatically try there if a Fimfiction link doesn't work? That might be overboard, though.)

I'm not interested in writing code to support an archive site for an existing supported site. If someone else wants to write it, I will review and accept it.

Originally Posted by ij26

Support for Fimfiction series metadata. I've been adding it manually so far. This might be tricky, since Fimfiction just has individual stories linked to each other rather than formally named and numbered series; ...

FanFicFare already collects prequel and sequel links. Due to lack of series name and allowing multiple sequels, that's as far as I plan to take it. This is a bit like AO3's tags--there's only so much we can reasonably do with the design decisions made by the source site.

Originally Posted by ij26

Option to enter multiple URLs at once into "Get Story URLs from Web Page".

That is a deliberate design decision. FanFicFare is meant to be a story downloading tool, not a large scale harvesting tool. Admittedly, the line is fuzzy, but I judge that to be over it.

Originally Posted by ij26

As for other sites to support: this is getting into feature creep territory; but at the same time, there are a lot of places on the Internet where you can find a wall of links to interesting content...

Outside the scope of this tool. FanFicFare is specifically intended for producing ebook versions of fiction stories for ebook readers and archiving. While there is support for a few original fiction sites, fanfiction is the primary focus--it's right in the name.

FanFicFare is an open source project under the Apache License. If you want to repurpose parts of the code into a more general page scrape -> ebook took, you can. But I'm not interested in making or supporting such a tool.

[saluli]

Recently all the storys I download from fanfiction.net are coming with the fandom in blank, and for what I remember, this worked before. So, I am assuming I broke something when editing the personal.ini...

Right now, my personal ini is like this:

[defaults]


is_adult:true
keep_summary_html:true
replace_br_with_p:true
#keep_style_attr: false
#keep_title_attr: false
strip_chapter_numbers:true
add_chapter_numbers:true
chapter_title_strip_pattern:^[0-9]+[\.: -]+
keep_in_order_characters:true
keep_in_order_category:true
keep_in_order_freeformtags:true
keep_in_order_genre:true
keep_in_order_fandoms:true
keep_in_order_ships:true
characters_label:Characters
ships_label:Relationships
series_label:Series
seriesUrl_label:Series URL
extratags_label:Extra Tags
titlepage_entries: seriesHTML,category,genre,language,characters,ships,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,numWords,site,description
collect_series: true
include_subject_tags:extratags, genre, category, characters, ships, status
extratags: Fanfiction
add_genre_when_multi_category: Crossover
chapter_title_addnew_pattern:${index}. (new) ${title}
include_titlepage: true
include_tocpage: true
add_to_custom_columns_settings: 
 fandoms=>#fandom
 description=>#summary



[www.fanfiction.net]
never_make_cover: true
username:
password:
add_genre_when_multi_category: Crossover
slow_down_sleep_time:0.5
add_to_extra_valid_entries:,oneshot,fandoms,description
include_in_oneshot:status
# add oneshot to the tags that are included in epub & calibre.
add_to_include_subject_tags:,oneshot
add_to_custom_columns_settings: 
 fandoms=>#fandom
 description=>#summary
 



And one more thing. If I remove a tag from include_subject_tags, it will not appear on the Calibre tags column. But, does this mean it will not appear on the custom columns?

[JimmXinu]

Originally Posted by saluli

Recently all the storys I download from fanfiction.net are coming with the fandom in blank, and for what I remember, this worked before. So, I am assuming I broke something when editing the personal.ini...

The fandoms metadata is specific to AO3. ffnet and most other sites use category instead.

Originally Posted by saluli

And one more thing. If I remove a tag from include_subject_tags, it will not appear on the Calibre tags column. But, does this mean it will not appear on the custom columns?

Custom column settings are unrelated to include_subject_tags.

[saluli]

Thanks for the info! This is make my Calibre a better place.

[ij26]

Originally Posted by JimmXinu

Rating is already collected. You can modify what's included in calibre's tags with the include_subject_tags ini parameter.

Works; thanks.

Edit: Any particular reason why everything isn't collected by default? Also, is there any way to retrieve the Sex and Gore tags where applicable? I tried "sex", "gore" and "content" as tag subjects, but they don't seem to work; and I notice that Fimfiction's API URL doesn't seem to show them (story tagged Sex and Gore; story not tagged Sex or Gore).

Originally Posted by JimmXinu

I'm not interested in writing code to support an archive site for an existing supported site. If someone else wants to write it, I will review and accept it.



FanFicFare already collects prequel and sequel links. Due to lack of series name and allowing multiple sequels, that's as far as I plan to take it. This is a bit like AO3's tags--there's only so much we can reasonably do with the design decisions made by the source site.



That is a deliberate design decision. FanFicFare is meant to be a story downloading tool, not a large scale harvesting tool. Admittedly, the line is fuzzy, but I judge that to be over it.



Outside the scope of this tool. FanFicFare is specifically intended for producing ebook versions of fiction stories for ebook readers and archiving. While there is support for a few original fiction sites, fanfiction is the primary focus--it's right in the name.

FanFicFare is an open source project under the Apache License. If you want to repurpose parts of the code into a more general page scrape -> ebook took, you can. But I'm not interested in making or supporting such a tool.

Fair enough. Fluffy pony stories could sort of technically be considered MLP fanfic, but it's a stretch. What about halo.bungie.org, though?

[Tanjamuse]

What have I done wrong with my email-setup? I've tried with labels to be able to separate different fanfiction sites. I use Gmail, and as long as I use INBOX I don't have any trouble, but I would like to have the labels instead.

calibre, version 2.45.0
ERROR: Unhandled exception: <b>error</b>:SELECT command error: BAD ['Could not parse command']

calibre 2.45 [64bit]  isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.9
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Successfully initialized third party plugins: Read MP3 AudioBook metadata && Modify ePub && Count Pages && View Manager && Generate Cover && Reading List && Wiki Reader && Find Duplicates && Quality Check && FanFicFare && User Category && Manage Series && EpubMerge && MultiColumnSearch && Consolidate All Library Metadata
Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 456, in get_urls_from_imap_menu
  File "calibre_plugins.fanficfare_plugin.fanficfare.geturls", line 185, in get_urls_from_imap
  File "imaplib.py", line 661, in select
  File "imaplib.py", line 1087, in _simple_command
  File "imaplib.py", line 917, in _command_complete
error: SELECT command error: BAD ['Could not parse command']

[JimmXinu]

Originally Posted by Tanjamuse

What have I done wrong with my email-setup? I've tried with labels to be able to separate different fanfiction sites. I use Gmail, and as long as I use INBOX I don't have any trouble, but I would like to have the labels instead.

calibre, version 2.45.0
ERROR: Unhandled exception: <b>error</b>:SELECT command error: BAD ['Could not parse command']
...

Without setting what your settings are, all I can do is guess from what you have given me.

I speculate that you've entered more than one label, probably in a comma separated list, instead of the one label/folder name that FanFicFare is expecting.

FanFicFare is designed to only read from one label/folder. If you want separate labels for separate sites, I suggest you also apply a common 'FanFic Notice' label to all of them and use it for FanFicFare.

[Tanjamuse]

Here's a screenshot of my settings. I've deleted my mail account and password, if you want another image with that, I'll send you one on a PM.

[JimmXinu]

Originally Posted by Tanjamuse

Here's a screenshot of my settings. I've deleted my mail account and password, if you want another image with that, I'll send you one on a PM.

Turns out this is a bug in Python's imaplib library--it doesn't quote the folder name when there's a space. Not sure why is hasn't been fixed before.

However, because it *does* check for starting/ending " and *not* quote strings that have them, it's safe for us to manually quote it.

New test version (direct link).

- Improvements to basic chapter_title_strip_pattern, add warning comments for same.
- Add dblquotes around folder name on IMAP lookup.

[Tanjamuse]

Thanks, I'll test in a little bit.