[GUI Plugin] FanFicFare

[firefoxxy]

Hi ILB,

thank you for attaching your file. I tried using yours and the problem again only occurrs on the readers (I tried both readers), no problems when reading in the computer.

[davidfor]

Originally Posted by firefoxxy

Hi ILB,

thank you for attaching your file. I tried using yours and the problem again only occurrs on the readers (I tried both readers), no problems when reading in the computer.

Trying the fie ILB uploaded and looking at the source, the problem is that the story text is wrapped in a table. Removing those have made it display properly one my Kobo Aura H2O. I assume the renderer is trying to keep the table on a single page.

[firefoxxy]

Hi davidfor,

Thank you for your answer. Jimm already noticed the table-code. As the epub-epub conversion that Jimm suggested earlier didn't work, I'm editing the epubs manually now.

[glenm125]

As I guess you're already aware , deliberate wait cycles in the foreground metadata fetch can make the host behave a little strangely. In my case, my mouse can disappear, and I'm blocked from switching mouse control to other hosts in my virtual desktop (using Synergy). But I understand that those waits are important because of FFN throttles.

I see some switches in the config file to tune the delays (in the FFN section), per the comment they're intended to reduce user inconvenience. However, I'm leery of reducing the delay because I don't want FFN to block me. Especially since, due to not being home for weeks at a time, I might have several 10s of stories to update - yesterday when I used "Get Story URLs from Email" there were around 65 stories to check. It seems likely I could hit a throttle.

Indeed, I'd be happy to increase the delay .. I honestly don't care how long the downloads take, so long as I can continue to work on something else, or perhaps read another book I already have. Take hours. Fine.

Is there a switch, or could there be , to perform the metadata fetch in a background job, maybe in the same background job that will eventually run to download the content? Or even just outright move the metadata fetch to the background (ie. no switch)? Or is there some reason folks want to watch it happen?

Thanks for reading.

[JimmXinu]

When you are updating existing stories in your library (you select books and use 'Update Existing Fanfiction Books') there is a checkbox for 'Background Metadata'. You can set it on by default in the plugin config.

There isn't a background metadata fetch option when adding/updating by URL or from email (which uses the 'by URL' code).

[MerlinMama]

I've always wondered - and sorry if this is really obvious, but it isn't to me - does it still fail for username/password and adult, if those are in the personal.ini?

[firefoxxy]

I'm back again with a new question.

I have my personal.ini set up to strip ship-names down to first name only. Now when reading fics about the Potters kids I get confused when the tags say 'Lily & Severus' instead of 'Lily Luna & Severus'. I tried to change the code and played around a bit, but I can't get it to work.

Here is an example story.

And here is my personal.ini:

Spoiler:

## Some sites also require the user to confirm they are adult for
## adult content.  Uncomment by removing '#' in front of is_adult.
is_adult:true

replace_br_with_p: true

[epub]
## include images from img tags in the body and summary of stories.
## Images will be converted to jpg for size if possible.  Images work
## in epub format only.  To get mobi or other format with images,
## download as epub and use Calibre to convert.
include_images:true

## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
keep_summary_html:true

## If set, the first image found will be made the cover image.  If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
make_firstimage_cover:true

## Resize images down to width, height, preserving aspect ratio.
## Nook size, with margin.
image_max_size: 580, 725

[archiveofourown.org]
username:XXXXXXXX
password:XXXXXXXX
include_in_characters: characters, ships
include_in_genre: genre,warnings,freeformtags
sort_ships:true
## some sites include images that we don't ever want becoming the
## cover image.  This lets you exclude them.
cover_exclusion_regexp:(userinfo\.gif|user\.png|88x31\.png)
exclude_metadata_pre:
 ships,characters,genre,freeformtags=~Other (Additional|Character|Relationship) Tags to Be Added
 warnings==Creator Chose Not To Use Archive Warnings
 warnings==No Archive Warnings Apply
 genre=~.*-centric
 genre==Alternate Ending
 genre=~Alternate Universe - Canon( Divergence)?
 genre=~([tT]he )?[aA]uthor [rR]egrets.*
 genre=~([tT]he )?[aA]uthor [iI]s.*
 genre==Avengers Tower
 genre=~.*Canon Compliant.*
 genre=~Canon Divergence.*
 genre=~Community:.*
 genre==Cute
 genre==Drabble
 genre==Het
 genre==Non-Explicit Sex
 genre=~Originally Posted.*
 genre==Post-Canon
 genre==Pseudo-Incest
 genre=~([mM]ale)?Slash
 genre=~[sS]orry [nN]ot [sS]orry
 genre=~.*Tumblr.*
 genre=~Wordcount.*
replace_metadata:
## general clean-up
 category=> (- All Media Types|- Fandom|\(.*[mM]ovies\)|\(Movie [0-9]+\)|\(Comics\)|\(TV\)|(MCU))$=>
 characters=>\(?mentions of\)? =>
 characters=> \([Ch]haracter\)=>
 characters=> ?\((cameo|briefly|if you squint|sort of)\)=>
 characters=> - Character=>
 characters=>^Other mentions$=>
 characters,category,genre=> - [fF]reeform=>
 ships=>(.*)[/&](.*) \(?[Ff]riendship(?: only)\)?=>\1 & \2
 ships=>([bB]ackground )? - Relationship=>
 ships,characters,genre=>[mM]inor =>
## end general clean-up
 category=>.*Harry Potter.*=>Harry Potter Works
 category=>.*Marvel.*|.*Avengers.*|.*Iron Man.*|.*Thor.*|.*Loki.*|^Captain America.*=>Marvel Universe
 characters=> \((Marvel|MCU|Harry Potter)\)$=>
## second name first for certain family-name-first names:
 characters=>^(Uchiha|Uzumaki|Haruno|Hyuuga|Hatake|Inuzuka|Umino) *([^ ]+)$=>\2 \1
 characters=>^(Namikaze|Aburame|Akimichi|Inuzuka|Hoshigaki|Yamanaka) *([^ ]+)$=>\2 \1
 characters=>^Alastor “Mad-Eye” Moody$=>Alastor Moody
 characters=>^Amora.*$=>Amora
 characters=>^Angrbo.*$=>Angrboda
 characters=>^Avengers$=>The Avengers
 characters=>^Balde?r.*$=>Baldr
 characters=>^Betty.*$=>Betty Ross
 characters=>^Bruce.*$=>Bruce Banner
 characters=>^Byleistr.*$=>Byleistr
 characters=>^(Clint.*|Hawkeye.*)$=>Clint Barton
 characters=>^Draco.*$=>Draco Malfoy
 characters=>^Fandral.*$=>Fandral
 characters=>^F.rbauti.*$=>Farbauti
 characters=>^Fenri[sr].*$=>Fenrir
 characters=>^Frigga.*$=>Frigga
 characters=>^Director Fury$=>Nick Fury
 characters=>^Harry.*$=>Harry Potter
 characters=>^Heimdall.*$=>Heimdall
 characters=>^Hel.*$=>Hel
 characters=>^Helblindi.*$=>Helblindi
 characters=>^Hermione.*$=>Hermione Granger
 characters=>^Hogun.*$=>Hogun
 characters=>^(James)?(Bucky.*|.*Barnes)$=>Bucky Barnes
 characters=>^Jane F.*$=>Jane Foster
 characters=>^Jarvis.*$=>Jarvis
 characters=>^J(o|ö)rmungand.*$=>Jörmungand
 characters=>^Laufey.*$=>Laufey
 characters=>^Lily Evans.*$=>Lily Potter
 characters=>^Loki.*$=>Loki
 characters=>^Lucius.*$=>Lucius Malfoy
 characters=>^Narcissa.*$=>Narcissa Malfoy
 characters=>^(Natasha.*|Black Widow)$=>Natasha Romanov
 characters=>^(OC.*|OFC|OMC|Original Characters?)$=>Original Character
 characters=>^Odin.*$=>Odin
 characters=>^Pepper.*$=>Pepper Potts
 characters=>^Peter Parker.*$=>Peter Parker
 characters=>^(Phil.*|(Agent )?Coulson)$=>Phil Coulson
 characters=>^Remus.*$=>Remus Lupin
 characters=>^Ron.*$=>Ron Weasley
 characters=>^Sam Wilson.*$=>Sam Wilson
 characters=>^Severus.*$=>Severus Snape
 characters=>^Sif.*$=>Sif
 characters=>^Sigyn.*$=>Sigyn
 characters=>^Sleipnir.*$=>Sleipnir
 characters=>^Steve.*$=>Steve Rogers
 characters=>^Thanos.*$=>Thanos
 characters=>^The Other.*$=>The Other
 characters=>^Thor.*$=>Thor
 characters=>^Tony.*$=>Tony Stark
 characters=>^(Voldemort.*|Tom Riddle.*)$=>Tom Riddle Jr.
 characters=>^Volstagg.*$=>Volstagg
 characters=>^Wanda.*$=>Wanda Maximoff
 characters=>(.*) \(Naruto\)$=>\1
## second name only for certain family-name-first names:
 ships_CHARS=>^(Uchiha|Uzumaki|Haruno|Hyuuga|Hatake|Inuzuka|Umino) *([^ ]+)$=>\2
 ships_CHARS=>^(Namikaze|Aburame|Akimichi|Inuzuka|Hoshigaki|Rock|Yamanaka) *([^ ]+)$=>\2
## full name for Potter children
 ships_CHARS=>^(James Sirius|Lily Luna|Albus Severus).*$=>\1
## strip each ships char down to first name only.
 ships_CHARS=>^([^ ]+).*$=>\1
 ships=>^James/Steve$=>Bucky/Steve
 ships=>^Lady/Thor$=>Loki/Thor
 ships,genre=>^[fF]rost[iI]ron( - [fF]reeform)?$=>Loki/Tony
 genre=>^[aA]ftermath of ([sS]exual [aA]ssault|[vV]iolence|[wW]ar)$=>Aftermath of Trauma
 genre=>^Age Regression.*$=>De-aging
 genre=>^Alcohol Abuse(/Alcoholism)?^$=>Substance Abuse
 genre=>^Alpha/Beta/Omega Dynamics$=>Omega Verse
 genre=>^Alternate Reality$=>Alternate Universe
 genre=>^([aA]lternate [uU]niverse) -(.*)$=>\1\,\2
 genre=>^(Loki.*|Steve.*|Tony.*|Thor.*|Light) [aA]ngst$=>Angst
 genre=>^[aA]ngst and [fF]luff and [sS]mut$=>Angst\,Fluff\,smut
 genre=>^[aA]ngst (and|with( a)?) [hH]appy [eE]nding$=>Angst\,Happy Ending
 genre=>^([aA]ngst) and (.*)$=>Angst\,\1
 genre=>^[aA]sexual (.*)$=>Asexual!\1
 genre=>^Attempted Rape/Non-Con$=>Attempted Rape
 genre=>^(.*) (au|AU)$=>\1\,\2
 genre=>^[aA][uU]( [uU]niverse)?$=>Alternate Universe
 genre=>^Avenger Loki$=>Avenger!Loki
 genre=>^(Book 4.*|Harry Potter and the Goblet of Fire)$=>4th Year
 genre=>^[bB]ottom Severus( Snape?)=>bottom!Severus
 genre=>^([cC]anonical|[iI]mplied/[rR]eferenced) [cC]haracter [dD]eath$=>Character Death
 genre=>^Complete$=>Completed
 genre=>^[cC]rack$=>Humour
 genre=>^(Loki.*|Steve.*|Tony.*|Thor.*) [fF]eels$=>Feels
 genre=>^[fF]emale(\s|!)Loki.*$=>Female Loki
 genre=>^([eE]ventual|dD]omestic) [fF]luff=>Fluff
 genre=>^([fF]luff) and (.+)=>Fluff\,\1
 genre=>^.* [iI]s [aA] [gG]ood [bB]ro$=>Friendship
 genre=>^[gG]ood Loki$=>Good!Loki
 genre=>^Grief/Mourning$=>Grief-Mourning
 genre=>[hH]umor=>Humour
 genre=>^([eE]motional )?[hH]urt/[cC]omfort$=>Hurt-Comfort
 genre=>^Intersex Loki$=>Intersex!Loki
 genre=>^J[oö]tunn?(\s|!)Loki$=>Jötunn!Loki
 genre=>^Jotunn \| Frost Giant$=>Frost Giant
 genre=>^[mM]emory [lL]oss$=>Amnesia
 genre=>^[mM]entions? of (.*)=>\1
 genre=>^[mM]ild (.*)=>\1
 genre=>^[nN]orse [rR]eligion & [lL]ore$=>Norse Mythology
 genre=>^.*[oO][oO][cC]=>Out of Character
 genre=>^Past [aA]buse$=>Abuse
 genre=>^[pP]ast Child Abuse$=>Child Abuse
 genre=>^[pP]ast Torture$=>Aftermath of Torture
 genre=>(.*)?Post-Traumatic Stress Disorder(.*)?=>PTSD
 genre=>^Pre-Avengers \(2012\)$=>Pre-Avengers
 genre=>^Post-Avengers \(2012\)$=>Post-Avengers
 genre=>^Pre-Avengers: Age of Ultron \(Movie\)$=>Pre-Avengers: Age of Ultron
 genre=>^Post-Avengers: Age of Ultron \(Movie\)$=>Post-Avengers: Age of Ultron
 genre=>^Pre-Thor \(2011\)$=>Pre-Thor
 genre=>^Post-Thor \(2011\)$=>Post-Thor
 genre=>^Pre-Thor: The Dark World$=>Pre-Thor: The Dark World
 genre=>^Post-Thor: The Dark World$=>Post-Thor: The Dark World
 genre=>^[pP]rotective.*$=>Protectiveness
 genre=>^([eE]xplicit )?[sS]ex.*$=>smut
 genre=>^([eE]ventual|[sS]hameless) [sS]mut=>smut
 genre=>^Sexual Content$=>smut
 genre=>^[sS]oul [mM]ates$=>Soulmates
 genre=>^[tT]op Harry( Potter)?=>top!Harry
 genre=>^[uU][sS][tT]$=>Unresolved Sexual Tension
 genre=>(Canon-Typical|Graphic Depictions Of) Violence=>Violence

## This section will override anything in the system defaults or other
## sections here.
[overrides]
## default varies by site.  Set true here to force all sites to
## collect series.
#collect_series: true

Could you have a look once again? Thanks.

[JimmXinu]

Originally Posted by MerlinMama

I've always wondered - and sorry if this is really obvious, but it isn't to me - does it still fail for username/password and adult, if those are in the personal.ini?

If you have correctly set username/password and is_adult in personal.ini, FFF will not stop and ask for those (for that site). That's kind of the point of having them in personal.ini.

[JimmXinu]

Originally Posted by firefoxxy

I have my personal.ini set up to strip ship-names down to first name only.

Now when reading fics about the Potters kids I get confused when the tags say 'Lily & Severus' instead of 'Lily Luna & Severus'.

Here is an example story.

It shouldn't really come as a surprise that when you strip ship-names down to first name only, it does exactly that.

The solution that seems to me to be easiest to handle and most straightforward is to make the cases where you want to keep two word names into one word names before doing the 'first name only' line. (Assuming there aren't a huge number of such cases.) Then you can change them back to two word names afterwards--or not if you want to save some typing.

replace_metadata:
## Make these two word names one word before the 
## 'first name only' line.
 ships_CHARS=>Lily Luna=>Lily_Luna
 ships_CHARS=>Albus Severus=>Albus_Severus

## strip each ships char down to first name only
 ships_CHARS=>^([^ ]+).*$=>\1

## Change the two words names back into two words (if desired)
 ships_CHARS=>Lily_Luna=>Lily Luna
 ships_CHARS=>Albus_Severus=>Albus Severus

Alternatively, remove the space instead of using underscore: Lily Luna=>LilyLuna

[glenm125]

Originally Posted by JimmXinu

If you have correctly set username/password and is_adult in personal.ini, FFF will not stop and ask for those (for that site). That's kind of the point of having them in personal.ini.

Yeah, that works well for me. Good thing, cuz fanfic authors and sites are really conservative, so lots of the fics I read are M.

[firefoxxy]

Originally Posted by JimmXinu

It shouldn't really come as a surprise that when you strip ship-names down to first name only, it does exactly that.

The solution that seems to me to be easiest to handle and most straightforward is to make the cases where you want to keep two word names into one word names before doing the 'first name only' line. (Assuming there aren't a huge number of such cases.)

lol! Heh, yes I know that it does what it should - quite too well even in this case

Your solution works like a charm. And fortunately there are only three children/names I want to change. Thank you!

[Atherton25]

Alright, this is an accumulation post, so there's several items I'm going to be bringing up here. I've been making notes when I noticed something and wanted to find out if it was something that could be fixed/modified.

1. Cover images dealing with RoyalRoadL.com. The earlier post that used cover_exclusion_regexp was great, I know how to block images. However, there's a problem with at least one story I've run into that I'm trying to figure out.
The story, Everybody Loves Large Chests ID#8894 (The story is more comedy than anything, and about a shape-shifting treasure chest. It does have some NSFW scenes later on in the story though, so I figured best not to direct link it.

When I download the story, the image that comes in is the same one that was discussed and blocked earlier on (the grey image with 200x300 in the center). It does have a cover image which is an SVG file. I cut/pasted the cover image into the story, but every time I update it reverts to that stupid empty grey image. I've looked through the basic log that calibre shows, and there's nothing in there. There is at least 2 other stories that do this to me, but I can't remember all of them I know "Galactic Fist of Legend" #4552 does it as well, but the others I'm not sure about.

In the logs the following message IS shown:
calibre_plugins.fanficfare_plugin.fanficfare.write rs.writer_epub(432): writer_epub: no new cover, has old cover, write image.

I've looked a bit (including searching) for a way to block cover overwrites. I'd like to do it for the specific story, or figure out where the hell the 'new' (old) cover is coming from and block it too, but I'm not sure how to find that out.... and that brings me to the next item on my list, also to do with RoyalRoad.

2. Finding/Blocking a specific image. When I download a story called How to date an android. http://royalroadl.com/fiction/7556 (...so far it's safe, though in the future it may not be.) I get a smiley face cover image. it's like 21x21 pixels. I remove it, update and get it again. Blocking the cover change for this story specifically (as with the one above) is a possible option, or blocking the image altogether. I can't figure out where the icon is coming from. I turned my adblock software off and looked through the story for several chapters including the front page and can't find the damn icon anywhere so I can use an exclusion with it. Is there a way for me to look through a log somewhere and see where it's coming from?

3. RoyalRoad again. This involves the stars, It's not a huge deal but I figured I'd ask as I've got all sorts of other questions as well. I'm really big about having all the information and it's great that we get the stars. When I add them to my 'score' column, which was for a few different sites, I get up to 6 decimals. Is there anyway to round this to 2? I can probably figure it out if I modify the column to only show up to 2 decimals, but I believe there's a reason I left it as is. I don't want to screw up another system/site that uses it, or add another column, so is there a way to put something in my personal.ini to fix it?

4. Unlikely you can help, but I figured I might as well ask. This happens on multiple sites, and I ran across it a few times lately and it made me wonder if I could fix it.

Several sites have a looser formatting option (unlike some like ff.net that's very specific/tight) royalroad and lucafel are some of them. There's nothing wrong with it, but occasionally when downloading stories I get 3-4 empty lines between paragraphs for stories. I've gone through and edited several stories directly through HTML to remove this issue.

Each site has some slightly different issues, but generally there's only a couple per site. My question is: Is there a way to 'edit' the raw HTML through the INI to remove/replace? Say: <br><br> and replace with <br> or something similar? Some sites have exact problems, and some have multiple ones, but with a few lines of replace code It would fix these up and make it a lot easier to read on my kindle without manually editing. When converting, sometimes calibre can fix this through the conversion, but it doesn't always work.

5. Fictionhunt. Semi-problem. Nothing huge, but something I saw and figured I should ask about.

After downloading a few book, I went through and put in a proper comment on several of these stories, finding the description elswhere. Because of the nature of the site, there wasn't a proper description available. I'm not sure what happened but I 'updated' a couple of the stories to include all the fanfiction data and promptly lost the description I'd put in. I only noticed this yesterday when I was converting a story. It was only a few that this happened with, and I was able to get the information back in.

My question is if there is a way in the personal.ini to prevent fictionhunt from updating the comment area when downloading information? I searched through the default.ini and couldn't find an option to do something like it. I know you can set a overall one for new only, but is there something site specific?

[JimmXinu]

Originally Posted by Atherton25

1. Cover images dealing with RoyalRoadL.com. ...
The story, Everybody Loves Large Chests ID#8894 ...

When I download the story, the image that comes in is the same one that was discussed and blocked earlier on (the grey image with 200x300 in the center). It does have a cover image which is an SVG file.
...
There is at least 2 other stories that do this to me, but I can't remember all of them I know "Galactic Fist of Legend" #4552 does it as well, but the others I'm not sure about.

In the logs the following message IS shown:
calibre_plugins.fanficfare_plugin.fanficfare.write rs.writer_epub(432): writer_epub: no new cover, has old cover, write image.

I just checked those with default settings, and the stories' cover images were applied correctly.

What's happened, I suspect, is that you picked up the default cover in those books (before the change to exclude them and before the authors put in the specific covers), and your settings don't update the cover inside the epub by default. But you do have the setting to update the calibre cover from the epub by default.

Those settings aren't in personal.ini, but in the GUI. Basic -> 'Default Update EPUB Cover when Updating EPUB' and Calibre Cover -> 'Update Calibre Cover (from EPUB)'.

If you like your current settings, you can update the EPUB cover to match the calibre cover using Calibre's Polish -> 'Update cover in the book files' feature.

Originally Posted by Atherton25

2. Finding/Blocking a specific image. When I download a story called How to date an android. http://royalroadl.com/fiction/7556 (...so far it's safe, though in the future it may not be.) I get a smiley face cover image. ...
Is there a way for me to look through a log somewhere and see where it's coming from?

After downloading the story, open the epub in calibre's Edit Book feature. Scroll down the file list to see that the image is named as ffdl-0.jpg (cover.jpg will be a copy). Search for ffdl-0.jpg in the text and you'll find that it's in file0057.xhtml, "Chapter 53 Part B" and the original image URL is in longdesc (for reasons) http://royalroadl.com/forum/images/smilies/smile.png

Removing all smilies from cover consideration seems a decent idea, so I'll add this to defaults in next version:

[royalroadl.com]
cover_exclusion_regexp:(imgur.com/dzOACJf.png|/forum/images/smilies/)

Originally Posted by Atherton25

3. RoyalRoad again. This involves the stars, ... I get up to 6 decimals. Is there anyway to round this to 2? I can probably figure it out if I modify the column to only show up to 2 decimals ...

I'd go with formatting the custom column, but you can fudge the decimal value with a replace_metadata:

replace_metadata:
 stars=>^([0-9]\.[0-9]{0,2}).*$=>\1

Note that 'How to date an android' had a stars score of 4 exactly when I tested, which won't be 4.00 unless you format your float column.

Originally Posted by Atherton25

4. Unlikely you can help, but I figured I might as well ask. ...
... occasionally when downloading stories I get 3-4 empty lines between paragraphs for stories. ...

I usually only apply it on a per-story basis, but some users use replace_br_with_p across the board:

## Some sites/authors/stories use br tags instead of p tags for
## paragraphs.  This feature uses some heuristics to find and replace
## br paragraphs with p tags while preserving scene breaks.
replace_br_with_p: true

Originally Posted by Atherton25

5. Fictionhunt. Semi-problem. Nothing huge, but something I saw and figured I should ask about.

After downloading a few book, I went through and put in a proper comment on several of these stories, finding the description elswhere. ...

My question is if there is a way in the personal.ini to prevent fictionhunt from updating the comment area when downloading information? I searched through the default.ini and couldn't find an option to do something like it. I know you can set a overall one for new only, but is there something site specific?

The closest I can think of is to suppress description entirely for the site. Which for fictionhunt.com isn't a terrible idea.

[fictionhunt.com]
include_in_description:

[bblatt]

can it auto update after it downloads the books

[JimmXinu]

Originally Posted by bblatt

can it auto update after it downloads the books

You can update stories downloaded to epub, and you can configure it to update stories from email notifications on command, but FanFicFare cannot do completely automatic updates.