[GUI Plugin] FanFicFare

[dwraith0877]

So this should work, then, to get around Calibre's turning "with" in the author field into an "&"?
add_to_replace_metadata:
author=>Old Man with a Pen=>Old Man with-a Pen

[JimmXinu]

Originally Posted by dwraith0877

So this should work, then, to get around Calibre's turning "with" in the author field into an "&"?
add_to_replace_metadata:
author=>Old Man with a Pen=>Old Man with-a Pen

Yes. You can also universally change it so Calibre doesn't split authors on 'with' by changing the authors_split_regex tweak in Calibre Preferences. It's a change I use myself. See attached.

[FaceDeer]

I seem to have run into a potentially destructive problem with downloading and updating stories that have embedded images, and I think it might be a new thing that just started happening. Unfortunately I've just noticed it and I'm going to be rather busy tomorrow so I haven't delved too deeply yet but I'll dump what information I've gathered so far.

An update was made to a story on FimFiction I've been following, The Enchanted Library. It's a very nicely produced story that has a couple of pieces of artwork at the top of almost every chapter. I told FFF to update it with my usual "Update EPUB Always" setting, and after it had updated I noticed that my copy of the story had lost almost all of its images and showed them as broken links (it went from 51 images to 6). Checking back through a bunch of other recent updates from the past couple of days shows other stories with similar image loss, I can post more examples if desired.

Fortunately I keep backups, so I dug up my un-updated copy: epub hosted on Mediafire. I can reproduce the problem reliably by putting that file in Calibre and updating it. I'm surprised and worried for two reasons:

• Previously I was pretty sure that "Update EPUB Always" left old chapters and their associated images untouched even when the image links were genuinely broken, and indeed when I update this story the log claims that all FFF did was download 1 new chapter. Why is it going through existing chapters it shouldn't be updating and nuking the older images?
• The image links aren't broken when I visit the source website, so FFF is apparently failing to download images that it was succeeding at previously.

I've got the settings include_images:true and no_image_processing:true in my personal.ini.

Fimfiction did recently switch from http: URLs to https:, and that caused me a bit of grief as FFF was considering all chapters to be "new" when I updated because the old chapter urls no longer matched. I fixed that with a script that went through all my epubs and updated the http: chapter source addresses to https: ones, though, so I don't think this problem is related to that. I did a bunch of recovery work on my epubs when that happened and images seemed to be downloading fine in the immediate aftermath. I think it may be the most recent FFF update that's causing this new behaviour, but I haven't tried rolling back a version yet to confirm that for certain.

Edit: Strange! I just finished re-downloading The Enchanted Library completely from scratch and it got the images correctly. In fact, I got 82 images compared to the old copy's 51. So whatever's going on is specific to updating my old copy. I suppose it's possible I inadvertently broke something when I ran that http/https conversion script on my epubs, though I'm still pretty certain there was a period between then and now when updating stories with images was working correctly. I really wish I had time to dig into this more, I hate dropping a bug report like this when it might be something I messed up locally. Figured it was worth the risk in this case if something's wiping images out of stories on update.

[JimmXinu]

New FIX Version Released

Version 2.12.0 - 20 May 2017
- FIX for issue with Epub Update discarding existing images in Epub. Oops. :-( (Came from ffnet img referer fix.)

Thanks to FaceDeer for pointing this out. I feel really bad that I didn't notice it in a month of testing.

[dwraith0877]

Okay... I've got FFF adding the ratings and status from storiesonline... but if I have FFF download a new story, instead of downloading the epub straight from SOL then having FFF update it, it does not add the other story tags, like Ma/Fa, etc...
I have tried adding ,forumtags on the add_to_include_subject_tags line, so it looks like
add_to_include_subject_tags:,tagrating,forumtags
but it is not working....
how do I go about this?

[davidfor]

Originally Posted by dwraith0877

Okay... I've got FFF adding the ratings and status from storiesonline... but if I have FFF download a new story, instead of downloading the epub straight from SOL then having FFF update it, it does not add the other story tags, like Ma/Fa, etc...
I have tried adding ,forumtags on the add_to_include_subject_tags line, so it looks like
add_to_include_subject_tags:,tagrating,forumtags
but it is not working....
how do I go about this?

The line I have is:

include_subject_tags:sitetags

Using "add_to_include_subject_tags" should work as well.

[dwraith0877]

Originally Posted by davidfor

The line I have is:

include_subject_tags:sitetags

Using "add_to_include_subject_tags" should work as well.

Thanks... I put it under defaults and it worked great...

[FaceDeer]

Sorry to hear that the image download problem wasn't just me. Glad to hear it was quick to fix, though. And the image referrer fix appears to be actually getting images that weren't being got before, so this could mean that I'm going to come out of this with fewer broken image links than I went into it. So a net positive in the end!

For anyone who may have used FFF to update image-bearing stories in the narrow window of time where this bug was in effect, I recommend the Quality Check plugin for finding books that might have been damaged by it.

You can set Quality Check's search scope to be the entire library or just selected books.
Under the "Epub structure" menu there is a "Check broken image links" option, it will mark ebooks that have any broken image links in them. If you don't have backups then an "Overwrite always" update may bring back the broken images.

[JimmXinu]

New test version posted

2017-05-12

- Fix for asianfanficscom site change.
- New site wattpad.com -- thanks, botmtl!

[JimmXinu]

Originally Posted by FaceDeer

Sorry to hear that the image download problem wasn't just me. Glad to hear it was quick to fix, though. So overall a positive?

I am glad you noticed it so promptly, but I'm going to be annoyed with myself a while yet.

Originally Posted by FaceDeer

For anyone who may have used FFF to update image-bearing stories in the narrow window of time where this bug was in effect, I recommend the Quality Check plugin for finding books that might have been damaged by it.
...

That's a good suggestion. However, users should be aware of a few points:

• Site fanfiction.net doesn't include images in the text (covers only). You don't need to worry about stories from there.
• You only need to worry about stories updated with v2.11.0 and v2.10.X test versions. If you only use the released version, that's updates since 2017-05-17 or since 2017-04-17 if you use test versions.
• It's common, in my experience and depending on site, for stories to link to images that aren't available anymore. Quality Check may still report missing images even though this problem has been corrected.

[Tanjamuse]

Would it be possible to add a feature that could add more than one page with links?

Much like we can add a lot of story links?

[Atherton25]

A couple of things.

First, Wattpad, thats wonderful! I've done some basic testing and so far it looks good. There is a small thing though, the 'category' section should probably be considered 'tags' instead. It can get a lot of different assignments, and isn't really something that should be considered a category.

Next I have a question. I've wanted to do this for a while but never could figure what I was doing wrong. Wattpad has made me sit down and try to get it right, and still fail. An example of what I want is to take the tag (or currently category) of harrypotter, fix its name and make sure that 'characters' gets the tag Harry Potter. Now based on previous posts I've read, I thought:

add_to_replace_metadata:
category=>harrypotter=>Harry Potter&&characters=>Harry Potter

would do it, but apparently not. Am I doing something wrong?

https://www.wattpad.com/story/110100995 is an example story

Here's my full ini file used in my testing database, just in case something else is interfering.

Spoiler:

[wattpad.com]
#username:XXXXXXXX
#password:XXXXXXXX
extratags:FFF
replace_metadata:
site=>www.wattpad.com=>WattPad
category=>harrypotter=>Harry Potter&&characters=>Harry Potter
calibre_std_tags_label:Tags
custom_columns_settings:
"WATT2"=>#collections,n
"FFF"=>#library,a
storyId=>#id,n
category=>#tags
datePublished=>#datepublished
dateUpdated=>#dateupdated
dateCreated=>#datecreated
numChapters=>#chapters
numWords=>#words,r
warnings=>#warnings,a
status=>#status,r
characters=>#characters,a
genre=>#genre,n
ships=>#ships,a
rating=>#rating,r
favs=>#favorites,r
reviews=>#reviews,r
follows=>#follows,r
"FFF"=>#library,n
titlepage_entries: seriesHTML,ships,status,datePublished,dateUpdated, dateCreated,numChapters,numWords,reads,site,warnin gs,characters,category,genre,calibre_std_tags,desc ription



[epub]
include_images:true
image_max_size: 600, 800
convert_images_to: jpg
grayscale_images: false
include_images:true
never_make_cover: true
no_image_processing: true
remove_transparency: true

[defaults]
website_encodings: auto, utf8, Windows-1252
collect_series: true
add_genre_when_multi_category: Crossover
default_value_status:Unknown
include_subject_tags:extratags, genre, category, characters, ships, rating

Oh and one last question. Is it possible to do a custom colum entry based on tag data (or currently category data)? I have a column I use do designate if a story is a webfiction or fanfiction and since wattpad is a little different and offers both, I'm trying to write something that basically says if category/tag says fanfiction, add to the custom column 'Fanfiction'. And I'd like to have a something that says if there ISNT a category/tag saying fanfiction then add Webfiction instead.

[JimmXinu]

(multi-reply)

Originally Posted by Tanjamuse

Would it be possible to add a feature that could add more than one page with links?

Much like we can add a lot of story links?

I believe that's been requested before. It's not a feature I want to provide because I worry that it would encourage excessively long download lists.

Originally Posted by Atherton25

First, Wattpad, thats wonderful! I've done some basic testing and so far it looks good. There is a small thing though, the 'category' section should probably be considered 'tags' instead. It can get a lot of different assignments, and isn't really something that should be considered a category.

botmtl is the developer to thank for wattpad support, not sure if they read this forum.

Wattpad tags are put into a site specific metadata field: tags. The default config when copies tags to the standard metadata field category since wattpad appears to only have one kind of tag.

Originally Posted by Atherton25

Next I have a question. I've wanted to do this for a while but never could figure what I was doing wrong. Wattpad has made me sit down and try to get it right, and still fail. An example of what I want is to take the tag (or currently category) of harrypotter, fix its name and make sure that 'characters' gets the tag Harry Potter.

So wattpad is even worse than AO3 for tags. Lovely. Witness this example with tags harry and potter vs harrypotter in your example.

Short answer: You can try, but you'll never get it all perfect without manually fixing tags.

Longer answer: You'll need to copy all of tags to characters using include_in_characters, then normalize and filter down to what you consider characters using the replace_metadata and include/exclude features.

Originally Posted by Atherton25

Now based on previous posts I've read, I thought:

add_to_replace_metadata:
 category=>harrypotter=>Harry Potter&&characters=>Harry Potter

would do it, but apparently not. Am I doing something wrong?

You seem to assume the part after && is an assignment. It is a condition that must be matched. Your line says: "Replace harrypotter with Harry Potter in category if metadata field characters already has an entry that matches Harry Potter.

See replace_metadata doc.

Originally Posted by Atherton25

Oh and one last question. Is it possible to do a custom colum entry based on tag data (or currently category data)? I have a column I use do designate if a story is a webfiction or fanfiction and since wattpad is a little different and offers both, I'm trying to write something that basically says if category/tag says fanfiction, add to the custom column 'Fanfiction'. And I'd like to have a something that says if there ISNT a category/tag saying fanfiction then add Webfiction instead.

By default, FanFicFare adds 'FanFiction' to Calibre's tags (using extratags) for most sites, including wattpad. You could clear extratags for wattpad in personal.ini and rely on the site tags, but your own example doesn't include a Fanfiction tag.

To populate a custom column, you'd add a new site-specific metadata field with add_to_extra_valid_entries, copy tags to it with include_in_X, then filter and normalize down to just what you want in that column with replace_metadata and include/exclude.

Frankly, I consider it all a Sisyphean task. The world seems to be moving to generic tags in all things--Calibre's already there with it's tags field. I choose not to fight it.

[Atherton25]

My own example wasn't actually fanfiction, I just grabbed something that said harrypotter as an example. It's someone grabbing text from pottermore and making it into a book sort of. At least I think that's what it was, as I said I just grabbed something I know said harrypotter in the tags.

I'ce seen wattpad is bad about Freeform Tags. Blah. If places are going to do that they should have 2 types of tags, one normal with a setup more akin to genre qith limitations, things like romance, drama, supernatural and such and a freeform section where people can do whatever the heck they want as tags. It's their site, access to the stories is good enough I suppose

Alright, now I figured out what I was doing wrong. I don't know why I'd gotten it in my head that the && inclusion meant that it was a part of the replace, thus making the tag appear in the category & character in the stated case, but I understand what you said. I must have got it mixed up a long time ago, because I've tried several times to make it work like I thought and never could. Drove me nuts. I guess sometimes when you get something in your head.

As for copying tags and doing the include/exclude I'll probably get to that, but maybe it'll be just better to leave as is. The only fix I suggest for the future is the tag/category fix.

Thanks for the time straightening my head out.

[JimmXinu]

Originally Posted by Atherton25

... The only fix I suggest for the future is the tag/category fix.

Either category gets filled with tags or it's empty. botmtl made the call to fill it by default.

You can reverse it in your settings with:

[wattpad.com]
include_in_category: