[GUI Plugin] FanFicFare

[version365]

Hello, Jim.. you are very nice taking people's requests.. can I make another request too?

Currently, for downloading from Literotica, FFF prints story categories (e.g. Celebrities, Illustrated, Mature) under Erotica Tags in title page. But, those are actually categories in Lit. Tags for a certain story are different, they are located at the bottom of the story's last page. see image ( I blocked some words to make it family friendly) >> http://i.imgur.com/nd9E22A.png

Can those tags be printed in title page under Erotica Tags or something other, and move the category to Erotica Category?
( If diff chapters of a Story have diff tags, merge all the tags)

[fnabenny's]

well because you cleared that up for me. i was able to find a way openoffice. and i can convert it to epub it just takes awhile.

[readerboy7]

I'm having some trouble understanding how to use calibre cover and personal.ini for the result I want.

Basically, I want calibre to use the cover that comes with the website, (i.e. download a cover and use that cover for the book cover, but not convert the cover), except for a few specific websites. I want to not download a cover from those websites.
How do I do this?

Note that I have FFF 2.2.18 and generate cover 1.5.17

[JimmXinu]

I see I'm not getting timely thread update notices again.

Originally Posted by version365

Currently, for downloading from Literotica, FFF prints story categories (e.g. Celebrities, Illustrated, Mature) under Erotica Tags in title page. But, those are actually categories in Lit. Tags for a certain story are different, they are located at the bottom of the story's last page. see image ( I blocked some words to make it family friendly) >> http://i.imgur.com/nd9E22A.png

Can those tags be printed in title page under Erotica Tags or something other, and move the category to Erotica Category?
( If diff chapters of a Story have diff tags, merge all the tags)

I don't read that site, so I don't have a list of story URLs to look at for things. If you PM me a few story URLs that exhibit that I'll look at it.

But I will warn you that I'm not likely to collect tags from all chapters if that's the way it works.

[JimmXinu]

Originally Posted by readerboy7

Basically, I want calibre to use the cover that comes with the website, (i.e. download a cover and use that cover for the book cover, but not convert the cover), except for a few specific websites. I want to not download a cover from those websites.
How do I do this?

So you are presumably getting downloaded cover images correctly, but you don't want them for certain sites? You don't need to use personal.ini for that, but you will have to essentially overwrite the downloaded cover.

Dhat you do is set Generate Calibre Cover to 'Yes, Always', use the Plugin Generate Cover, but only configure a GC setting for the site(s) you don't want downloaded covers for. Attached is an example showing it for AO3.

[JimmXinu]

New test version posted (direct link)

2016-03-19
- Add support for 'This story has not been validated' in base_efiction_adapter
- Some mcstoriescom stories include <h3> tags that aren't title/chapter/byline.

Unless something major comes up, this is going to be the last test version before the next release.

[JimmXinu]

Originally Posted by JimmXinu

I don't read that site, so I don't have a list of story URLs to look at for things. If you PM me a few story URLs that exhibit that I'll look at it.

But I will warn you that I'm not likely to collect tags from all chapters if that's the way it works.

(version365 sent me some URLs)

It looks like that is the case. For whatever reason, those tags are only shown on the last 'page' of each chapter and they can be different for each chapter.

The architecture of FanFicFare is designed to collect metadata with as few page requests as possible for comparison before collecting story chapters separately. When FFF is showing you a progress bar "Downloading Metadata" before going to background, it's doing the metadata-only collection.

Fetching two additional pages for each chapter (the chapter's first 'page' to get the last page URL, and then the last page) during metadata-only collection strikes me as excessive.

[davidfor]

Originally Posted by JimmXinu

(version365 sent me some URLs)

It looks like that is the case. For whatever reason, those tags are only shown on the last 'page' of each chapter and they can be different for each chapter.

The architecture of FanFicFare is designed to collect metadata with as few page requests as possible for comparison before collecting story chapters separately. When FFF is showing you a progress bar "Downloading Metadata" before going to background, it's doing the metadata-only collection.

Fetching two additional pages for each chapter (the chapter's first 'page' to get the last page URL, and then the last page) during metadata-only collection strikes me as excessive.

Literotica also has the tags in the metadata of each page in the chapter. That means you can get it from the first page of the chapter. I have a version of the adapter that includes this:

    def getCategories(self, soup):
        if self.getConfig("use_meta_keywords"):
            categories = soup.find("meta", {"name":"keywords"})['content'].split(', ')
            categories = [c for c in categories if not self.story.getMetadata('title') in c]
            if self.story.getMetadata('author') in categories:
                categories.remove(self.story.getMetadata('author'))
            logger.debug("Meta = %s" % categories)
            for category in categories:
    #            logger.debug("\tCategory=%s" % category)
                self.story.addToList('category', category.title())

It includes some cleanup as older stories seem to have the story/chapter name and author in the keywords.

I call that against the parsed page in extractChapterUrlsAndMetadata. In the current adapter, it is the variable "soup1". I also call it from within the chapter processing so it can pick up extra tags from the later chapters.

It's been a while since I looked at it, but my version does two other things:

• Parses the title of chapters to normalize them to "Chapter 1" or "Part 4". If it can't do that, it uses the full title.
• Puts the chapter description at the start of each chapter with a line to separate it from the text.

There are some differences in how I parse the story text, but I think that's mainly code style. Though I do strip the outer level of div tags from around each page.

[JimmXinu]

Originally Posted by davidfor

Literotica also has the tags in the metadata of each page in the chapter. That means you can get it from the first page of the chapter. I have a version of the adapter that includes this:
...

If you have a better version of the adapter for Literotica you'd care to share, I'd be happy to have it.

I didn't write the original support for that site and I don't read it. So unless it's broken, my interest in writing and testing complex enhancement myself is pretty low.

[davidfor]

Originally Posted by JimmXinu

If you have a better version of the adapter for Literotica you'd care to share, I'd be happy to have it.

I didn't write the original support for that site and I don't read it. So unless it's broken, my interest in writing and testing complex enhancement myself is pretty low.

Better? I don't know. But it suits my purposes. I'll clean it up a bit and add your recent changes plus some options for my differences. But, it will probably be a few days before I have a chance.

And if anyone who does read Literotica wants to comment on the changes, please do.

[JimmXinu]

Originally Posted by davidfor

Better? I don't know. But it suits my purposes. I'll clean it up a bit and add your recent changes plus some options for my differences. But, it will probably be a few days before I have a chance.

And if anyone who does read Literotica wants to comment on the changes, please do.

Sounds good, thanks.

[JimmXinu]

New Version Released

Version 2.3.0 - 20 Mar 2016

- Fix for adapter_thehexfilesnet missing chapter text.

- Fixes for finestories.com, now shares code with storiesonline.net, thanks davidfor for this.

- Allow ini keyword universe_as_series in finestories.com as well as storiesonline.net.

- Fix for fimfiction.net site update -- thanks Facedeer

- New site: www.artemis-fowl.com - Thanks gcomyn!

- New site: www.naiceanilme.net - Thanks gcomyn!

- New site: fanfiction.lucifael.com - Thanks gcomyn!

- New site: buffygiles.velocitygrass. - Thanks gcomyn!

- New site: www.andromeda-web.com - Thanks gcomyn!

- New site: www.therabidreader.com - Thanks gcomyn!

- Performance improvements (extra caching) for complex metadata manipulation.

- Changes to all plugin logging output (debug) to add datetime and reduce inadvertent smilies. ("FFFEBUG")

- Additional ini options for base_xenforoforum adapters. Examples:

Spoiler:

CODE]## When given a thread URL, use threadmarks as chapter links when
## there are at least this many threadmarks. A number of older
## threads have a single threadmark to an 'index' post. Set to 1 to
## use threadmarks whenever they exist.
minimum_threadmarks:2

## When 'first post' (or post URL) is being added as a chapter, give
## the chapter this title.
first_post_title:First Post

## In normal operation, if given a post URL or a thread URL with less
## than minimum_threadmarks, the given post or the first post of the
## thread will be included as the first chapter (with chapter title
## from first_post_title) unless that post is explicitly linked to in
## the collected chapter list. First post is not included when using
## thread marks.
##
## If always_include_first_post:true, then the given or first post
## will be included as above even if it is a link in the post or even
## if threadmarks are used. Can result in a duplicated chapter.
always_include_first_post:false

## In normal operation, forumtags will only be populated when
## threadmarks are used for chapters (see minimum_threadmarks above).
## When always_use_forumtags:true, always populate forumtags.
always_use_forumtags:false[/CODE]

- Add storyUrl ini section feature. Add custom configuration for individual stories by adding personal.ini sections specifically for them. Example:

Spoiler:

[https://forums.spacebattles.com/threads/346640/]
always_include_first_post:true
first_post_title:Prologue and Chapter 1
extratags: Original, Sci-Fi, AI

- Fixes for quotev.com, author and chapter list.

- Add 'averrating' and tweak description: adapter_literotica

- Fix adapter_literotica setDescription

- Make HTTP 410 StoryDoesNotExist in tthfanfic.org, make conn fail trace debug, not error level.

- Allow a second kind of 'chapter missing' for portkey.org.

- Change fanfiction.mugglenet.com to base_efiction after site update.

- Hide user/pass in debug log, add 'View Safe personal.ini' option that also hides user/pass.

- Re-organize personal.ini config tab in plugin.

- Yet more fixes for quotev.com changes.

- Find and include chapter specific images for quotev.com.

- Fix for a bug on a TtH story (empty category tag).

- Add site specific reads, reviews for hpfandom.net.

- Fix for base_efiction_adapter stories with spans w/o class attr.

- Add support for 'This story has not been validated' in base_efiction_adapter

- Some mcstoriescom stories include <h3> tags that aren't title/chapter/byline.

- Update translations.

[rishidiams]

Originally Posted by JimmXinu

The add_to_X mechanism works by concatenating what you give it to the X setting. So if extra_valid_entries:categories and you put add_to_extra_valid_entries:mycharacters, the final result is categoriesmycharacters. But if you use add_to_extra_valid_entries:,mycharacters, then it will be categories,mycharacters which is what you want.

It's a bit non-intuitive, I admit, but otherwise there would be problems using add_to_ with multi-line values like replace_metadata.



Custom and site-specific metadata entries are never shown in the drop downs. Those only show standard metadata. You have to use custom_columns_settings, which I thought you where already doing from:

custom_columns_settings:
 mycharacters=>#characters

Not sure what to tell you, there. For me those settings are doing what I understood you to want. Note that your problem may be integrating with the other settings you have. You can only have one setting for each keyword in each section, so you have to combine them correctly.

Here's the exact personal.ini I used and the title page that results:

Spoiler:

# 2nd Test Library

[archiveofourown.org]
include_subject_tags: characters, status, fandoms, freeformtags, ships

add_to_titlepage_entries:,fandoms, freeformtags, ships,mycharacters,characters

[defaults]
add_to_extra_valid_entries:,mycharacters,mytags,mystatus
include_in_mycharacters:characters,ships

custom_columns_settings:
 mycharacters=>#extranotes
replace_metadata:
 mycharacters=>(.*)[/&](.*)=>\1 \, \2
 mycharacters,ships=>(The )?Tenth Doctor=>The Doctor.Tenth

To Kiss it All Better by Brilliantly_Strange

Series: Prompt Fics [3]
Category: Doctor Who
Genre: Cold Weather, F/M, Fluff, Hot Chocolate, Kissing, Snowball Fight
Language: English
Relationships: The Doctor.Tenth/Rose Tyler
Status: Completed
Published: 2016-03-11
Updated: 2016-03-11
Packaged: 2016-03-13 13:14:53
Rating: Teen And Up Audiences
Warnings: Creator Chose Not To Use Archive Warnings
Chapters: 1
Words: 585
Publisher: archiveofourown.org
Summary:
When Rose drinks her hot chocolate, she gets burned. The Doctor makes a proposition she can't refuse.....


Fandoms: Doctor Who
Freeform Tags: Cold Weather, Fluff, Hot Chocolate, Kissing, Snowball Fight
Relationships: The Doctor.Tenth/Rose Tyler
Mycharacters: Rose Tyler, The Doctor.Tenth

Here's an example that works for me to exclude just "Alternate Universe" if you also have "Alternate Universe(anything)":

Spoiler:

[archiveofourown.org]

# add another copy of freeformtags that doesn't have 
# replacements/in/exclude applied.  Otherwise it gets 
# thrown out in the infinite-recursion filter.
add_to_extra_valid_entries:,rawfreeformtags
include_in_rawfreeformtags: freeformtags.NOREPL

add_to_exclude_metadata_post:
# discard exact freeformtags "Alternate Universe" if there's a 
# (raw) freeformtags that is is "Alternate Universe(anythingelse)"
 freeformtags==Alternate Universe&&rawfreeformtags=~Alternate Universe.+

Thank you again!

Everything works fine as is.
However, the Ships entry will not work correctly when the characters' names are separated by an ampersand.

Using this story: http://archiveofourown.org/works/5886202

The result in my Characters column is:
Rose Tyler
amp; Clara Oswin Oswald
Clara Oswin Oswald
The Doctor.Twelfth
Twelfth Doctor


Also, thinking I'd be clever, I attempted to create a rawcharacters entry as well.

Here is the link to an example story:
http://archiveofourown.org/works/4611972

The three characters that I wish to combine are:
Rose Tyler
Rose Tyler | Bad Wolf
Bad Wolf (Doctor Who)
so that they appear only as Rose Tyler.Bad Wolf.


The relevant portions of my personal.ini are:

[archiveofourown.org]
include_subject_tags: characters, status, fandoms, freeformtags, ships

[defaults]
add_to_extra_valid_entries:,rawcharacters,mycharac ters,mytags,mystatus,rawfreeformtags,fandoms,freef ormtags
include_in_mycharacters:characters,ships
include_in_mytags:warnings,freeformtags
include_in_mystatus:status
include_in_rawfreeformtags: freeformtags.NOREPL
include_in_rawcharacters: characters.NOREPL
custom_columns_settings:
fandoms=>#fandoms
mycharacters=>#characters
mytags=>#description
mystatus=>#status
replace_metadata:
ships=>(.*)[/&](.*)=>\1 \, \2
characters=> (mentioned|\(Doctor Who\))=>
mycharacters,ships=>(The )?(Tenth|10th) Doctor$=>The Doctor.Tenth
mycharacters,ships=>(The )?(Twelfth|12th) Doctor$=>The Doctor.Twelfth
mycharacters,ships=>Bad Wolf?( - Character)$=>Rose Tyler.Bad Wolf
mycharacters,ships =>Rose Tyler \| Bad Wolf$=>Rose Tyler.Bad Wolf
mycharacters,ships =>Clara Oswald$=>Clara Oswin Oswald
add_to_exclude_metadata_post:
freeformtags==Alternate Universe&&rawfreeformtags=~Alternate Universe.+
characters==Rose Tyler&&rawcharacters=~Rose Tyler.Bad Wolf
characters==The Doctor&&rawcharacters=~The Doctor.+


First, the solo tag "Bad Wolf" does not convert to Rose Tyler.Bad Wolf, and second, the solo tag "Rose Tyler" does not get thrown out.

[version365]

Originally Posted by davidfor

And if anyone who does read Literotica wants to comment on the changes, please do.

Puts the chapter description at the start of each chapter with a line to separate it from the text.

This would be a great feature to have with existing FFF options..

Parses the title of chapters to normalize them to "Chapter 1" or "Part 4". If it can't do that, it uses the full title.

For example, take the following story..

Once Upon A Time: 3 Part Series
1. Once Upon A Time: Bad Girl
2. Once Upon A Time: Not Forgotten
3. Once Upon A Time: Shattered

Would it change the chapter names to 'Once Upon A Time : Chapter 1' or only 'Chapter 1' and strip the name 'Bad Girl'?
If that is so, then it's not a good idea..


Oh, and another issue happened..
Some literotica stories have x.xx in place of numbered rating, because the author chose the story not to be voted. For these stories FFF can't download, because obviously it expects numbers not letter.. What can be done about this? maybe, change to 0.00 when x.xx is found..

[davidfor]

Originally Posted by version365

This would be a great feature to have with existing FFF options..

Glad someone else likes the idea.

For example, take the following story..

Once Upon A Time: 3 Part Series
1. Once Upon A Time: Bad Girl
2. Once Upon A Time: Not Forgotten
3. Once Upon A Time: Shattered

Would it change the chapter names to 'Once Upon A Time : Chapter 1' or only 'Chapter 1' and strip the name 'Bad Girl'?
If that is so, then it's not a good idea..

No, the chapter headings and in the ToC would be:

• Bad Girl
• Not Forgotten
• Shattered

Basically, it is stripping the book title from the chapter title and then making them look better. But, it also tries to make sure nothing is lost. So:

• "Once Upon A Time: Ch. 1" becomes "Chapter 1"
• "Once Upon A Time: Pt. 1" becomes "Part 1"
• "Once Upon A Time: Not Forgotten" becomes "Not Forgotten"
• "Catwoman / Huntress: Eros Risen" stays as "Catwoman / Huntress: Eros Risen"

The last one, and the other chapter in that story, do not parse properly with the book title, so it gets left alone. And I suspect the author isn't intending them to be chapters of the same book. The Literotica authors listing seems to group stories that start with the same words.

Oh, and another issue happened..
Some literotica stories have x.xx in place of numbered rating, because the author chose the story not to be voted. For these stories FFF can't download, because obviously it expects numbers not letter.. What can be done about this? maybe, change to 0.00 when x.xx is found..

I'd always assumes "x.xx" meant "not enough votes". I'll fix it. For a single chapter story or were all chapters are like this, it just won't get set. Otherwise, it will be the average of the actual ratings.