[GUI Plugin] FanFicFare

[musicalife]

Originally Posted by JimmXinu

Covers aren't working because you're downloading to mobi and cover images are only supported in epub.

Will AZW3 be added in place of mobi if/when it gets removed?

[Tanjamuse]

The rating is now added to the print-out page on dark-solace.org, but FFF still doesn't collect the rating.

Here's an example URL: http://dark-solace.org/elysian/viewstory.php?sid=5241

[jennifer_voorhee]

Not sure if it's been reported yet, but TWCS Library has changed to STARS Library. I went to update some in progress stories today and noticed neither link will work in Fanficfare anymore.

http://www.twcslibrary.net brings up a "Not Found" error when you pull up the website in a browser now (when it was redirecting for a while). Similar with the stories in Calibre.

http://www.starslibrary.net/ works fine in a browser, but Fanficfare doesn't recognize it yet for downloading stories.

[Tanjamuse]

I've also added this to my Personal.ini, but it only seems to work on some sites:

add_to_replace_metadata:
    authorId=>^(.*)$=>FF\1
    storyId=>^(.*)$=>FF\1

When I try it on Fanfiction.net I get the following error which isn't much of an error:

Status
Title
Author
Comment
URL
Error
All My Love To Long Ago
Saoirse Mooney
Bad Story URL: (https://www.fanfiction.net/s/FF12293661/1/) for site: (www.fanfiction.net) Example: (https://www.fanfiction.net/s/1234/1/ https://www.fanfiction.net/s/1234/12/ http://www.fanfiction.net/s/1234/1/Story_Title http://m.fanfiction.net/s/1234/1/)
https://www.fanfiction.net/s/FF12293661/1/

If I only have the author-ID bit it works fine.

here's the link: https://www.fanfiction.net/s/1229366...ve-To-Long-Ago

[JimmXinu]

Mass reply to many questions:

Originally Posted by Tanjamuse

Can I make it site specific?

Yes.

Originally Posted by musicalife

Will AZW3 be added in place of mobi if/when it gets removed?

I don't have any plans to because Calibre's convert works so well. That's what I use for my own reading device, a Kindle.

Originally Posted by Tanjamuse

The rating is now added to the print-out page on dark-solace.org, but FFF still doesn't collect the rating.

Here's an example URL: http://dark-solace.org/elysian/viewstory.php?sid=5241

The dark-solace.org adapter uses the base_efiction code. And while they are now including a rating, they've elected to put it up in the pagetitle tag rather than in the usual metadata list where base_efiction expects to find it. Here's an example of how it normally works.

It's probably possible to customize the dark-solace.org adapter, but I don't remember how off hand. I will investigate.

Originally Posted by jennifer_voorhee

Not sure if it's been reported yet, but TWCS Library has changed to STARS Library. I went to update some in progress stories today and noticed neither link will work in Fanficfare anymore.

Are you using the most recent test version? Because as far as I know it's already fixed and waiting for the next version.

If you are having problems with the most recent test version, please post a story URL that exhibits the problem.

Originally Posted by Tanjamuse

I've also added this to my Personal.ini, but it only seems to work on some sites:

add_to_replace_metadata:
    authorId=>^(.*)$=>FF\1
    storyId=>^(.*)$=>FF\1

Bad Story URL: (https://www.fanfiction.net/s/FF12293661/1/) for site: (www.fanfiction.net)

The adapter uses the storyId internally to create the storyUrl. Which, because you've changed it, is now broken. Most of the adapters work that way because I never anticipated that someone would try to change it.

I'd like to know why you want to change the Id values before I entertain any ideas about 'fixing' this.

[Ghostdraconi]

Hi, would it be possible to add support for http://www.alternatehistory.com/, the board is running on Xenforo and they recently added threadmarks.

[Atherton25]

Wuxia's working much better now. There are still two items that are having issues.
However, the first thing of note, this is not really a problem, just a stupid issue they have with the site. Some of the index's go too far, and have empty pages because they haven't put the pages on the site yet even if they're in the index. There's at least three stories that do that. The easiest thing to do is to just to specify the ending chapter.

Now the two stories with issues. I'm not sure if they're the same issue, but it doesn't look like it.

http://www.wuxiaworld.com/absolute-choice-index is the first. I'm not sure why, it downloads the book for me but only as a single/index page with no chapters.

The second issue is http://www.wuxiaworld.com/wmw-index/ it has several chapters that dont appear correctly. Chapters 97-100, 102-103, 105,107, 110,113,115-117,119,122-124,126-128,131,133-134,136-138 - it looks like these have a https link instead of a http link. I'm not sure if those are the only ones, I just know there's about 20 chapters missing when I do a partial download it doesn't sync correctly with the chapter numbers. I looked through it and saw that those chapters were missing, but didn't look any higher than 138, as they all seemed to have the same problem.

Just of note, this story is also one of those I described at the beginning, as having more chapter links than chapters (the last few dozen are not up yet).

There were a couple others with an incorrect index chapter link that created a failure - I've emailed the site about them. They corrected the first one I found, but I haven't seen any response to the others.

I've found one problem with RoyalRoad on one of my books - http://royalroadl.com/fiction/410. I'm not sure if it's happening in other as well, but Chapter 16 (Spell 15 : The Celebrations) pulls the comments page. I have gone through dozens of books on the site, and this is the first one I've run across with this problem, but I thought I should note it.

[JimmXinu]

New test version posted (direct link)

2017-01-03
- Add chapter URL normalization to adapter_tthfanficorg for http to https change to recognize pre-existing chapters. Also protects against title changes.
- Bump test version.

[JimmXinu]

Originally Posted by Ghostdraconi

Hi, would it be possible to add support for http://www.alternatehistory.com/, the board is running on Xenforo and they recently added threadmarks.

The forum site alternatehistory.com does not appear to be a very good candidate for FanFicFare.

FanFicFare is really about download stories. And while alternatehistory.com does appear to have some threads that are written as stories, it seems to lack the standing conventions that makes it possible to pick out stories from threads on the other forums.

[JimmXinu]

Originally Posted by Atherton25

Wuxia's working much better now. There are still two items that are having issues.
...

I'll see if GComyn has anything to say about these before I look into them in detail.

Originally Posted by Atherton25

I've found one problem with RoyalRoad on one of my books - http://royalroadl.com/fiction/410. I'm not sure if it's happening in other as well, but Chapter 16 (Spell 15 : The Celebrations) pulls the comments page. I have gone through dozens of books on the site, and this is the first one I've run across with this problem, but I thought I should note it.

That chapter has some broken HTML in it that's confusing the chapter parser. There's a not a lot we can do about it when the HTML presented is that flawed.

[Tanjamuse]

Hello JimmXinu.

I have a plugin which makes some of the columns in calibre Uneditable, but the Fanfiction plugin can't download or update stories with that running, is there something you can do or do I need all columns editable when I use the FFF plugin?

Here's the error:

calibre, version 2.70.0
ERROR: Error Updating Metadata: <p>An error has occurred while FanFicFare was updating calibre's metadata for <a href='http://dark-solace.org/elysian/viewstory.php?sid=505'>A Fluffy Ending</a>.</p>The ebook has been updated, but the metadata has not.

Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 1560, in update_books_loop
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 2025, in update_metadata
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 1942, in set_custom
Exception: Trying to set entry (storyUrl) value(http://dark-solace.org/elysian/viewstory.php?sid=505) to column (#story_url) failed (Column u'story_url' is not editable)

Story Details:password:
     
comments:
     A very Spuffy Ending!
calibre_columns:

listorder:
     0
title:
     A Fluffy Ending
tags:
     [u'Post-Series', u'Angst', u'Buffy: The Vampire Slayer', u'Romance']
end:
     None
good:
     True
status:
     Add
calibre_id:
     74
begin:
     None
added:
     True
outfile:
     C:\Users\Tanja\AppData\Local\Temp\calibre_g4x65q\fanficfare_uw7d3f\A Fluffy Ending-AuthoressNebula-a0oagu.epub
showerror:
     True
all_metadata:
     authorHTML:
          <a class='authorlink' href='http://dark-solace.org/elysian/viewuser.php?uid=EF62'>AuthoressNebula</a>
     numWords:
          745
     dateCreated:
          2017-01-05 08:43:42
     cover_image:
          
     version:
          FanFicFare v2.7.4
     langcode:
          
     title:
          A Fluffy Ending
     storyUrl:
          http://dark-solace.org/elysian/viewstory.php?sid=505
     language:
          
     category:
          Buffy: The Vampire Slayer & Post-Series
     rating:
          
     status:
          Completed
     words_added:
          
     authorUrl:
          http://dark-solace.org/elysian/viewuser.php?uid=EF62
     seriesHTML:
          Fluffyverse Series
     formatext:
          .epub
     genre:
          Angst, Romance
     characters:
          Buffy Summers & Spike
     seriesUrl:
          
     site:
          Elysian Fields
     datePublished:
          2006-11-16
     numChapters:
          1
     series:
          Fluffyverse Series
     extratags:
          
     output_css:
          
body { background-color: #ffffff;
text-align: justify;
margin: 2%;
adobe-hyphenate: none; }
pre { font-size: x-small; }
sml { font-size: small; }
h1 { text-align: center; }
h2 { text-align: center; }
h3 { text-align: center; }
h4 { text-align: center; }
h5 { text-align: center; }
h6 { text-align: center; }
.CI {
text-align:center;
margin-top:0px;
margin-bottom:0px;
padding:0px;
}
.center   {text-align: center;}
.cover    {text-align: center;}
.full     {width: 100%; }
.quarter  {width: 25%; }
.smcap    {font-variant: small-caps;}
.u        {text-decoration: underline;}
.bold     {font-weight: bold;}
.big { font-size: larger; }
.small { font-size: smaller; }
     formatname:
          epub
     dateUpdated:
          2006-11-16
     description:
          A very Spuffy Ending!
     storyId:
          505
     author:
          AuthoressNebula
     lastupdate:
          Last Update Year/Month: 2006/11, Last Update: 2006/11/16
     authorId:
          EF62
     ships:
          Spike/Buffy Summers
     siteabbrev:
          dksl
     warnings:
          
site:
     dark-solace.org
publisher:
     Elysian Fields
series:
     Fluffyverse Series
pubdate:
     2006-11-16 00:00:00+01:00
icon:
     plus.png
author_sort:
     [u'AuthoressNebula']
timestamp:
     2017-01-05 08:43:42.984000+01:00
author:
     [u'AuthoressNebula']
is_adult:
     False
savemetacol:
     <p><span class='label'>Author</span>: <div class='metadata list' id='author'><ul>
<li>AuthoressNebula</li>
</ul></div><p>

<p><span class='label'>Authorhtml</span>: <div class='metadata' id='authorHTML'><a class='authorlink' href='http://dark-solace.org/elysian/viewuser.php?uid=EF62'>AuthoressNebula</a></div><p>

<p><span class='label'>Author ID</span>: <div class='metadata list' id='authorId'><ul>
<li>62</li>
</ul></div><p>

<p><span class='label'>Author URL</span>: <div class='metadata list' id='authorUrl'><ul>
<li>http://dark-solace.org/elysian/viewuser.php?uid=EF62</li>
</ul></div><p>

<p><span class='label'>Category</span>: <div class='metadata list' id='category'><ul>
<li>Buffy: The Vampire Slayer</li>
<li>Post-Series</li>
</ul></div><p>

<p><span class='label'>Characters</span>: <div class='metadata list' id='characters'><ul>
<li>Buffy</li>
<li>Spike</li>
</ul></div><p>

<p><span class='label'>Packaged</span>: <div class='metadata datetime' id='dateCreated'>2017-01-05T08:43:42.984000</div><p>

<p><span class='label'>Published</span>: <div class='metadata datetime' id='datePublished'>2006-11-16T00:00:00</div><p>

<p><span class='label'>Updated</span>: <div class='metadata datetime' id='dateUpdated'>2006-11-16T00:00:00</div><p>

<p><span class='label'>Summary</span>: <div class='metadata' id='description'>A very Spuffy Ending!</div><p>

<p><span class='label'>File Extension</span>: <div class='metadata' id='formatext'>.epub</div><p>

<p><span class='label'>File Format</span>: <div class='metadata' id='formatname'>epub</div><p>

<p><span class='label'>Genre</span>: <div class='metadata list' id='genre'><ul>
<li>Angst</li>
<li>Romance</li>
</ul></div><p>

<p><span class='label'>Hits</span>: <div class='metadata list' id='hits'><ul>
<li>676</li>
</ul></div><p>

<p><span class='label'>Lastupdate</span>: <div class='metadata list' id='lastupdate'><ul>
<li>Last Update Year/Month: 2006/11</li>
<li>Last Update: 2006/11/16</li>
</ul></div><p>

<p><span class='label'>Chapters</span>: <div class='metadata int' id='numChapters'>1</div><p>

<p><span class='label'>Words</span>: <div class='metadata int' id='numWords'>745</div><p>

<p><span class='label'>Series</span>: <div class='metadata' id='series'>Fluffyverse Series</div><p>

<p><span class='label'>Series</span>: <div class='metadata' id='seriesHTML'>Fluffyverse Series</div><p>

<p><span class='label'>Relationships</span>: <div class='metadata list' id='ships'><ul>
<li>Spike/Buffy</li>
</ul></div><p>

<p><span class='label'>Publisher</span>: <div class='metadata' id='site'>dark-solace.org</div><p>

<p><span class='label'>Site Abbrev</span>: <div class='metadata' id='siteabbrev'>dksl</div><p>

<p><span class='label'>Status</span>: <div class='metadata' id='status'>Completed</div><p>

<p><span class='label'>Story ID</span>: <div class='metadata' id='storyId'>505</div><p>

<p><span class='label'>Story URL</span>: <div class='metadata' id='storyUrl'>http://dark-solace.org/elysian/viewstory.php?sid=505</div><p>

<p><span class='label'>Title</span>: <div class='metadata' id='title'>A Fluffy Ending</div><p>

<p><span class='label'>Downloader Version</span>: <div class='metadata' id='version'>FanFicFare v2.7.4</div><p>

url:
     http://dark-solace.org/elysian/viewstory.php?sid=505
updatedate:
     2006-11-16 00:00:00+01:00
username:
     (was set, removed for security)
comment:
     Download epub completed, 1 chapters.

[JimmXinu]

Originally Posted by Tanjamuse

I have a plugin which makes some of the columns in calibre Uneditable, but the Fanfiction plugin can't download or update stories with that running, is there something you can do or do I need all columns editable when I use the FFF plugin?

To the best of my knowledge, as long as your unnamed plugin is preventing the use of db.set_custom FanFicFare will not be able to set those columns.

There is a newer custom column API FanFicFare could switch to (at the potential cost of backward compatibility to earlier calibre versions), but I have no reason to believe your plugin doesn't also block it.

[Sojiro888]

Hi! First I'd like to say a big thank you, FanficFare is the reason I'm using Calibre in the first place. I love it.

I've been having a problem on and off for a few months now, when trying to add or update a story from fanfiction.net.

I always get an error message of that type:
'utf8' codec can't decode byte 0x92 in position 44: invalid start byte
https://www.fanfiction.net/s/5536346/1/


It's always been "can't decode byte <something>" and "invalid start byte". I have looked for other case of these error messages popping up, but I've never found anything similar.

When it happens, every fanfic I try to download just fails (note that if a fanfic doesn't need to be updated, there's no problem, it only happens when it actually tries to download). The first time it happened it lasted for several days until I gave up, but coming back a few weeks later it worked once more.

Of course I always made sure to update both the plug-in and Calibre, but that had no effect. It took me a long while to discover that the problem was that the plugin was being blocked by my firewall (I use TinyWall). For some reason, despite allowing Calibre access to the internet (to download plugins and such), it can cause these issues with FanFicFare.

In order to let it work through the firewall (rather than just shutting down the firewall to update my fics) I applied the white-list to the process started by the Calibre job of running FanFicFare (just white-listing the executable for Calibre wasn't enough).



So I think that's about it. After spending hours scouring the internet in general and this thread in particular looking for people who had the same problem as me, I thought I'd share what I learned to spare someone else the grief.

Also, I thought it was a good occasion to express my gratitude for FanFicFare!

[JimmXinu]

Originally Posted by Sojiro888

...
So I think that's about it. After spending hours scouring the internet in general and this thread in particular looking for people who had the same problem as me, I thought I'd share what I learned to spare someone else the grief.

Also, I thought it was a good occasion to express my gratitude for FanFicFare!

Many thanks for sharing your findings. I believe I've seen the same or similar issues reported before without figuring it out.

I've added an FAQ about it.

[JimmXinu]

New test version posted (direct link)

2017-01-06
- Get non-label rating for base_efiction adapter_darksolaceorg.
- Strip 'Crossover' prefix from SV titles--base_xenforoforum_adapter.
- Fixes for extended/accented chars in URLs for adapter_wwwlushstoriescom.
- Bump test version.

Some of the wuxiaworld.com issues are still unaddressed.