[GUI Plugin] FanFicFare

[MrTyton]

Hey.

Fanfiction.net downloads have been broken since the update yesterday.

Here's an example:

Status | Title | Author | Comment | URL
Error | Pokemon: The Origin of Species | daystar721 | too many values to unpack | https://www.fanfiction.net/s/9794740/1/

Download FanFiction Book
FFF:INFO:calibre_plugins.fanficfare_plugin.jobs(36 ):FanFicFare v2.2.10
FFF:INFO:calibre_plugins.fanficfare_plugin.jobs(40 ):Adding jobs for URLs:
FFF:INFO:calibre_plugins.fanficfare_plugin.jobs(42 ):https://www.fanfiction.net/s/9794740/1/
FFF:INFO:calibre_plugins.fanficfare_plugin.jobs(78 ):Logfile for book ID 342 (Pokemon: The Origin of Species)
FFF:INFO:calibre_plugins.fanficfare_plugin.jobs(79 ):urlhttps://www.fanfiction.net/s/9794740/1/) id342)
FFFEBUG:calibre_plugins.fanficfare_plugin.jobs(1 39):cookiejar:<LWPCookieJar[]>
FFFEBUG:calibre_plugins.fanficfare_plugin.fanfic fare.adapters.adapter_fanfictionnet(100):URL: https://www.fanfiction.net/s/9794740/1/
FFFEBUG:calibre_plugins.fanficfare_plugin.fanfic fare.adapters.base_adapter(282):################## ###################
pagecache HIT: https://www.fanfiction.net/s/9794740/1/
FFF:WARNING:calibre_plugins.fanficfare_plugin.fanf icfare.adapters.base_adapter(346):Caught an exception reading URL: https://www.fanfiction.net/s/9794740/1/ sleeptime(0) Exception too many values to unpack.
FFFEBUG:calibre_plugins.fanficfare_plugin.fanfic fare.adapters.base_adapter(282):################## ###################
pagecache HIT: https://www.fanfiction.net/s/9794740/1/
FFF:WARNING:calibre_plugins.fanficfare_plugin.fanf icfare.adapters.base_adapter(346):Caught an exception reading URL: https://www.fanfiction.net/s/9794740/1/ sleeptime(0.5) Exception too many values to unpack.
FFFEBUG:calibre_plugins.fanficfare_plugin.fanfic fare.adapters.base_adapter(282):################## ###################
pagecache HIT: https://www.fanfiction.net/s/9794740/1/
FFF:WARNING:calibre_plugins.fanficfare_plugin.fanf icfare.adapters.base_adapter(346):Caught an exception reading URL: https://www.fanfiction.net/s/9794740/1/ sleeptime(4) Exception too many values to unpack.
FFFEBUG:calibre_plugins.fanficfare_plugin.fanfic fare.adapters.base_adapter(282):################## ###################
pagecache HIT: https://www.fanfiction.net/s/9794740/1/
FFF:WARNING:calibre_plugins.fanficfare_plugin.fanf icfare.adapters.base_adapter(346):Caught an exception reading URL: https://www.fanfiction.net/s/9794740/1/ sleeptime(9) Exception too many values to unpack.
FFF:ERROR:calibre_plugins.fanficfare_plugin.fanfic fare.adapters.base_adapter(348):Giving up on https://www.fanfiction.net/s/9794740/1/
FFF:ERROR:__init__.py(1193):too many values to unpack
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.base_adapter", line 337, in _fetchUrlOpened
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.base_adapter", line 283, in _fetchUrlRawOpened
ValueError: too many values to unpack
FFF:INFO:calibre_plugins.fanficfare_plugin.jobs(25 7):Exception: {u'url': u'https://www.fanfiction.net/s/9794740/1/', u'author': [u'daystar721'], u'status': u'Error', u'icon': u'dialog_error.png', u'good': False, u'outfile': u'C:\\Users\\Joshua\\AppData\\Local\\Temp\\calibre _ornebz\\fanficfare_gldim1\\Pokemon_ The Origin of Species-daystar721-lllyl7.epub', u'calibre_id': 342, u'updatedate': datetime.datetime(2015, 8, 1, 7, 49, 48, tzinfo=SafeLocalTimeZone()), u'pubdate': datetime.datetime(2013, 10, 26, 2, 50, 19, tzinfo=SafeLocalTimeZone()), u'series': '', u'comments': u"<p>Enter the world of Pok\xe9mon from a rational perspective. Instead of starting his journey in ignorance, Red has spent his years studying the creatures so central to his world... and he doesn't quite agree with all the information in his books. No time for rookie mistakes here: he's on a quest to discover the true nature of Pok\xe9mon, and maybe even find out where they really come from.</p>", u'username': 'NoneGiven', u'comment': u'too many values to unpack', u'epub_for_update': u'C:\\Users\\Joshua\\AppData\\Local\\Temp\\calibre _ornebz\\fanficfare_gldim1\\old-342-kxyzqd.epub', u'timestamp': datetime.datetime(2015, 8, 1, 16, 7, 25, 321000, tzinfo=SafeLocalTimeZone()), u'publisher': u'www.fanfiction.net', u'all_metadata': {'status': u'In-Progress', 'author': u'daystar721', 'storyUrl': u'https://www.fanfiction.net/s/9794740/1/', 'formatext': '', 'language': u'English', 'siteabbrev': u'ffnet', 'langcode': u'en', 'genre': u'Adventure', 'warnings': u'', 'seriesHTML': '', 'authorHTML': u"<a class='authorlink' href='https://www.fanfiction.net/u/5118664/daystar721'>daystar721</a>", 'series': '', 'extratags': u'FanFiction', 'lastupdate': u'Last Update Year/Month: 2015/08, Last Update: 2015/08/01', 'numWords': u'141,079', 'seriesUrl': '', 'authorUrl': u'https://www.fanfiction.net/u/5118664/daystar721', 'rating': u'M', 'ships': u'', 'formatname': '', 'description': u"Enter the world of Pok\xe9mon from a rational perspective. Instead of starting his journey in ignorance, Red has spent his years studying the creatures so central to his world... and he doesn't quite agree with all the information in his books. No time for rookie mistakes here: he's on a quest to discover the true nature of Pok\xe9mon, and maybe even find out where they really come from.", 'datePublished': u'2013-10-26 02:50:19', 'version': u'FanFicFare v2.2.10', 'reviews': u'374', 'storyId': u'9794740', 'cover_image': '', 'dateCreated': u'2015-08-01 16:07:25', 'favs': u'772', 'numChapters': u'22', 'title': u'Pokemon: The Origin of Species', 'dateUpdated': u'2015-08-01 07:49:48', 'site': u'www.fanfiction.net', 'category': u'Pok\xe9mon', 'follows': u'1,014', 'characters': u'Blue O./Green O. (male), Leaf, Red', 'output_css': '', 'authorId': u'5118664'}, u'tags': [u'Leaf', u'Adventure', u'In-Progress', u'Blue O./Green O. (male)', u'Pok\xe9mon', u'Red', u'FanFiction'], u'is_adult': False, u'listorder': 0, u'title': u'Pokemon: The Origin of Species', u'site': 'www.fanfiction.net', u'added': False, u'calibre_columns': {}, u'begin': None, u'author_sort': [u'daystar721'], u'end': None, u'password': ''}:too many values to unpack
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.jobs", line 142, in do_download_for_worker
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.base_adapter", line 429, in getStoryMetadataOnly
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.adapter_fanfictionnet", line 104, in doExtractChapterUrlsAndMetadata
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.adapter_fanfictionnet", line 85, in _fetchUrl
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.base_adapter", line 322, in _fetchUrl
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.base_adapter", line 350, in _fetchUrlOpened
ValueError: too many values to unpack

FFF:INFO:calibre_plugins.fanficfare_plugin.jobs(83 ):
Successful:


FFF:INFO:calibre_plugins.fanficfare_plugin.jobs(85 ):
Unsuccessful:
https://www.fanfiction.net/s/9794740/1/

Been getting this with all of the updates that I've been trying with fanfiction.net

[JimmXinu]

Originally Posted by MrTyton

Fanfiction.net downloads have been broken since the update yesterday.

Here's an example:

 FFF:INFO:calibre_plugins.fanficfare_plugin.jobs(36 ):FanFicFare v2.2.10

Been getting this with all of the updates that I've been trying with fanfiction.net

They're working fine for me, including with that story. And trust me--if it was broken for everybody I'd be hearing about it.

Your debug output still says v2.2.10. Trying installing the plugin again making sure to restart calibre.

[MrTyton]

Originally Posted by JimmXinu

They're working fine for me, including with that story. And trust me--if it was broken for everybody I'd be hearing about it.

Your debug output still says v2.2.10. Trying installing the plugin again making sure to restart calibre.

Restarted calibre, reinstalled, didn't work. Rebooted computer, then did work. Not really sure why (maybe win10 issue idk), but I'll take it. Thanks!

[JimmXinu]

Originally Posted by MrTyton

Restarted calibre, reinstalled, didn't work. Rebooted computer, then did work. Not really sure why (maybe win10 issue idk), but I'll take it. Thanks!

I dunno why, but it's not the first time I've heard of this happening.

[jjm6383]

I'm getting the ' too many values to unpack' with all updates. I don't know how to get a debug report.

One of the stories I'm getting it with is:

Status
Title
Author
Comment
URL
Error
Punch Drunk and Reeling
Brennah_K
too many values to unpack
http://archiveofourown.org/works/312793

I checked manually, and there is a new chapter. The User Plugin list in Calibre shows version 2.2.11 for Fanficfare.

[JimmXinu]

Originally Posted by jjm6383

I'm getting the ' too many values to unpack' with all updates. I don't know how to get a debug report.
...

If you look at the last few prior posts, a different user had the same problem and cleared it by rebooting their computer. I suggest trying it.

I don't know why that would help, but given the error, I suspect part of the plugin zip file has been cached somewhere and isn't refreshing properly.

[rinn]

Hi! I've been downloading from archiveofourown.org, and if there's a locked (restricted to logged-in users) fic in a list FFF stops collecting the Additional Tags for that fic and any fics that come after it, regardless of whether the rest are locked or not. It doesn't give an error message so I only noticed by accident.

For example, if I make an anthology from this page:
http://archiveofourown.org/users/Isl...ndom_id=721553

The first 8 fics are fine but starting from the locked 9th fic, the Additional Tags don't show up on the title pages. The same thing happens with 'Download from URLs' and 'Make anthology Epub from URLs.' I put in a random list of 6 fics where the 4th fic was locked and the rest were unlocked, and in both cases the first 3 turned out fine, but from the 4th fic on, no additional tags.

[MerlinMama]

Is something from FanFicFare adding the following 4 lines to each chapter of updated stories, even chapters that were already there?

<meta name="chapterurl" content="https://www.fanfiction.net/s/3691757/1/"/></meta>
<meta name="chapterorigtitle" content="1. Chapter 1"></meta>
<meta name="chaptertoctitle" content="1. Chapter 1"></meta>
<meta name="chaptertitle" content="1. Chapter 1"></meta>

I have not actively enabled any of the new features, but did start after I updated the plugin. It clutters up the files while, for me, being pointless. Can I turn this off somewhere (meaning which feature is causing it, if from FFF)? Is there some terrible problem that might occur if I remove these lines. Oh, and the last chapter updated does not have the closing </meta> but other chapters do.

[JimmXinu]

Originally Posted by rinn

Hi! I've been downloading from archiveofourown.org, and if there's a locked (restricted to logged-in users) fic in a list FFF stops collecting the Additional Tags for that fic and any fics that come after it, regardless of whether the rest are locked or not. It doesn't give an error message so I only noticed by accident.
...

Can you please provide a few specific story URLs that exhibit the problem? The list page you linked to changes frequently and I'm not seeing a problem with what it currently shows me.

[JimmXinu]

Originally Posted by MerlinMama

Is something from FanFicFare adding the following 4 lines to each chapter of updated stories, even chapters that were already there?

Yes. That's part of a couple new features, one that allows FFF to add new chapters out of order if the site uses unique chapter URLs (primarily the new forumxeno sites) and for the mark_new_chapters feature.

Originally Posted by MerlinMama

...
Can I turn this off somewhere (meaning which feature is causing it, if from FFF)? Is there some terrible problem that might occur if I remove these lines. Oh, and the last chapter updated does not have the closing </meta> but other chapters do.

If this additional metadata causes actual problems, I'd like to hear about it.

If you find the invisible metadata offensive for some reason, you can customize the xhtml used at the beginning of each chapter file to remove it. Default is like so:

[epub]
chapter_start:
 <?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <title>${chapter}</title>
 <link href="stylesheet.css" type="text/css" charset="UTF-8" rel="stylesheet"/>
 <meta name="chapterurl" content="${url}"></meta>
 <meta name="chapterorigtitle" content="${origchapter}"></meta>
 <meta name="chaptertoctitle" content="${tocchapter}"></meta>
 <meta name="chaptertitle" content="${chapter}"></meta>
 </head>
 <body>
 <h3>${chapter}</h3>

See this wiki page about custom output.

I don't see anywhere in my files where </meta> close tags are missing, nor anyway for them to go missing. Can you provide an example epub?

[MerlinMama]

Here is just the top of one of the stories I added just as a test. In this one, none of the chapters have the </meta> closing tag (not even at the bottom of the page, I did a search through all chapters and it came up empty for it. In the example I originally used, only the last, updated chapter was missing the </meta> but it had been added to all the other chapters, which didn't have those tags originally. Hope that makes sense...

Spoiler:

<?xml version='1.0' encoding='utf-8'?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>1. Chapter 1</title>
<link href="stylesheet.css" type="text/css" charset="UTF-8" rel="stylesheet"/>
<meta name="chapterurl" content="https://www.fanfiction.net/s/11360264/1/"/>
<meta name="chapterorigtitle" content="1. Chapter 1"/>
<meta name="chaptertoctitle" content="1. Chapter 1"/>
<meta name="chaptertitle" content="1. Chapter 1"/>
</head>
<body>
<h3>1. Chapter 1</h3>
<div class="storytextp" id="storytextp" style="padding:0 0.5em 0 0.5em;">
<div class="storytext xcontrast_txt nocopy" id="storytext"><p style="text-align:center;"><strong>~ Prologue ~</strong></p>
<p><em>Minerva McGonagall looked at the young couple standing in her office. She’d been head down over preparations for the Halloween Feast, always a problematic event that resulted in alcohol poisoning, dragging students out from behind bushes, and usually at least one assault. It was her least favorite school celebration but Dumbledore insisted on keeping it with sanctimonious noises about Samhaim and tradition and fun. How a party that was mostly about candy and purloined fire whiskey had anything to do with harvest festivals and the advent of the Horned God McGonagall had no idea but every suggestion that she add a bit of historical relevance to the curriculum was pooh-poohed.</em></p>

If you need me to add anything else for you to check, let me know.

[JimmXinu]

    <link href="stylesheet.css" type="text/css" charset="UTF-8" rel="stylesheet"/>
    <meta name="chapterurl" content="https://www.fanfiction.net/s/11360264/1/"/>
    <meta name="chapterorigtitle" content="1. Chapter 1"/>
    <meta name="chaptertoctitle" content="1. Chapter 1"/>
    <meta name="chaptertitle" content="1. Chapter 1"/>

See how the tags end with />, just like the link tag for stylesheet? That makes it a self closing tag.

For most purposes <x></x> and <x/> are functionally identical. I left these tags as <meta...></meta> because something (epubcheck maybe? or nook? don't remember now) complained about self closing meta tags.

So I speculate that the tool you're using to view the source with replaced <meta...></meta> with <meta.../>. It's not uncommon, especially for 'inspectors'.

tl;dr: <meta...></meta> == <meta.../>. If it's not actively causing problems, don't worry about it.

[MerlinMama]

Ah, yes I did notice that the chapters that didn't have the actual closing tag did end with that forward slash, but I didn't equate it with other self-closing tags I know of. Probably simply because of the other that DID have the separate tag. I see these simply when using Calibre editor, and I see both options in the same story when updating. New story only shows the self-closing. A little strange, but as long as no problem I won't wonder anymore

[Firedancer885]

Jimm,

when Rinn first posted the problem, I downloaded a locked story:
http://archiveofourown.org/works/3224735
from the page (s)he provided. Even that single story showed the problem. I haven't tried batch downloading.

[Ghostdraconi]

Originally Posted by JimmXinu

Version 2.2.11 - 31 Jul 2015

New sites forums.spacebattles.com and forums.sufficientvelocity.com[INDENT]
...both base_xenforoforum adapters sharing almost all their code.

Forums are not really designed to host story, but people use them that way anyway. I recently started reading a fandom (the Worm web serial by Wildbow) and the largest collection of stories I found for it was on a couple of near identical forum sites.

SpaceBattles Creative Writing

SufficientVelocity User Fiction

Hi, would it be possible to add an adapter for Questionable Questing, it uses the same Xenforo software.
Thanks for all your work on this plugin.