[GUI Plugin] FanFicFare - Page 73

JimmXinu · 02-20-2016, 11:11 PM

Quote:

Originally Posted by edeniz

...
Ok, that's how I understood it, especially since I've had to add characters lines as well if I wanted names in ships to be changed. But if that is right, basically lines like this:

Spoiler:

should pose no problem, shouldn't it? Because, again using this story as an example case, using a line like the above means it will make the replacement, but the ships will not be split into characters. My characters field remains empty.

No ships lines, including those, effect the ships2chars lines when ships2chars is defined by include_in_ships2chars:ships.NOREPL.

I've just tested (again) with that story and it gets the two characters just fine.

The entirety of the personal.ini I used is:

Spoiler:

So you're still having a problem, it's something else.

Zero Rei Maru · 02-21-2016, 03:07 AM

Aloha. Having some trouble downloading from Spacebattles forums threads. Originally, I thought the problem was trying to download forum stories without threadmarks. However, even with Threadmarks, nothing downloads. I've tried from a random page with the story on it, i have tried using the url of the very first page of the thread, and I've even opened up the threadmarks page in a new tab, and tried that link. I've put them all both into "Download from URL" as well as "Get URLs from webpage" and nothings doing.

I know that it can be done, because I somehow have a copy of JonBerry's A New Jedi in an Old Republic on calibre from Spacebattles. I just can't figure out what I did to get it to work in the first place.

I am trying to download https://forums.spacebattles.com/thre...rt-nge.257658/

and when it doesn't work, this is the log it spits at me.

Status
Title
Author
Comment
URL
Error
Coupled Effort
Ranma-sensei
u'do_wordcount'
https://forums.spacebattles.com/threads/257658/

So, any ideas?

edeniz · 02-21-2016, 05:26 AM

Quote:

Originally Posted by JimmXinu

[...]So you're still having a problem, it's something else.

You were right, it was something else. I deleted everything line by line, and found the culprit. As suspected, it was a dumb mistake.

Having this in my defaults screwed it up:

Spoiler:

I liked your suggestion here, and so added those rawgenre lines to move that tag from genre to characters. But I had added them in the defaults, which turned out to be a mistake. Because the ao3 part of the ini already had this:

Spoiler:

and so while those lines did their intended job, there was apparently a conflict with ships2chars. I simply had not noticed. Now that I deleted them from the defaults and added to ao3 where they belongs, there is no problem.

for your indulgence.

edeniz · 02-21-2016, 05:38 AM

Quote:

Originally Posted by Zero Rei Maru

Aloha. Having some trouble downloading from Spacebattles forums threads. [...] I am trying to download https://forums.spacebattles.com/thre...rt-nge.257658/

[...]

Feedback, if it helps: I for one have had no problems whatsoever downloading that or any other spacebattles links, and I have only this in my ini in the xenforoforum part:

Spoiler:

davidfor · 02-21-2016, 07:34 AM

Quote:

Originally Posted by JimmXinu

An initial look does suggest that finestories.com is similar to storiesonline.net now, but I'll have to spend some time looking more closely before I know for sure.

Yes, they are identical. Attached are updates for the two adapters. Basically I just made the FineStories adapter a subclass of the StoriesOnline. And there are a couple of changes to the StoriesOnline adapter to make this easier.

The other thing needed is a change to the ini defaults. But, copying those from the StoriesOnline is all that is needed.

JimmXinu · 02-21-2016, 10:24 AM

Quote:

Originally Posted by Zero Rei Maru

...
I am trying to download https://forums.spacebattles.com/thre...rt-nge.257658/

and when it doesn't work, this is the log it spits at me.

u'do_wordcount'
https://forums.spacebattles.com/threads/257658/

do_wordcount is from a new feature to allow FFF to do word counts internally. Did you make sure to restart calibre after updating the plugin?

If it still happens, please run calibre in debug mode (under the calibre Preferences drop down), try the download again, quit calibre, then post the debug log here.

JimmXinu · 02-21-2016, 12:14 PM

Quote:

Originally Posted by edeniz

...
and so while those lines did their intended job, there was apparently a conflict with ships2chars. I simply had not noticed. Now that I deleted them from the defaults and added to ao3 where they belongs, there is no problem.
...

I can see how those lines caused the problem, but it sounds like your understanding of why is a bit off. So let me explain the weirdness a bit.

It has to do with how Python's INI parser, FFF's use of that parser and add_to_* work.

Excessively long explanation:

Spoiler:

Code:

# First, if the same keyword appears more than once in the same
# section, the last one 'wins'.  This is part of the python ini parser
# and we can't change it.

[defaults]

is_adult: true

is_adult: false

# is_adult == false

Code:

# Still true even if the section appears more than once.

[defaults]

is_adult: false

[defaults]

is_adult: true

# is_adult == true

Code:

# Second, FFF interprets sections in 'priority' order, from least
# priority to most they are:
#
#   [defaults]
#   [archiveofourown.org] (or other site)
#   [epub] (or [html], [mobi], [txt])
#   [archiveofourown.org:epub] (or html, mobi, txt)
#   [overrides]
#
# For all keywords, only the one from the highest priority section
# will be used.  This is a design feature of FFF.
#
# Here's an example that demonstrates the biggest limitation of that:

[defaults]

replace_metadata:
 genre=>Comedy=>Humor

# replace_metadata == genre=>Comedy=>Humor ...
 
[forums.spacebattles.com]

replace_metadata:
 title=>Wail of the Banshee 3: Boxhead Blues and the Zombocalypse=>Wail of the Banshee

# Now for forums.spacebattles.com:
# replace_metadata == title=>Wail...=>Wail of the Banshee
# *overriding* any and all values from [defaults].

Code:

# Now here's an example that shows how the 'add_to_*' mechanism
# addresses that problem.  Any keyword that starts with 'add_to_'
# *adds* to what was set in *lower* priority sections instead of
# replacing it.  That's why list type keyword values need to begin
# with ',' when used with add_to_.
#
# It's actually a bit more complex than that even, as at each
# priority level, there's the defaults.ini section *then* the
# personal.ini section.  So defaults.ini [epub] has higher priority
# than personal.ini [defaults].
#
# *And* as a convenience, both with and without www work: 
# [www.archiveofourown.org] and [archiveofourown.org] are both 
# valid with [archiveofourown.org] being higher priority.  
# Pick one and stick with it.

# From defaults.ini[defaults]:
# category_label:Category

[defaults]

add_to_category_label:(defaults)
# category_label == Category(defaults)

add_to_extra_valid_entries:,ships2chars
# extra_valid_entries == ships2chars

replace_metadata:
 genre=>Comedy=>Humor

[epub]

add_to_category_label:(epub)
# category_label == Category(defaults)(epub)

[forums.spacebattles.com]

add_to_category_label:(SB)
# category_label == Category(defaults)(epub)(SB)

[archiveofourown.org]

add_to_extra_valid_entries:,series00HTML,series01HTML,series02HTML,series03HTML
# extra_valid_entries == ships2chars,series00HTML,series01HTML,series02HTML,series03HTML

[forums.spacebattles.com]

replace_metadata:
 title=>Wail of the Banshee 3: Boxhead Blues and the Zombocalypse=>Wail of the Banshee

# Now for forums.spacebattles.com:
# replace_metadata == genre=>Comedy=>Humor 
#                     title=>Wail...=>Wail of the Banshee

Code:

# Now, I believe the problem you were having was actually an
# interaction between the first rule (imposed by ini parser) and
# add_to_.  Because it's add_to_, the intuitive expectation is that it
# will add, but it doesn't work if they're in the same section:

[defaults]

add_to_extra_valid_entries:,ships2chars

# overrides above.

add_to_extra_valid_entries:,rawgenre

# extra_valid_entries == rawgenre

Code:

[defaults]

add_to_extra_valid_entries:,ships2chars

[archiveofourown.org]

# works as expected because now it's in a different section.

add_to_extra_valid_entries:,rawgenre

# extra_valid_entries == ships2chars,rawgenre

That all makes sense, right?

JimmXinu · 02-21-2016, 12:25 PM

Quote:

Originally Posted by davidfor

Yes, they are identical. Attached are updates for the two adapters. Basically I just made the FineStories adapter a subclass of the StoriesOnline. And there are a couple of changes to the StoriesOnline adapter to make this easier.

The other thing needed is a change to the ini defaults. But, copying those from the StoriesOnline is all that is needed.

New test version posted (direct link)

2016-02-21
- Fixes for finestories.com, now shares code with storiesonline.net, thanks davidfor for this.

edeniz · 02-21-2016, 12:58 PM

Quote:

Originally Posted by JimmXinu

I can see how those lines caused the problem, but it sounds like your understanding of why is a bit off. So let me explain the weirdness a bit.

It has to do with how Python's INI parser, FFF's use of that parser and add_to_* work.

Excessively long explanation:
[....]
That all makes sense, right?

Yes, it makes sense.

It's awesome of you to take the time to explain these things and indulging me even when my questions are of a dumb nature, really appreciate it.

But for all that effort, I can honestly say I understand it better and so not likely to keep making the same mistakes and cluttering up the forum.

Zero Rei Maru · 02-21-2016, 05:23 PM

Quote:

Originally Posted by JimmXinu

do_wordcount is from a new feature to allow FFF to do word counts internally. Did you make sure to restart calibre after updating the plugin?

If it still happens, please run calibre in debug mode (under the calibre Preferences drop down), try the download again, quit calibre, then post the debug log here.

I could have SWORN I had restarted Calibre after updating the plug in.

And of course, I restarted Calibre, and now it downloaded. Thank you.

While I'm here, the plugin only works for stories with Threadmarks correct?

JimmXinu · 02-22-2016, 12:19 AM

Quote:

Originally Posted by Zero Rei Maru

While I'm here, the plugin only works for stories with Threadmarks correct?

It's a bit complicated because forums sites really aren't ideal for posting stories. All of this applies only to the SB, SV and QQ sites.

If you use a thread URL, FFF will use threadmarks if there are at least 2 (to allow for threads that only have a threadmark for Index).

If you use a thread URL and there are < 2 threadmarks, FFF looks for URLs in the first post and treat them as chapter URLs. Chapter links have to be on the same site, but can be in other threads. (There are a number of heuristics included to allow for some common errors in linking I've seen.)

If you use a post URL in either of the two form, FFF will download that post and treat it as an index page to get chapters from.

If the post URL given is not itself included in the chapters links, it will be included first and titled 'First Post'. This can be used to download individual posts containing no links, BUT:

In all cases, the story will take the author and title from the thread OP and title.

This still doesn't cover all cases that I've seen, but it covers most of them.

JimmXinu · 02-22-2016, 10:20 AM

Test version updated again:

2016-02-22
- Allow ini keyword universe_as_series in finestories.com as well as storiesonline.net.

ij26 · 02-22-2016, 07:11 PM

I just tried to download a bunch of stories from Fimfiction, and found that while some worked, others return 404 errors (although they seem to download fine in a Web browser).

Stories I was able to download:

Quote:

Stories I got 404 errors for:

Quote:

A recent site update might be related.

FaceDeer · 02-23-2016, 01:25 AM

Quote:

Originally Posted by ij26

I just tried to download a bunch of stories from Fimfiction, and found that while some worked, others return 404 errors (although they seem to download fine in a Web browser).
...
A recent site update might be related.

The update says that userscripts making use of AJAX calls will be broken, and comparing the two sets of stories suggests a cause for this: all of the ones that work have 8 or fewer groups, and those that don't have 9 or more groups. When there's 9 or more groups you need to click the "show more groups" button to see them all, which is an AJAX call. I bet that the adapter's failing when it follows that link to try getting the complete group list.

I'm currently completely wiped out from several days of not sleeping enough, I'll poke around with this tomorrow if I'm conscious and nobody's beaten me to it. Hopefully the adapter can be updated to the new AJAX calls easily, I use that groups metadata in my library's organization.

Edit: looks like an easy enough fix.

Learning to use Git while having had 3 hours of sleep in the past 48 is beyond me, if this fix is good I assume Jimm will take it.

You should be able to fix this locally by digging into your installed FanFicFare.zip, finding the file adapter_fimfictionnet.py, and replacing the line

Code:

groupResponse = self._fetchUrl("http://www.fimfiction.net/ajax/groups/story_groups_list.php?story=%s" % (self.story.getMetadata("storyId")))

with

Code:

groupResponse = self._fetchUrl("https://www.fimfiction.net/ajax/stories/%s/groups" % (self.story.getMetadata("storyId")))

(while maintaining proper indentation of course).

I just tried one of the 404 stories with that modification and it worked. I'll try the rest now, if I don't edit this further I guess assume that it worked or that I fell asleep.

JimmXinu · 02-23-2016, 10:49 AM

New test version posted (direct link).

2016-02-23
- Fix for fimfiction.net site update -- thanks Facedeer

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] KindleUnpack - The Plugin	DiapDealer	Plugins	523	07-15-2025 06:45 PM
[GUI Plugin] Open With	kiwidude	Plugins	404	02-21-2025 05:42 AM
[GUI Plugin] Marvin XD	Philantrop	Plugins	126	01-29-2017 12:48 PM
[GUI Plugin] KiNotes	-axel-	Plugins	0	07-14-2013 06:39 PM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

02-21-2016, 03:07 AM	#1082
Zero Rei Maru Junior Member Posts: 2 Karma: 10 Join Date: Jan 2012 Device: Kindle Fire	Aloha. Having some trouble downloading from Spacebattles forums threads. Originally, I thought the problem was trying to download forum stories without threadmarks. However, even with Threadmarks, nothing downloads. I've tried from a random page with the story on it, i have tried using the url of the very first page of the thread, and I've even opened up the threadmarks page in a new tab, and tried that link. I've put them all both into "Download from URL" as well as "Get URLs from webpage" and nothings doing. I know that it can be done, because I somehow have a copy of JonBerry's A New Jedi in an Old Republic on calibre from Spacebattles. I just can't figure out what I did to get it to work in the first place. I am trying to download https://forums.spacebattles.com/thre...rt-nge.257658/ and when it doesn't work, this is the log it spits at me. Status Title Author Comment URL Error Coupled Effort Ranma-sensei u'do_wordcount' https://forums.spacebattles.com/threads/257658/ So, any ideas?

02-22-2016, 10:20 AM	#1092
JimmXinu Plugin Developer Posts: 6,984 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	Test version updated again: 2016-02-22 - Allow ini keyword universe_as_series in finestories.com as well as storiesonline.net.

02-23-2016, 10:49 AM	#1095
JimmXinu Plugin Developer Posts: 6,984 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	New test version posted (direct link). 2016-02-23 - Fix for fimfiction.net site update -- thanks Facedeer