[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by readerboy7

I'm not the person who asked this, but would that actually be that big of a deal? Assuming the stories are on wayback, would it be possible to use an old version of FFF (or even FFDL for the older stories) to download them? This also applies for FFNet fics, now that wayback is disobeying their robots.txt

You're welcome to try.

Making it work for a few specific stories would be a doable project if someone really wanted to do it. Making it work as a general solution would be a remarkable PITA.

Either way, it isn't really something I'm interested in doing. And I would need to be convinced to add it to the project even if somebody else did.

[Atherton25]

Originally Posted by gcomyn

I'll look at it tonight, after I finish working.

GComyn

Any news about webnovel.com? I noticed that github had a comment about accessing after the update.

[gcomyn]

Originally Posted by Atherton25

Any news about webnovel.com? I noticed that github had a comment about accessing after the update.

I've been working on it, but I am having trouble with the cookies... I've never been able to figure them out...

I'll keep working on it.

GComyn

[ILB]

It seems fanfiktion.de changed also to https

I´ve got a downloaded story ( http://www.fanfiktion.de/s/505f126500016f9206d00fa2/1 ) but every new story I´d like to download has now https ( https://www.fanfiktion.de/s/505f1265...ag-auf-Schalke )
and I have to manually delete the "s" to download.

[JimmXinu]

Originally Posted by ILB

It seems fanfiktion.de changed also to https

Change is committed--it will be in the next test version.

[bugstomper]

There's a story that I would like to convert to epub that is posted to tumblr http://docfuture.tumblr.com/post/111...-call-contents which I understand fanficfare can't support as there is no standard tumblr format for books.

I could easily write a script to scrape this book and put each chapter in a file. I could probably go on to work the book into a reasonable html book format and figure out the settings for Calibre to convert that to a reasonable epub, but would it be possible for fanficfare to have an adapter for some specified format on local disk to facilitate converting books like this that are not on a site that fanficfare directly supports but have a regular enough format that it can be scraped in a more manual process?

Or is there already an adapter for locally stored books in some defined format?

[JimmXinu]

Originally Posted by bugstomper

...
Or is there already an adapter for locally stored books in some defined format?

I had to stop and think about it for a while, but in point of fact, there is!

FanFicFare has a fake site, test1.com, that I use to generate stories with different attributes without needing to hit anyone's site.

The test1.com adapter has a teststory feature where you can construct whatever metadata you like with storyId >= 1000.

And at one point, for my own purposes, I extended that feature to also have a 'chapter_urls' setting that will pull a page from a URL for each chapter (which can be a file:// URL) and look for a <blockquote> tag inside it. The blockquote will be changed to a div (because reasons), and it and everything inside it will used for the chapter text.

personal.ini section:

Spoiler:

# make a new section and change 1000 to other 1000+ numbers for other stories.
[teststory:1000]
title:Your Story Name
author_list:Author Name
authorUrl_list:https://author.url
datePublished:2015-07-26
dateUpdated:2015-10-02
category_list:Harry Potter,Superman
genre_list:Action/Adventure,Drama
# leave numWords empty and, depending on other setting, FanFicFare
# will use Calibre's code to count words.
numWords:
description:<p>A HP x Superman story with a description!</p>
 <p>'Make using' line below isn't required, but I find it handy.</p>
 <p>Made using [teststory:{{storyId}}] in personal.ini.</p>
status:Completed

# file URL,chapter name
chapter_urls:
 file:///C:/Users/user/Desktop/storysrc/chap01.html,Chapter 1
 file:///C:/Users/user/Desktop/storysrc/chap02.html,Chapter 2

Example chap01.html (Use UTF8 if encoding needed.):

Spoiler:

<html>
  <body>
    <blockquote>
<div>
<h3 extra="value">Chapter title from site chapter 1</h3>
br breaks<br><br>
Puella Magi Madoka Magica/魔法少女まどか★マギカ
<p>
  Chapter 2 text...
</p>
br breaks<br><br>
Don't—e;ver—d;o—that—a;gain, 法 &#xE9;
<hr>
horizontal rules
<hr size=1 noshade>
<p>"Lorem ipsum dolor sit amet", consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore--et dolore magna aliqua. 'Ut enim ad minim veniam', quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
</div>
    </blockquote>
  </body>
</html>

Then, after setting personal.ini and creating the chapter files, give FanFicFare the story URL: http://test1.com?sid=1000

Warning:

This is an existing undocumented feature. I have no issues with someone who can write their own page scrape scripts using it--something bugstomper specifically mentioned being able to do.

But I'm not going to be interested in expending much energy on either enhancing or supporting it. It's a dev tool for my use.

[Loba]

Originally Posted by JimmXinu

You're welcome to try.

Making it work for a few specific stories would be a doable project if someone really wanted to do it. Making it work as a general solution would be a remarkable PITA.

Either way, it isn't really something I'm interested in doing. And I would need to be convinced to add it to the project even if somebody else did.

I'll look into a work around, maybe I can Frankenstein something together with older versions. If I come up with a workable solution (at least partially) I'll post it here just for others to search up in the future.

[Tharos]

Processing Warlock of the Magus World (巫界术士) by OMA (epub) Source
Processing Skyfire Avenue (天火大道) by Xiao Lai (epub) Source

fanficfare.appspot.com has held this status for over 12 hours.
Caliber does not create any file at all.

[JimmXinu]

Originally Posted by Tharos

fanficfare.appspot.com has held this status for over 12 hours.
Caliber does not create any file at all.

Both of those stories have problems on the site. wuxiaworld.com/sfl-index/ has a bad URL for chapter 483 and wuxiaworld.com/wmw-index/ has the same chapter URL for every chapter from 649 on.

So those two aren't going to download correctly regardless.

More generally:

It's not uncommon for the web service to not work on stories with hundreds of chapters (those two have 700+ & 500+). There are limits and timeouts that apply. See FAQ.

"Caliber does not create any file at all" is not a terribly useful problem report. In future, include any error messages and a description of what specifically happened. Ideally, you would also run calibre in debug mode (under calibre Preferences dropdown) to see any errors.

With several hundred chapters, a download will often run for a really long time before finishing. Especially if you have a slow_down_sleep_time set.

It's also possible that the site has started slowing or even blocking your requests if you've been downloading many and/or large stories. In that case, a slow_down_sleep_time setting will often help by spreading your requests out more over time. I don't know enough about wuxiaworld.com to know for sure.

[AndersW]

I am trying to get prequelHTML to be added to a custom column, but I can't get it to work.

[JimmXinu]

Originally Posted by AndersW

I am trying to get prequelHTML to be added to a custom column, but I can't get it to work.

Can you be more specific? It works fine for me as long as it's a Long Text custom column.

personal.ini:

[www.fimfiction.net]
add_to_custom_columns_settings:
 prequelHTML=>#long_text

[bugstomper]

Originally Posted by JimmXinu

I had to stop and think about it for a while, but in point of fact, there is!

That is so cool! Even cooler, as I looked at the adapters I could see that it would be almost as easy to write my own adapter for the site based on the ones you have. The only question I am left with about it is that every adapter I have looked at assumes that the first URL it looks at contains somewhere a list of links to every chapter in the story. The story I am looking at has no such index page, only prev and next links on each chapter. Is there any example of an adapter that gets the chapter links as it iterates through the chapters or is it a requirement that a site has something that lists all chapters in one place?

I guess I can write something that generates all the chapter links and use it with your test site adapter or an adapter that I write. Or even override extractChapterAndMetadata to read all the chapters and then read all of them again in each call to getChapterText. Is that what would be necessary? Are there any adapters that do something similar whose code I can look at?

[JimmXinu]

Generally speaking, if a site doesn't have chapter lists available, I declare it unsupportable for being too far outside the accepted paradigm.

I vaguely recall somebody writing an adapter for a site like that, but I don't remember if I allowed it to be included, let alone what site it was...

As for why, it's about updates. I assume that most users, like me, follow hundreds of in-progress stories. IMHO, updating existing ebooks with new chapters without downloading all the old ones again is just as important as downloading whole stories.

[AndersW]

Originally Posted by JimmXinu

Can you be more specific? It works fine for me as long as it's a Long Text custom column.

personal.ini:

[www.fimfiction.net]
add_to_custom_columns_settings:
 prequelHTML=>#long_text

In configure FFF under custom columns. When I select the column I want prequelHTML to go to, prequelHTML isn't an option that I can choose.