[GUI Plugin] FanFicFare

[Vego22]

Originally Posted by darth1

Last time round FFnet fixed itself in a week, can there be a wait period?

Ever since ff.net started having issues, I started using cache download feature and it works without problem... even now. Just use that because i dont think cloud is ever going to go away. Ending ff.net support would literally kill me since i download lots and lots of those >_> ff.net needs to chill a bit.

[JimmXinu]

Originally Posted by junni

Since today, i can't update metadata from website novelfull.com, because it gives me a 403 error. Anyone else with the same situation? Could it be that the site implemented some kind of new protocol?

I'm not seeing any issues with that site, but I don't read it regularly.

When reporting issues, please include a story URL that demos the problem, even if you think it's universal.

"403 Forbidden" may indicate that you are being specifically blocked, which some site do when someone downloads 'too much'.

[junni]

Originally Posted by JimmXinu

I'm not seeing any issues with that site, but I don't read it regularly.

When reporting issues, please include a story URL that demos the problem, even if you think it's universal.

"403 Forbidden" may indicate that you are being specifically blocked, which some site do when someone downloads 'too much'.

Hi JimmXinu

Thanks for the fast reply. I tried at a different place (my workplace), and i still gives me the same error:

HTTP Error in FFF '403 Client Error: Forbidden for url: https://novelfull.com/the-nine-cauldrons.html'(403)
https://novelfull.com/the-nine-cauldrons.html

This is the error for trying to update the "metadata only", for the epub. I'm not sure if this is all the information you need, but this is all i get when i try to see why the update failed.

[JimmXinu]

I am now getting 403 Forbidden for all attempts on novelfull.com also.

novelfull.com, too, is using cloudflare.

Setting a different user_agent or use_cloudscraper:true don't seem to help.

Running a FlareSolverr proxy and setting use_flaresolverr_proxy:true does seem to work.

[code]
[novelfull.com]
use_flaresolverr_proxy:true
[\code]

Note that only FlareSolverr v1.2.9 is known to work with FFF if you choose to try it.

Note also that I'm not going to offer a lot of help with getting FlareSolverr running--I consider that outside my scope.

[junni]

Originally Posted by JimmXinu

I am now getting 403 Forbidden for all attempts on novelfull.com also.

novelfull.com, too, is using cloudflare.

Setting a different user_agent or use_cloudscraper:true don't seem to help.

Running a FlareSolverr proxy and setting use_flaresolverr_proxy:true does seem to work.

[code]
[novelfull.com]
use_flaresolverr_proxy:true
[\code]

Note that only FlareSolverr v1.2.9 is known to work with FFF if you choose to try it.

Note also that I'm not going to offer a lot of help with getting FlareSolverr running--I consider that outside my scope.

Hi JimmXinu, thanks for the tip!! I got it to work pretty simply . Hopefully this solution can last for a while . Cheers.

By the way, any possibility to add new supported websites to FFF? Readnovelfull.com is very similar to novelfull.com but has many more novels. Any guide which could help to DIY, would be great so we could add ourselves.

[ownedbycats]

Jim doesn't really add new sites anymore (unless they're exactly the same as an existing one, I think) - he's got his hands full just fixing the ones that already exist and keep breaking! You can post a request on github and hope a volunteer picks it up, or if you're good with code take a look at the adapters yourself and see if you can get it working.

[JimmXinu]

FAQ: Please add site XYZ to FanFicFare?

I rarely add new sites myself anymore for several reasons.

Most, if not all, of the new sites requested these days are new fiction or translated fiction instead of fanfiction. Newer, more interactive sites are frequently more difficult to implement. And mostly, I ran out of ambition for adding sites I don't read myself years ago.

These days new sites are mostly added by other people who want to read them. Which is kinda how it should be with an open source project, IMO.

[junni]

Originally Posted by ownedbycats

Jim doesn't really add new sites anymore (unless they're exactly the same as an existing one, I think) - he's got his hands full just fixing the ones that already exist and keep breaking! You can post a request on github and hope a volunteer picks it up, or if you're good with code take a look at the adapters yourself and see if you can get it working.

since it was very similar to novelfull.com, i got a py script that gets the metadata working. Unfortunately i can't scrap all the chapters nor get the correct chapter count, since the website requires that you click on chapter list, which turns the url like this : https://readnovelfull.com/the-nine-c...chapters-title , which gives me all sort of errors on flare solver because of that last part. I guess i will put a request on github.

Thanks for all the help guys. Happy reading!

[ackomb]

Hi,

I just got this error when downloading:
HTTP Error in FFF '500 Server Error: Internal Server Error for url: http://localhost:8191/v1'(500)

Does it have to do with ff-net being on the fritz?
I read it as something local, but beyond that... so just wondering?
I've been able to update about 15 stories last night, another 5 failed with this specific error, en one more with a very large error code which I thought I'd copied but I didn't . I'll post it if I come across it again.

[ownedbycats]

Yes. If you open Flaresolverr's console window you'll probably see it timing out when trying to connect to ffnet. Use the cache for now.

[MonicaOP]

Hi, well, I was wondering, how can you add sites to the plug in? Is there a way to do it? Taking into account that I'm completely clueless to everything that is searchable in google or something.

Thank you, be well, Monica

[fanfictioneer]

Originally Posted by MonicaOP

Hi, well, I was wondering, how can you add sites to the plug in? Is there a way to do it? Taking into account that I'm completely clueless to everything that is searchable in google or something.

Thank you, be well, Monica

First of all, you need to be able to program in python - ownedbycats posted the link directly to the place where you can check how the adapters are coded (basically, each site has an "adapter" that makes FFF able to download from it).

If you don't know how to program in python, you can always create an issue in the github for FFF, specifying which site it is that you want added. Note that generally, people mostly create adapters for sites that they are interested in reading themselves, so your issue might languish, unfixed, for quite some time. Your best bet to get a site added is if it is very similar to an existing supporting site, such as if it is a xenforo-forum site.

If you want to learn how to do it yourself, your best bet (IMHO) is studying one of the existing (preferably simpler) adapters and making sure you understand how that one functions, then trying to write your own adapter for the site you are interested in. Once you've got it working for the stories you are interested in, try it out for a few other stories on that site, too, then create a pull request for JimmXinu to add your adapter to FFF.

Good luck!

[JimmXinu]

Originally Posted by fanfictioneer

First of all, ...

Well said, thanks.

[ackomb]

Originally Posted by ownedbycats

Yes. If you open Flaresolverr's console window you'll probably see it timing out when trying to connect to ffnet. Use the cache for now.

Thx, I'll just do that.

[junni]

Hi @JimmXinu


I've been fiddling around in python and got a rough understanding of it, so i tried to make an adapter for the readnovelfull.com website, since it is similar to novelfull.com. I got it working, but i am sure the code could be way pretty. Anyways, if anyone is interested on it, here it goes.

P.S. i don't know how to use github, so i'm sorry if i can't post it there, or if i'm permitted to post here.

from __future__ import absolute_import
import logging
import re

# py2 vs py3 transition
from ..six.moves.urllib import parse as urlparse

from .base_adapter import BaseSiteAdapter


logger = logging.getLogger(__name__)


def getClass():
    return ReadNovelFullSiteAdapter


class ReadNovelFullSiteAdapter(BaseSiteAdapter):
    @staticmethod
    def getSiteDomain():
        return "readnovelfull.com"

    def getSiteURLPattern(self):
        return r"https?://%s/(index\.php/)?(?P<story_id>.+?)(/.*)?\.html?" % re.escape(self.getSiteDomain())

    def __init__(self, configuration, url):
        super(ReadNovelFullSiteAdapter, self).__init__(configuration, url)

        story_id = re.match(self.getSiteURLPattern(), url).group('story_id')
        self.story.setMetadata('storyId', story_id)

        self._setURL("https://%s/%s.html" % (self.getSiteDomain(), story_id))

        self.story.setMetadata('siteabbrev', 'rnvlfl')

    def extractChapterUrlsAndMetadata(self):
        data = self.get_request(self.url)

        soup = self.make_soup(data)

        self.story.setMetadata("title", soup.select_one("h3.title").text)

        for author in soup.find("h3", text="Author:").fetchNextSiblings(
            "a", href=re.compile("/author/")
        ):
            self.story.addToList("authorId", author.text)
            self.story.addToList(
                "authorUrl", urlparse.urljoin(self.url, author.attrs["href"])
            )
            self.story.addToList("author", author.text)

        status = soup.find("a", href=re.compile("/completed-novel"), class_="text-primary").text

        if status == "Completed":
            self.story.setMetadata("status", "Completed")
        else:
            self.story.setMetadata("status", status)

        cover_url = soup.find("div", class_="book").find("img").attrs["src"]
        self.setCoverImage(self.url, urlparse.urljoin(self.url, cover_url))

        self._crawl_chapters(self.url)

        self.setDescription(self.url, soup.select_one("div.desc-text"))

        for genre in soup.find(class_="info info-meta").find_all("a", href=re.compile("/genre/")):
            self.story.addToList("genre", genre.text)

    def _crawl_chapters(self, url):
        data1 = self.get_request(self.url)
        soup1 = self.make_soup(data1)	
        novel_id = soup1.select_one('div#rating')['data-novel-id']
	
        full_chapter_url = 'https://readnovelfull.com/ajax/chapter-archive?novelId=%s'

        data = self.get_request(full_chapter_url % novel_id)
        soup = self.make_soup(data)

        for a in soup.select("ul.list-chapter a"):
            self.add_chapter(a.attrs["title"], urlparse.urljoin(url, a.attrs["href"]))

        next_page = soup.select_one("#list-chapter .next a")

        if next_page:
            self._crawl_chapters(urlparse.urljoin(url, next_page.attrs["href"]))

    def getChapterText(self, url):
        data = self.get_request(url)
        soup = self.make_soup(data)

        content = soup.find(id="chr-content")

        # Remove chapter header if present
        chapter_header = content.find(["p", "h3"], text=re.compile(r"Chapter \d+:"))

        if chapter_header:
            chapter_header.decompose()

        # Remove generic end-text added to all books

        for extra in content.find_all(attrs={"align": "left"}):
            extra.decompose()

        return self.utf8FromSoup(url, content)