[GUI Plugin] FanFicFare

[chrnno]

hpfanficarchive.com is gone as of two weeks or so ago.

Web archive has most stories from there, including many that were long since deleted, AO3 is supposed to have backed things up too but something apparently went wrong.

[JimmXinu]

Was there an official announcement that it was going away? Or it's just gone?

The web server isn't configured, but there is DNS and a server there.

I generally don't remove broken sites that still have DNS because of how many times I've seen broken sites come back.

That said, my notes show that hpfanficarchive.com has been broken since ~Sep 2020.

[chrnno]

Originally Posted by JimmXinu

Was there an official announcement that it was going away? Or it's just gone?

The web server isn't configured, but there is DNS and a server there.

I generally don't remove broken sites that still have DNS because of how many times I've seen broken sites come back.

That said, my notes show that hpfanficarchive.com has been broken since ~Sep 2020.

Yes, there was announcements.

Please note that the custodian of HPFanFicArchive.com will be working with the OTW Open Doors committee to transfer the archive's fanworks to the Archive of Our Own (AO3).

The Harry Potter FanFic Archive will be closed to new and updated works from 12 December 2021, and the site will be permanently closed down on 31 December 2021.

For more information, please see the announcement post on AO3: NSFW link

Originally Posted by In the NSFW link

Welcome to the Harry Potter FanFic Archive! The creator and owner of hpfanficarchive.com, Chad (CazBandit), died suddenly in March of 2020, but hpfanficarchive.com will continue through the Archive of Our Own (AO3).

The Harry Potter FanFic Archive is closed to new works.

The import of this archive to the AO3 was approved by the OTW Open Doors Committee in June 2021, and began in [MONTH DATE].

(Closed, Unmoderated)

I suppose since something went wrong with AO3's transfer it is possible it does come back up for a while.

As for being broken since Sep 2020 I think it was down for a while looking back through web archive and whoever brought it up here didn't mention it after? Was definitely still working as of early this month though.

[JimmXinu]

Okay, removed in the next test version. Thanks for the info.

[estherflails]

Thanks so much for the solution! 😊
It works, but I'm having some trouble with the metadata.

The main issue is that the fandom and the pairing are both in the same category on the site. Like so: Fanfiction > Könyv > Harry Potter > Nagyterem > Draco/Harry
From this list I only really need the fandom and pairing, and I tried to separate them into different columns, but it didn't quite work as I wanted. The best I could do is have "Draco/Harry" in the pairing column and "Harry Potter, Draco/Harry" in fandoms.

Here are the relevant parts from my personal.ini:

Spoiler:

[defaults]
replace_metadata:
 category=>\>=>,

[merengo.hu]
add_to_include_in_ships: category

add_to_replace_metadata:
 ships=>(.*)Draco/Harry=>Draco/Harry
 category=>(.*)Harry Potter=>Harry Potter
 category=>Nagyterem=>

Trying to exclude the "Draco/Harry" tag from category just makes both columns "Harry Potter". Is there a solution or elegant workaround for this? The only way I can think of is to mess with the adapter file, which isn't a problem for me tbh if there is no solution, but I figured I'd ask. (I'd rather not set up a different column for category and pull metadata from that if I can avoid it, I feel like it would get messy)


Also, is there a way to add something to an empty column through replace_metadata or some other method, or is extratags the only one that can do that? I tried language=>(.*)=>Hungarian and some variations, but either I didn't find the right one or, well, it's not possible. 😅 In which case, I can do it manually, but I wanted to ask to be sure. Would probably be useful in the future, I mess around in personal.ini quite a lot lol.


Last thing, I swear 😅
In the base_eficition_adapter, would you mind changing this line:

        elif key == 'Completed':
            if 'Yes' in value or 'Completed' in value or 'Ja' in value:

to this?

        elif key == 'Completed':
            if 'Yes' in value or 'Completed' in value or 'Ja' in value or 'Igen' in value:

I tried status=>Igen=>Yes in replace_metadata, but unfortunately that didn't work.

Sorry for the trouble and thank you! 😊

[JimmXinu]

estherflails,

Please always include the story URL that exhibits whatever you're asking about. Since you didn't give one, I'm using https://merengo.hu/viewstory.php?sid=147405 (I hope it's SFW, I can't tell--Hungarian.)

The category as given by merengo.hu has a lot of redundancy.

Here's what I've come up with that I think does basically what you're asking:

Spoiler:

[merengo.hu]
## Base eFiction adapter is really pretty smart--it noticed and
## collected these metadata entries for itself.  But you need to
## declare them here to use them.  Add to titlepage_entries or columns
## as you like.  Or include them in existing metadata entries.
add_to_extra_valid_entries:,jellemzők,figyelmeztetések

## Copy category to ships, but without running category's
## replace_metadata on it.
add_to_include_in_ships:,category.NOREPL

## Only keep category,ships values containing Fanfiction.  May want to
## add Original, Crossover, etc,.  I was mostly discarding 'Kihívások'
## (challenges?)
add_to_include_metadata_pre:
 category,ships=~Fanfiction

add_to_replace_metadata:
## If you want more than Fanfiction, or 'Books' and Film, this
## will need more work.
 category=>^Fanfiction > (Könyv|Film) > ([^>]+?)( >|$).*$=>\2
## This removes everything except the final A/B part.
 ships=>^.*> ([^/]+/.*)$=>\1

## Discard anything in ships that made it through that doesn't have a
## '/' in it.  Could be an issue if site uses '&' for non-romantic
## 'ships'.
add_to_exclude_metadata_post:
 ships=~\>

It likely isn't perfect, but hopefully it will point you in the right directions.

[riothamus]

I searched the thread a bit and also looked at the github site. I've succesfully been using the FanFictionDownloader browser cache with FFF to update stories or download new stories. However, is there a solution for the "Get Story URLs from Webpage" option without turning off the broswer cache? What are FFF users utilizing to get all of an author's stories in one go or are they copy/pasting each story link for FFDL?

[JimmXinu]

riothamus,

I assume you're asking for fanfiction.net.

FFF has the ability (in windows, at least; I don't test mac & linux very often) to drag and drop selected text from a browser and parse out all the story URLs.

So if you are on an author's page, select all (usually with ctrl-a), the drag the selected text onto the FFF button in Calibre, FFF should give you the list of story URLs. You can copy/paste that to a file and use FFDL's "Read links from file" feature.

That should work, but it's not something I do regularly, so YMMV.

[riothamus]

Originally Posted by JimmXinu

riothamus,

I assume you're asking for fanfiction.net.

FFF has the ability (in windows, at least; I don't test mac & linux very often) to drag and drop selected text from a browser and parse out all the story URLs.

So if you are on an author's page, select all (usually with ctrl-a), the drag the selected text onto the FFF button in Calibre, FFF should give you the list of story URLs. You can copy/paste that to a file and use FFDL's "Read links from file" feature.

That should work, but it's not something I do regularly, so YMMV.

Yes, I was asking about ff.net. Sorry about that! Wow, so many features in FFF that I didn't know about! I'll give that a shot the next time I download a new author.

Thanks for all your hard work that keeps us downloading fics!

Riothamus

[ownedbycats]

Question: For date-columns, is there an option for "last downloaded"? Last updated is the fic date, last checked also counts for skipped updates.

[JimmXinu]

dateCreated which by default is labeled "Packaged" on title page and 'Created' in the custom columns tab.

[icecold]

Would it be possible(/Not too much effort) for "Get Story URLs from Web Page" to accept multiple urls at the same time, like "Download from URLs"? I'm mostly interested in this so I can download all of an ao3 author's fanfics and all my bookmarks at once.

[ownedbycats]

Originally Posted by JimmXinu

dateCreated which by default is labeled "Packaged" on title page and 'Created' in the custom columns tab.

Thanks. I was a bit mislead and thought it was the date the book entry was created in Calibre.

[JimmXinu]

Originally Posted by icecold

Would it be possible(/Not too much effort) for "Get Story URLs from Web Page" to accept multiple urls at the same time, like "Download from URLs"? I'm mostly interested in this so I can download all of an ao3 author's fanfics and all my bookmarks at once.

I'm unlikely to add that.

First, "Get Story URLs from Web Page" runs in the foreground, blocking Calibre from doing anything else. I'm not interested in re-architecting it to do otherwise.

Second, we're always walking a fine line between making it easy for users and not generating too much load for servers.

I feel like 'get all the story URLs from this page' is acceptable, but 'get all the story URLs from this arbitrarily long list of pages'--with the stated intention of downloading all of them--is going too far.

I still feel bad about the guy who used the CLI version to try and download all of ffnet. I wonder if that contributed to where we are now. And how many others did that that I never heard about.

[ownedbycats]

Originally Posted by JimmXinu

I still feel bad about the guy who used the CLI version to try and download all of ffnet. I wonder if that contributed to where we are now. And how many others did that that I never heard about.

Now I am curious as to whether the bot attacks FictionPress mentioned on their twitter was actually people scraping the entire site.