[GUI Plugin] FanFicFare

[jesscat]

I've just started using this plugin, and I usually try to read the entire thread before posting a question, but this one is so long that I just don't have the hours to do it, so I'm going to go ahead and hope I'm not being too repetitive. I'd like to find a way to download from AO3 all stories with a particular tag. When I do the search on AO3, there are about 50 pages of stories with this tag (and I can't find any way on AO3 to show more than 20 search results on a page).

Do I have to do "Get Story URLs from Web Page" for each of those 50 pages individually, or is there a quicker/more efficient way that I'm missing?

What if I wanted to download all the works for a particular fandom? (I think this is basically the same question.)

Thank you!!

[JimmXinu]

Originally Posted by jesscat

...
Do I have to do "Get Story URLs from Web Page" for each of those 50 pages individually, or is there a quicker/more efficient way that I'm missing?

Short answer is yes, that's the way it works.

FFF is a tool for downloading stories, but I deliberately don't make it super easy to download hundreds or thousands stories at a time. The reason is that the story sites tend to notice large amounts of traffic like that and then work to block that user and/or the tool they are using.

I don't really care if you personally get blocked (and it does happen sometimes--especially with ffnet), but I don't want you getting me blocked.

Plus I didn't want the bother of implementing getting multiple pages of lists on multiple sites. The "Get Story URLs from Web Page" feature is kind of a compromise. It's relatively easy to grab all valid story URLs from any individual web page, and it indirectly limits mass downloads.

[jesscat]

Okay thanks! At least it's not something I was missing!

Another question that I'm sure is answered somewhere but I haven't been able to easily find it - how can I edit my personal.ini so that if a certain tag is on a story, a specific tag is added when downloading? For example, let's say AO3's tag is "Harry Potter" (or whatever) - I don't want to *replace* that tag completely (I've discovered how to do that), instead I want to keep it and *add* the tag "HP."

I feel like there must be a way to do this.

[JimmXinu]

Originally Posted by jesscat

Another question that I'm sure is answered somewhere but I haven't been able to easily find it - how can I edit my personal.ini so that if a certain tag is on a story, a specific tag is added when downloading? For example, let's say AO3's tag is "Harry Potter" (or whatever) - I don't want to *replace* that tag completely (I've discovered how to do that), instead I want to keep it and *add* the tag "HP."

The most straightforward way is to replace it--with itself and your new tag. Using \, splits replacement string into separate list items.

# Harry Potter only, verbose, limited to those metadata entries
replace_metadata:
 category,fandoms,freeformtags,ao3categories=>^Harry Potter$=>Harry Potter\,HP

# Any tag containing harry potter or Harry potter, etc.
replace_metadata:
 category,fandoms,freeformtags,ao3categories=>^(.*[Hh]arry [Pp]otter.*)$=>\1\,HP

To get the most out of replace_metadata you'll need to learn at least a bit about regular expressions.

You'll also eventually have to accept that there's no way to completely normalize AO3's tags so long as they allow all authors to enter whatever they want. It's just a losing proposition. I've posted about it here several times before.

[jesscat]

Thanks! I do understand regular expressions but haven’t used them in a long long time so will need to pull out my ancient handy reference book. It’s useful to know that’s what FFF uses - but I have no idea how I’m supposed to figure out all the things I can change with my personal.ini. The documentation I’ve found is not very helpful or comprehensive. I’ll keep searching.

I know AO3 doesn’t normalize tags in general, but at least it does use standard ones for fandom, which all authors use or else their stories really can’t be searched at all.

I am going to start reading through this thread!

[ilovejedd]

Originally Posted by jesscat

I know AO3 doesn’t normalize tags in general, but at least it does use standard ones for fandom, which all authors use or else their stories really can’t be searched at all.

Nope, not even fandom tags are normalized. They do have tag wranglers who link tags together. Thankfully though, most authors appear to use the normalized values for fandom at least

[theArchitect21]

Posting here since it looks like the google group is kinda dead.


In some sites like AO3 each chapter has a summary at the top and author notes at the bottom. While this is nice on the website it becomes cumbersome in an ebook, especially if reading on your phone. Is there a way to have FanFicFare download the fanfic and omit the summaries & notes? I noticed that, on AO3, the summary and author note have special DIVs so it seems like it should be possible to have FanFicFare just skip those divs. Any advice would be great!

[PandathePanda]

Originally Posted by theArchitect21

Posting here since it looks like the google group is kinda dead.


In some sites like AO3 each chapter has a summary at the top and author notes at the bottom. While this is nice on the website it becomes cumbersome in an ebook, especially if reading on your phone. Is there a way to have FanFicFare download the fanfic and omit the summaries & notes? I noticed that, on AO3, the summary and author note have special DIVs so it seems like it should be possible to have FanFicFare just skip those divs. Any advice would be great!

Hi

try what is written here: https://www.mobileread.com/forums/sh...postcount=3308

[PandathePanda]

Hi JimmXinu

Any Ideas with this? https://www.mobileread.com/forums/sh...8&postcount=44

[jesscat]

I feel like a total idiot, but despite wracking my brains, reading the documentation, and copying directly from other posts, I can't get what seems like this most basic metadata replacement in personal.ini to work. Can you possibly help me see what I'm doing wrong?

[archiveofourown.org] 
username: XXX 
password: XXX
replace_metadata:
      All For The Game - Nora Sakavic => All For The Game
      publisher => archiveofourown.org => AO3

Neither of those replacements is getting made in any field.

[JimmXinu]

Originally Posted by jesscat

I feel like a total idiot, but despite wracking my brains, reading the documentation, and copying directly from other posts, I can't get what seems like this most basic metadata replacement in personal.ini to work. Can you possibly help me see what I'm doing wrong?

...

Neither of those replacements is getting made in any field.

Without knowing the story URL to test it (hint-hint: it helps me a lot if you include a story URL ) all I can point to is that whitespace matters. Use one leading space to indicate continuation lines and don't put extra space around => separators.

[archiveofourown.org] 
username:XXX 
password:XXX
replace_metadata:
 All For The Game - Nora Sakavic=>All For The Game
 publisher=>archiveofourown.org=>AO3

[jesscat]

Sorry, I didn't think of it! Here's a URL but I was having the problem with any story from this fandom & archive. I've made your suggested change, and it solves the first replace problem, thank you!, but I'm still not able to replace "archiveofourown.org" with AO3 - though I did also try escaping the "." since I had forgotten to do that, but it made no difference.

https://archiveofourown.org/works/14...pters/32639742

[jesscat]

I also have another question. Is there a way to conditionally populate a custom yes/no checkbox column? What I'm trying to do is to mark the stories that I have bookmarked on AO3. As I understand it, all of these will have a flag in the bookmarkprivate field - as I don't have any private bookmarks, the flag should always be no, though I'm not sure if it matters as long as there is a flag there. So what I need is a "yes" check in my custom "bookmarked" column if the bookmarkprivate field is has a flag.

Can this be accomplished with replace_metadata or some other way? Since I can't even get a basic replace_metadata to work, this one is definitely beyond me right now!

[jesscat]

I figured my archiveofourown.org replace_metadata problem out - I should have been using site rather than publisher!

[JimmXinu]

Originally Posted by jesscat

I'm still not able to replace "archiveofourown.org" with AO3 - though I did also try escaping the "." since I had forgotten to do that, but it made no difference.

FanFicFare has a metadata entry site that is put into Calibre's Publisher field. Do replace_metadata on site instead of publisher. (I had forgotten they were different names myself.)

Originally Posted by jesscat

I also have another question. Is there a way to conditionally populate a custom yes/no checkbox column? What I'm trying to do is to mark the stories that I have bookmarked on AO3.

Boolean custom columns can be set from custom_columns_settings in personal.ini. Any of yes/y/true/t/1 for Yes vs any other value for No.

(Unless you have three state booleans enabled in Calibre Behavior settings, in which case no/n/false/f/0 for No and any other value will cause it to be set to 'Clear'.)

bookmarkprivate is only collected if you have always_login:true. But it also defaults to True/False, so you shouldn't even need replace_metadata for it.

Spoiler:

I was about to recommend using the 'Saved Metadata Column' and 'Update Calibre Metadata from Saved Metadata Column' feature to avoid hitting story sites over and over again while experimenting with metadata, but it looks like bookmarkprivate doesn't work correctly with that.

Oh look, python thinks boolean is also an int... I have a fix for that, but I want to look at it a bit more before putting it out there. Tomorrow probably.