[GUI Plugin] FanFictionDownLoader

[SallyK]

While I'm bothering you, could I ask you about a completely different topic?

I want to get some metadata for the stories I download - title, author, series, but I don't want the tags, particularly not the over-abundance that you often get on AO3. What I'd really like to do is to automate what I do when I've downloaded the stories, which is to remove all the existing tags, and add one that says New.

I've poked around a bit in the settings and couldn't see any obvious way of doing it, am I missing something or is it something I can do through my personal.ini?

[Jester11]

I am having problems with the fanfiction downloader plugin for calibre. It gives me a "http error 403: forbidden" for all stories that come from hpfanficarchive. Two weeks ago I didn't have that problem with the stories so any help is appreciated.

[JimmXinu]

Originally Posted by Jester11

I am having problems with the fanfiction downloader plugin for calibre. It gives me a "http error 403: forbidden" for all stories that come from hpfanficarchive. Two weeks ago I didn't have that problem with the stories so any help is appreciated.

This is a known issue.

I'm waiting to hear back from the hpfanficarchive.com webmaster.

[seabream]

Originally Posted by SallyK

remove all the existing tags, and add one that says New.

is it something I can do through my personal.ini?

The short answer is that you can do it through personal.ini.

The middle length answer is that there's probably more than one way to do this, and what to do may vary by site. What I have below is just the first thing that occurred to me and may not be the simplest solution. [Edit: A much simpler solution is described in the post immediately following this by JimmXinu. If you haven't read the one here yet, I recommend skipping to his.]

You can use the replace metadata function to match any string in the category(ies) of tag that you don't want to receive, and replace it with an empty string, functionally deleting it (them, if there are multiple tags). There's more than one way to add "New", and which one is best depends on what you want to do with it. You can do it in personal.ini, (Add a line at the end of your list of metadata replacements that matches anything (including empty) in the category of metadata that you want "New" to appear in and replaces it with "New") but if you are using it to manage a reading list, then it would probably be better to use the Reading List plugin together with FFDL, and Calbre's custom columns. You can have Reading List add a tag to a given column, (e.g.: a Read/Unread column, which you'd define in Calibre's custom columns settings) and have FFDL automatically add new/updated stories to the relevant list (set in the Reading List tab of the FFDL settings). The nice thing about this is that, if your device is compatible, when you sync the reading list, the Read/Unread can get updated automatically.

Note: If you use replace_metadata to do this, you need to do the search and replace for each class of metadata (defined in personal.ini) that generates a tag you don't want (I believe). So for example, for AO3 if you want nothing more than Title, Author, Series, you would need to do it for fandoms, freeformtags, ao3categories, characters, and ships, off the top of my head.

The long answer would have all the lines for the metadata replacement written out, but since I don't know which of the many, many sites FFDL is compatible with you actually use, and hence which metadata you want to delete, that seems like potential for a lot of wasted effort.

The basic format though would be (I believe):
metadata type=>search string=>replacement string (either nothing for deleting, or New, if you're doing it this way)
e.g.:
characters=>(.*)=>

The space before characters is necessary. "(.*)" matches anything
If you want to know more about constructing search strings using Python's regular expressions - see
http://manual.calibre-ebook.com/rege...regexptutorial
http://docs.python.org/2/library/re.html http://docs.python.org/2/howto/regex.html#regex-howto

If you want more on how to use personal.ini's replace_metadata beyond what's described in personal.ini, there's quite a bit in this thread if you go back a bit, mostly starting from around version 1.6.8, or September 2012, .

If something isn't clear, and unpacking is required, feel free to ask, I've certainly found people to be helpful here.

[JimmXinu]

Originally Posted by SallyK

While I'm bothering you, could I ask you about a completely different topic?

I want to get some metadata for the stories I download - title, author, series, but I don't want the tags, particularly not the over-abundance that you often get on AO3. What I'd really like to do is to automate what I do when I've downloaded the stories, which is to remove all the existing tags, and add one that says New.

Sorry, I missed seeing this post entirely until seabream replied to it.

Actually, it's really pretty easy. Go into personal.ini under [defaults] and add:

include_subject_tags:
extratags:New

FYI, the default values are:

include_subject_tags: extratags, genre, category, characters, ships, status
extratags: FanFiction

So you can adjust what ends up tags as you like.

And you can set it differently for different sites. For example, to only change it for archiveofourown.org put this at the end:

[archiveofourown.org]
include_subject_tags:
extratags:New

[Jade Aislin]

Originally Posted by JimmXinu

I'd experiment with using something like ([^\(]*?)--match (non-greedy) anything except open paren.

Frankly, there's a lot of trial and error in using complex regular expressions, even for experienced users.

After hours of reading the regular expression site and hours of tweaking and testing different codes, I went back to the basic one I started with, at least I thought I did. Of course, it worked. So I must have changed something in it to make it work.

ships_LIST=>(, )?(.*?)\((.*?)\)/(.*?)(, |$)=>\2/\4 (\3)\5

Now I'm working on consolidating all my regex code for categories, characters, and ships. I'm trying to make one list of changes that work for all sites.

I've placed the code into the [epub] section, but I'm wondering if it would work better in the defaults or overrides section. However, I've run into a couple categories that include a comma.

I already use these codes to get rid of periods and '|'s in the metatags:

replace_metadata:
 category=>\|=> -\s
 characters=>\|=> -\s
 ships=>\|=> -\s
 
 characters=>\.=>
 category=>\.=>
 ships=>\.=>

I've tried substituting the '.' or '|' with the comma and have placed the code in different places, including: under the test website [test1.com], [epub], [defaults], and [overrides] sections.

I've also tried:

 category=>Alice in Wonderland\, 2010=>Movie.Alice in Wonderland.2010

I even tried using category_LIST. Although, I'm not sure when the the metatags are separated by commas and sorted.

I thought that this would work:

replace_metadata:
 category=>\,=> -\s
 characters=>\,=> -\s
 ships=>\,=> -\s

But I'm not sure where to put it.

The strange thing is that I looked at one of my other libraries that has my old personal ini and it had this code, which still works (at least in that library):

category=>Alice in Wonderland, 2010=>Alice in Wonderland (2010)  
  category=>(Alice in Wonderland)=>Movie.\1

The code was under [www.fanfiction.net], but if it needed to be under the site, [test1.com] should have worked in my testing library.



Am I right in assuming that the '\,=>' should replace the commas in each individual tag? If so, where should I be placing it? Or should I be looking at the metatags that have commas in a different way?

[JimmXinu]

Fixes (hopefully) for http://dark-solace.org/elysian changes.

UPDATE Jul 28, 2013 - Remove obsolete beta versions

[JimmXinu]

Originally Posted by Jade Aislin

After hours of reading the regular expression site and hours of tweaking and testing different codes, I went back to the basic one I started with, at least I thought I did. Of course, it worked. So I must have changed something in it to make it work.

Yep. That's the way it goes with regular expressions. However, I don't know of a better way to give you the level of data customization you seem to want. The alternative is for you to learn python and start writing code instead. That is an option, if you like.

Originally Posted by Jade Aislin

Now I'm working on consolidating all my regex code for categories, characters, and ships. I'm trying to make one list of changes that work for all sites.

I've placed the code into the [epub] section, but I'm wondering if it would work better in the defaults or overrides section. However, I've run into a couple categories that include a comma.

I believe those are two unrelated issues. Section choice vs comma, that is.

If you plan to use the same replace_metadata for all sites and all formats, [defaults] is the most appropriate section. Then you can still put replace_metadata under specific sites if needed, or use add_to_replace_metadata in those sites.

Section precedence is:

1. [defaults]
2. [test1.com]
3. [epub]
4. [test1.com:epub]
5. [overrides]

So you see, if you use [epub], anything site specific in [test1.com] will be 'overwritten' by [epub].

It gets more complicated when you consider add_to_* config lines.

If you put a replace_metadata in a higher precedence section([test1.com] for example), it overwrite replace_metadata from lesser precedence sections([defaults] for example).

But if you use add_to_replace_metadata instead, it does just that: adds to the replace_metadata instead of replacing it.

[defaults]
replace_metadata:
 characters=>Robert=>Bob

[test1.com]
add_to_replace_metadata:
 characters=>Susan=>Suzie

is equivalent to:

[test1.com]
replace_metadata:
 characters=>Robert=>Bob
 characters=>Susan=>Suzie

Spoiler:

Technically, the full precedence list is:

1. [defaults]
2. [www.test1.com]
3. [test1.com]
4. [epub]
5. [www.test1.com:epub]
6. [test1.com:epub]
7. [overrides]

But that's a level of detail you probably don't need.

FYI, while you're consolidating, you can also apply the same regex to more than one entry at a time for example:

# instead of:
replace_metadata:
 category=>\|=> -\s
 characters=>\|=> -\s
 ships=>\|=> -\s

# use:
replace_metadata:
 category,characters,ships=>\|=> -\s

Originally Posted by Jade Aislin

Am I right in assuming that the '\,=>' should replace the commas in each individual tag? If so, where should I be placing it? Or should I be looking at the metatags that have commas in a different way?

Yes, it should. Except you don't need to escape commas in regex.

If you succeed in getting all your replace_metadata into one big list, I'd put it at the end in case any of the earlier patterns depend on commas.

If you want to allow for possible sites specific regex still, put it under overrides as an add_to_replace_metadata. That will put it at the end of any and all replace_metadata chains that get built up.

[overrides]
add_to_replace_metadata:
 category,characters,ships=>,=>\s

[Linwe]

I haven't found something quite like this in the pages, but forgive me if it's already been discussed.

I made a custom column named Fandom that I want to put shorthand or abbreviated categories into. I.E., "Ouran High School Host Club" becomes "OHC", and "How To Train Your Dragon" becomes "HTTYD" and are then loaded into the custom Fandom column.

If this is possible how do I go about it?

(I'm currently drugged up on Dayquil fighting a cold, so my attention span is about the same as ... @@)

THank you for any and all help.

[JimmXinu]

Originally Posted by Linwe

I made a custom column named Fandom that I want to put shorthand or abbreviated categories into. I.E., "Ouran High School Host Club" becomes "OHC", and "How To Train Your Dragon" becomes "HTTYD" and are then loaded into the custom Fandom column.

It's possible, but it's not necessarily simple.

If you want the abbreviations on the title page and in tags as well, you can make the changes to the category entry directly in personal.ini:

[defaults]
replace_metadata:
 category=>Ouran High School Host Club=>OHC
 category=>How To Train Your Dragon=>HTTYD

Then you can fill your Fandom column from the GUI settings of the FFDL Custom Column tab.

However, if you want only to have the abbreviations in the Fandom column, you need to Allow custom_columns_settings (towards the bottom of the FFDL Custom Column tab) and include something like this in personal.ini:

[defaults]
# Adds a new 'extra' valid entry
add_to_extra_valid_entries:,fandom

# copy category to fandom
include_in_fandom:category

# replace with abbrevs in fandom
replace_metadata:
 fandom=>Ouran High School Host Club=>OHC
 fandom=>How To Train Your Dragon=>HTTYD

# set custom column with fandom value.
custom_columns_settings:
 fandom=>#fandom

replace_metadata can be quite complex.

Spoiler:

## Use regular expressions to find and replace (or remove) metadata.
## For example, you could change Sci-Fi=>SF, remove *-Centered tags,
## etc.  See http://docs.python.org/library/re.html (look for re.sub)
## for regexp details.
## Make sure to keep at least one space at the start of each line and
## to escape % to %%, if used.
## Two, three or five part lines.  Two part effect everything.
## Three part effect only those key(s) lists.
## *Five* part lines.  Effect only when trailing conditional key=>regexp matches
## metakey[,metakey]=>pattern=>replacement[&&conditionalkey=>regexp]
## Note that if metakey == conditionalkey the conditional is ignored.
## You can use \s in the replacement to add explicit spaces.  (The config parser
## tends to discard trailing spaces.)
## replace_metadata <entry>_LIST options: FFDL replace_metadata lines
## operate on individual list items for list entries. But if you
## want to do a replacement on the joined string for the whole list,
## you can by using <entry>_LIST. Example, if you added
## calibre_author: calibre_author_LIST=>^(.{,100}).*$=>\1
replace_metadata:
 genre,category=>Sci-Fi=>SF
 Puella Magi Madoka Magica.* => Madoka
 Comedy=>Humor
 Crossover: (.*)=>\1
 title=>(.*)Great(.*)=>\1Moderate\2
 .*-Centered=>
 characters=>Sam W\.=>Sam Witwicky&&category=>Transformers
 characters=>Sam W\.=>Sam Winchester&&category=>Supernatural

There was talk once of some of the users who do tons of data customization sharing their setups, but I don't remember seeing any posted.

[JimmXinu]

Version 1.7.34 - 28 Jul 2013

• Fixes for fanfiktion.de, storiesofarda.com, hpfanficarchive.com, and dark-solace.org/elysian sites.

[Jade Aislin]

Originally Posted by JimmXinu

The alternative is for you to learn python and start writing code instead. That is an option, if you like.

I thought about trying to teach myself python. Aside from the fact that I'm terrible at teaching myself anything, I was only able to make it through the basic classes for C++ and Java in college.

Originally Posted by JimmXinu

I believe those are two unrelated issues. Section choice vs comma, that is.

Yes. I didn't mean to make it seem like it was one problem.

Originally Posted by JimmXinu

If you plan to use the same replace_metadata for all sites and all formats, [defaults] is the most appropriate section. Then you can still put replace_metadata under specific sites if needed, or use add_to_replace_metadata in those sites.

Section precedence is:

1. [defaults]
2. [test1.com]
3. [epub]
4. [test1.com:epub]
5. [overrides]

I was not aware of the order of precedence. If I am understanding this right, I should use replace_metadata in [defaults](Would there be a reason to use add_to_metadata there?). Then if I don't want one of the lower sections to overwrite that, I use add_to_replace_metadata.

Originally Posted by JimmXinu

FYI, while you're consolidating, you can also apply the same regex to more than one entry at a time

I knew that and was planning on combining the characters and ships. Ironically, I didn't think about it for the simple changes to get rid of key characters like commas and '|'.

Originally Posted by JimmXinu

If you succeed in getting all your replace_metadata into one big list, I'd put it at the end in case any of the earlier patterns depend on commas.

If you want to allow for possible sites specific regex still, put it under overrides as an add_to_replace_metadata. That will put it at the end of any and all replace_metadata chains that get built up.

[overrides]
add_to_replace_metadata:
 category,characters,ships=>,=>\s

I'm a little confused. Wouldn't I want to put it before the category,characters,ships lists get sorted?

I'm trying to transfer my code into the default personal ini to reduce the clutter I seem to have from different tweaks, so I'm not sure where I placed the replace_metadata at the moment as I moved it a few times. But I can try placing the ',=>\s' in [overrides] and see what happens.

[JimmXinu]

Originally Posted by Jade Aislin

I was not aware of the order of precedence. If I am understanding this right, I should use replace_metadata in [defaults](Would there be a reason to use add_to_metadata there?). Then if I don't want one of the lower sections to overwrite that, I use add_to_replace_metadata.
...
I'm trying to transfer my code into the default personal ini to reduce the clutter I seem to have from different tweaks, so I'm not sure where I placed the replace_metadata at the moment as I moved it a few times. But I can try placing the ',=>\s' in [overrides] and see what happens.

I can really only give you generalities because I've no idea what rules you're using--I just assume by now it's a HUGE rule set.

If you truly can get all the rules you want for all the sites you read into one set of replace_metadata rules, then [defaults] is the best place for them, IMO.

The precedence order exists for those cases when you can't or don't want all your rules the same. For example, when you want different rules for different sites.

Let's assume for the moment epub and not other formats from the same personal.ini. That simplifies the precedence order to:

1. [defaults]
2. [test1.com]
3. [overrides]

The idea that overrides config values replace test1.com config values which replace defaults config values is pretty simple and I assume you get that. I confess that add_to_ can be a bit more confusing.

Here's a little demo I used to remind myself of how it works. It uses the title label because that is simple to see and generally doesn't show up very much (logpage only, really).

Spoiler:

# initial value set from defaults.ini
[defaults]
title_label: Title

# from personal.ini
[defaults]
add_to_title_label:,defaults

[test1.com]
add_to_title_label:,test1.com
add_to_titlepage_entries: ,title

[overrides]
add_to_title_label:,overrides

After generating test1.com?sid=2 for example, at the end of it's title page is:

Title,defaults,test1.com,overrides: Test Story Title 2

See how each add_to_title_label was added in order?

Additions go on regardless of which level the 'base' value is set from.

# initial value from defaults.ini
[defaults]
title_label: Title

# from personal.ini
[defaults]
add_to_title_label:,defaults

[test1.com]
# replaces defaults value:
title_label:Test1Title

add_to_title_label:,test1.com
add_to_titlepage_entries: ,title

[overrides]
add_to_title_label:,overrides

...yields: Test1Title,defaults,test1.com,overrides: Test Story Title 2

[Jade Aislin]

Originally Posted by JimmXinu

I can really only give you generalities because I've no idea what rules you're using--I just assume by now it's a HUGE rule set.

Yes, it is huge. Many of my fandoms have lots of characters who names needs to be fixed which is why I want to consolidate it.

If I am looking for a story in the fandom: Transformers, I don't want to have to search through: Transformers, Transformers/Beast Wars, and Transformers Generation one. Those first two are even from the same site (FFN). Nor do I want to search through multiple tags of the same character. That's what most of my regex code is for.

I thought if I could put the code in one place, I'll only need to adjust it when a site throws a new way to label a particular character. This would also reduce the amount of code I have.

Originally Posted by JimmXinu

If you truly can get all the rules you want for all the sites you read into one set of replace_metadata rules, then [defaults] is the best place for them, IMO.

Unfortunately, I finally figured out what was different about the code that did work for me.

I tested the code from an older personal ini which was under my FFN section:

replace_metatag:
 #Movie
 category=>Alice in Wonderland, 2010=>Alice in Wonderland (2010)
 
 category=>(Alice in Wonderland|^Avatar$)=>Movie.\1

 #Movie
   
 #Alice in Wonderland
 characters=>Alice K=>Movie.Alice in Wonderland.Alice Kingsleigh&&category=>Alice in Wonderland \(2010\)

I tested it under defaults, test site, and the original site (FFN, with an actual storyurl (http://www.fanfiction.net/s/7461968/1/)). The only one it worked under was FFN, where both the character (Alice K) and (Alice in Wonderland, 2010) converted to the way I wanted without extra code. Through some testing with the test site, it seems that the &&category is preventing the regex from working in any but active sites. Is this how it is supposed to work?

Because unfortunately, I need the &&category as I know of at least one set of character names (Sam W) that belong to two different fandoms (Transformers, Supernatural). Depending on which fandom the character belongs to, the regex will have different output.

Originally Posted by JimmXinu

The precedence order exists for those cases when you can't or don't want all your rules the same. For example, when you want different rules for different sites.

This sounds great. I can leave code that only applies to the one site in the sites section. That way, I'm not constantly reusing the same code for each site. If I could get the characters and ships to work in defaults with the &&category, I'd be all set.

[redscarlett]

Hi
I have a lot of ebooks in calibre that are works-in-progress and have been downloaded via FFDL, so I use that to update them. The thing is it often wants to update fics that don't need to be updated - the log states that it wants to re-download all the chapters, and I have no idea why. Is this just a quirk?