[GUI Plugin] FanFictionDownLoader

[Firedancer885]

Traceback (most recent call last):
  File "downloader.py", line 317, in <module>
    main(sys.argv[1:])
  File "downloader.py", line 248, in main
    adapter.getStoryMetadataOnly()
  File "/fanficdownloadercli/fanficdownloader/adapters/base_adapter.py", line 261, in getStoryMetadataOnly
    self.story.setMetadata('dateUpdated',self.story.getMetadataRaw('datePublished'))
  File "/fanficdownloadercli/fanficdownloader/story.py", line 269, in setMetadata
    self.addToList('lastupdate',value.strftime("Last Update Year/Month: %Y/%m"))
AttributeError: 'NoneType' object has no attribute 'strftime'

Thank you for the cli suggestion I used the Imagine.e-fic adapter as base for the new adapter. What do I need to change?

[JimmXinu]

Originally Posted by Firedancer885

Traceback (most recent call last):
  File "downloader.py", line 317, in <module>
    main(sys.argv[1:])
  File "downloader.py", line 248, in main
    adapter.getStoryMetadataOnly()
  File "/fanficdownloadercli/fanficdownloader/adapters/base_adapter.py", line 261, in getStoryMetadataOnly
    self.story.setMetadata('dateUpdated',self.story.getMetadataRaw('datePublished'))
  File "/fanficdownloadercli/fanficdownloader/story.py", line 269, in setMetadata
    self.addToList('lastupdate',value.strftime("Last Update Year/Month: %Y/%m"))
AttributeError: 'NoneType' object has no attribute 'strftime'

Thank you for the cli suggestion I used the Imagine.e-fic adapter as base for the new adapter. What do I need to change?

Work through it by visiting each of the stack trace areas. This time, start from the bottom.

1. story.py, line 269 - When setting dateUpdated, it's trying to strftime on a None.
2. base_adapter.py, line 26 - When dateUpdated isn't set, it tries to set it to the same value as datePublished.

So that tells you that your adapter is setting datePublished to None, or not setting it.

[skb]

Okay, getting scared now. I've just run the plugin from the CMD line. But I'm getting closer (I think).

File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fictioningnet", line 109, in extractChapterUrlsAndMetadata
TypeError: object of type 'NoneType' has no len()

Line 109 is, I think, about the Story Title?

self.story.setMetadata('title',stripHTML(a)[:(len(a.string)-3)])

In case anyone doesn't know how to do this, I used this code (you'd replace the URL with your target site, of course):

Spoiler:

calibre-debug -r "FanFictionDownLoader" "http://fictioning.net/viewstory.php?sid=503"

[JimmXinu]

Originally Posted by skb

File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fictioningnet", line 109, in extractChapterUrlsAndMetadata
TypeError: object of type 'NoneType' has no len()

Line 109 is, I think, about the Story Title?

self.story.setMetadata('title',stripHTML(a)[:(len(a.string)-3)])

If that's line 109, sure. We're not seeing your code, you know.

So think about what happens before that--where did variable 'a' come from? What's in 'a'? Toss a line before that prints out the value of 'a'.

[skb]

LOL I've got NO idea where the 'a' came from. I used someone else's py adapter (that's using efiction - I think the ncis one). I'm beginning to see that it may've been a bad idea...

I'll keep going.

[Jade Aislin]

I've recently been going through my personal ini to clean it up. I've been working on the categories (fandoms) and characters for both FFN and AO3. Unfortunately, AO3 lets the author put anything in the character slots. I downloaded 135 stories and now have 70 characters that I consider junk (ex: and some more).

I've asked about getting rid of these characters before and received this code:
characters=>^.*?(|Anime/Manga|Book.LOTR.Aragorn <Estel-Strider>)$=>\1


But that isn't working. It seems to be erasing all the characters, even the ones I want to keep.

Most of the code for editing characters is under default, as I want it to be used for all sites. However, as AO3 is the only site that needs to have junk characters deleted, I placed that code under the testing site in the Personal ini. Should I place it under the default settings with the rest of the code?

Also, I'm hoping to tweak the code so that it can look for keywords at the beginning of each character (Book|Cartoon|Movie|etc) and then delete any that don't match.

Is there any code I can use or do I need to delete those junk characters manually?

[JimmXinu]

Originally Posted by Jade Aislin

I've recently been going through my personal ini to clean it up.
...

If you want to give specific examples of problems you're having with metadata values and your settings, I'll try to help.

But I don't do anything nearly as complicated with metadata as some of you guys do. So I don't have a tool kit of solutions for metadata manipulation; just the answers given here to previous questions. Which I've mostly forgotten.

[Jade Aislin]

One of the stories I'm working with is:
http://archiveofourown.org/works/935...apters/1821204

I don't have a problem managing the ones I want to keep (ie. Bilbo Baggins, Gandalf, etc). It's the metadata that I don't want to keep (ie. anyone else who feels like popping in) that I'm having trouble with.

my metadata looks like this:

Spoiler:

[defaults]
 #Note: The code has been cut down to the necessary code for the above URL.
 #Step one: delete any periods in the category and character name.
add_to_replace_metadata:
 category,characters,ships=>\.=>
 category,characters,ships=>\|=> -\s
 category,characters,ships=>,=>

#clean up fandom names

 category=>( - )?(( )?Fandom|All (Media Types|Series)|(J R R )?Tolkien|[Ww]orks|\(((Bay )?(Jackson )?[Mm]ovies|Cartoon Generation One|TV|Video Game|Manga|AU|2001 2002 2003|2012|1986|Novel)\))=>

 #Book
 category=>^(The )?[Hh]obbit( - )?(: An Unexpected Journey)?=>LOTR.Hobbit


 category=>(^|, )((LOTR|HP|Sherlock Holmes|Twilight|Black Jewels Trilogy)(.)?(.*)?)=>\1Book.\2


#Step four:  Clean up character,ships names 

 characters,ships=>((\()?[Ff]em(ale)?(\))?(( |!))?|\((([Tt]he )?Hobbit|Tolkien|Character|Labyrinth|Mass Effect|BtVS)\)| - Character)=>
 
 #Book

 #LOTR
 characters,ships=>([Bb]ilb[ao](g)?|B[ie]lla|Beryl|Bilbi)( 'briar')?( [Bb])?(aggins)?( reincarnation)?=>Bilbo Baggins&&category=>LOTR

 characters,ships_LIST=>(^|, )(Thranduil|Thorin's Company|Gandalf|Bilbo Baggins)=>\1Book.LOTR.\2&&category=>LOTR


[test1.com]
extratags: FanFiction,Testing
add_to_replace_metadata:

#blanks
 
 characters=>^.*?(|Anime/Manga|Book.LOTR.Bilbo Baggins)$=>\1

The last line of code is supposed to get rid of the extra metadata I don't want. I figured out why the testing (Aragorn <Estel-Strider>) was not working. Unfortunately, this code means I have to give the entire metadata I want to keep (ie. Book.LOTR.Bilbo Baggins, Book.LOTR.Gandalf). I would have to include a lot of names and pairings to get that to work.

I'd rather just look at the beginning of the the metadata to determine whether or not to keep it.

I've tried to replace the last line of code with four different lines:

 #Let's nothing through
 characters=>^.*?(|Anime/Manga|Book)$=>\1

 #Let's everything through
 characters=>^.*?(|Anime/Manga|Book)=>\1  
 characters=>^.*?((|Anime/Manga|Book)(.*?))$=>\1
 characters=>^.*?((Anime/Manga|Book)(.*?))$=>\1

I've been testing with: all the hobbits,Bilbo. I'm trying to get Bilbo changed to Book.LOTR.Bilbo Baggins and 'all the hobbits' should be erased.

None of the codes I've tried has worked. Basically, I want the code to look for keywords at the beginning of the metadata (Book, Game, Movie, etc.). If the metadata does not start with those keywords, then it should be erased.

Of course, I am assuming that all the code in the defaults section will be done before the code in the website sections. Maybe it would work better if I put that last line of code into the defaults section at the end of the replace_metadata code.

[JimmXinu]

Well, for one thing, settings under [test1.com] obviously aren't going to effect a story from an AO3 URL.

So presumably you also have a [teststory:1001] section or some other number you're testing with? Because right now, I can't even duplicate your test setup.

(Does putting blank lines and comments in an extended replace_metadata setting block even work? I guess it must.)

[JimmXinu]

You know what? This has gotten really out of hand.

I'm sure there are other users who'd like a simple feature to include or exclude metadata values that's separate from the regex driven replace_metadata feature.

Let me go see what I can think up.

[Firedancer885]

Originally Posted by JimmXinu

You know what? This has gotten really out of hand.

I'm sure there are other users who'd like a simple feature to include or exclude metadata values that's separate from the regex driven replace_metadata feature.

Let me go see what I can think up.

That would be awesome!!!

[Jade Aislin]

Originally Posted by JimmXinu

You know what? This has gotten really out of hand.

I'm sure there are other users who'd like a simple feature to include or exclude metadata values that's separate from the regex driven replace_metadata feature.

Let me go see what I can think up.

That would be great.

Of course, after I had condensed my personal ini code and was writing a reply to your last post, I think I solved my problem.

I tried this code:

 characters=>^.*?(|(Anime/Manga|Book)(.*?))$=>\1

Before the first '|' was within the second parentheses, which meant it was accepting anything.

Even though I figured out my problem, a way to do that without the replace_metadata feature would be great.

[iakovl]

ok... just found this pluging and might be the only reason i would use calibre right now
i read a lot of fimfiction storys, right now i still use manual download as a solution

will calibre and this plugin give the the right solution to auto download new chapters and showing changes in storys i'm reading?

[JimmXinu]

Originally Posted by iakovl

will calibre and this plugin give the the right solution to auto download new chapters and showing changes in storys i'm reading?

Yes, and no.

FFDL can update existing epubs--that is, download only the new chapters when new chapters are added to the end of a story.

But it won't do it automatically.

It's not recommended to use FFDL to poll for changes; ie, select every book in your library and tell FFDL to try to update them. I highly recommend that you use each fanfic site's email notification for the stories you follow and then use that to know which books to update.

You can drag story URLs from email onto the FFDL icon on the calibre toolbar, or into the URL box when FFDL's add/update by URL dialog is open.

[JimmXinu]

Well, I set out to make a simple way to include or exclude metadata values. But like most of my projects, it quickly spiraled into something considerably more complex than I'd first intended.

This is very much a test version because I'm not convinced this should even be done, let alone done this way.

And the explanation is crazy long:

Spoiler:

Let's start with some simple cases first.

UPDATE May 5, 2014 - Remove obsolete beta versions

'Include' in this case really means 'include ONLY these.' If a
metadata type has lines in include, only those values will be kept,
all others are discarded.

UPDATE May 5, 2014 - Remove obsolete beta versions

(Note that the separator is '==', which is different than in
replace_metadata.)

## in stories from site dramione.org, only use these 4 characters, all
## others will be omitted.  Because only characters are listed, only
## characters will be effected.
[dramione.org]
include_metadata_pre:
 characters==Draco Malfoy
 characters==Hermione Granger
 characters==Harry Potter
 characters==Ron Weasley

## in stories from site www.fimfiction.net, only keep these 2
## characters, all others will be omitted.  And only Comedy and
## Adventure genres will be used--all other genres will be omitted.
## All other metadata will be as usual.
[www.fimfiction.net]
include_metadata_pre:
 characters==Twilight Sparkle
 characters==Rainbow Dash
 genre==Comedy
 genre==Adventure

If, on the other hand, you want to keep most values, and want to
exclude only certain values you use 'exclude' instead. (The =~ will
be explained next.)

## in stories from site thequidditchpitch.org, exclude warnings 'Minor
## Fluff', AU and anything with 'Contains Spoiler' in it; and the genre
## PWP.  All other values will be kept.
[thequidditchpitch.org]
exclude_metadata_pre:
 warnings==Minor Fluff
 warnings==Alternate Universe
 warnings=~Contains Spoiler
 genre==PWP

In both include and exclude, you can use regular expressions(regex)
instead of simple strings by using '=~' for the separator.

## in stories from site buffynfaith.net, only include ships with Buffy
## in them, even if spelled with a lower case 'b'.
[buffynfaith.net]
include_metadata_pre:
 ships=~[Bb]uffy

## in stories from site www.fimfiction.net, only use these 2
## characters, all others will be omitted.  All other metadata will be
## as usual.
[www.fimfiction.net]
include_metadata_pre:
 characters=~^(Twilight Sparkle|Rainbow Dash)$

## exclude the Mane Six or Main 6 or variation.
exclude_metadata_pre:
 characters=~(Mane|Main) (Six|6)

But what happens when you're using a site that has more than one
category of story and you only want to affect one fandom at a time?
You can use conditionals similar to replace_metadata. And in
include/exclude, conditionals can be either exact string(==) or
regex(=~) too.

While an include line for say 'characters' without a conditional means
only characters explicitly matched by include lines will be kept, an
include line with a conditional only effects stories that meet the
conditional.

## in stories from site archiveofourown.org, when category contains
## 'The Hobbit', only include these three characters.  All other
## stories will continue to have full characters.
[archiveofourown.org]
include_metadata_pre:
 characters==Bilbo&&category==The Hobbit
 characters==Samwise&&category==The Hobbit
 characters==Gandalf&&category==The Hobbit

## still on archiveofourown.org, exclude any character with Bucky in
## it, but only for Marvel, Captain America & Avengers
exclude_metadata_pre:
 characters=~Bucky&&category=~(Marvel|Captain America|Avengers)

Note that include is applied first, then exclude.

## in stories from site buffynfaith.net, only include ships with Buffy
## in them, even if spelled with a lower case 'b'.
[buffynfaith.net]
include_metadata_pre:
 ships=~[Bb]uffy

## But exclude any Buffy/Dawn.
exclude_metadata_pre:
 ships=~[Bb]uffy/Dawn

You may have noticed that all the examples so far ended in '_pre'.
That's because all of these were applied *before* replace_metadata is
applied. You can also use include_metadata_post and
exclude_metadata_post to filter *after* replace_metadata is applied.

[default]
replace_metadata:
 ... tons of code to tweak metadata to your exacting standards
 ... and normalize all the character names juuuust right

## Now make sure that only characters that start with your corrected
## formatting are included.
include_metadata_post:
 characters=~^(Books|Anime|Movies)\.

Now one last twist. In addition to == and =~ to match exact strings
and regex, you can also use != and !~ to *negate* the match. Much of
what you can do with negated matches can be done by include/exclude,
but some things can't. Don't go crazy with negated matches--Figuring
out which to use when and why can be quite complicated.

[www.fimfiction.net]
## These two on their own are functionally equivalent: exclude
## 'Twilight Sparkle' in characters
include_metadata_pre:
 characters!=Twilight Sparkle

exclude_metadata_pre:
 characters==Twilight Sparkle

## But these two are *not*:

## excludes these two as expected.
exclude_metadata_pre:
 characters==Twilight Sparkle
 characters==Rainbow Dash

## includes all characters because Rainbow Dash matches != Twilight
## Sparkle and Twilight Sparkle matches != Rainbow Dash.  So both
## match an include line and are kept.
include_metadata_pre:
 characters!=Twilight Sparkle
 characters!=Rainbow Dash

Realistically, I expect negated matches to be more useful in the
conditionals.

[fanfiction.net]
## Exclude Harry Porter, Harry Pearson, etc from other fandoms.
exclude_metadata_pre:
 characters==Harry P.&&category!=Harry Potter

And finally, if you need a trailing space in a string or regex, you can use \s and
FFDL will convert it to a space.

Everybody, but Jade & FireDancer especially, let me know what you think.