[GUI Plugin] FanFictionDownLoader

[fathertyme]

Thanks JimmXinu, that test version solved the problem for me at least :P

Thats what I love about you guys, I come, I explain, you solve...

aint love grand :P

-tyme

[JimmXinu]

Originally Posted by JimmXinu

I'm seeing that problem, too. But it's only with Chapter 27 of that story.

So I suspect there's something wrong in the HTML of that chapter. I will investigate further, but there are a couple other issues to look at this morning, too.

The problem is being caused by some incorrect tag nesting in the chapter 27 text:

<em><br><center>You know this has to be
<br>we always were so free
<br>we promised we'd be
<p>
<strong>...perfect...</em></strong></center>

(This author (or their HTML editor) apparently also likes using <dd> tags. A lot. That's the left ident in most of this story.)

The HTML parser we use is somewhat forgiving of incorrect nesting, but that example takes it too far for the current parser configuration.

I can get it to parse that chapter by telling it to accept <p> tags as self-closing, but that change will effect all future chapters downloaded from fanfiction.net.

I don't think that will cause problems, but I can't promise it won't either. So this is only going into test versions for a while.

I'm sure cryzed is willing to remind me that there are newer parsers available, but they are not easy to include in all the different forms FFDL takes. I may take another look at them, but I'm more concerned right now about calibre 2.10 breaking keyboard shortcuts in FFDL.

[cryzed]

Originally Posted by JimmXinu

I'm sure cryzed is willing to remind me that there are newer parsers available, but they are not easy to include in all the different forms FFDL takes. I may take another look at them, but I'm more concerned right now about calibre 2.10 breaking keyboard shortcuts in FFDL.

Yes, I know that; I remember that I tried but that there were some complications. I'm just trying to help, I don't want to nag. Good luck!

[JimmXinu]

Originally Posted by cryzed

Yes, I know that; I remember that I tried but that there were some complications. I'm just trying to help, I don't want to nag. Good luck!

I'm just acknowledging it, so you know I'm not just ignoring you.

[MidwestJen]

I apologize first if this has been answered somewhere else. I've been researching for the last few days and haven't found anything, but Google does occasionally let me down.

I like to keep my metadata in a certain way (I may have a touch of OCD when it comes to keeping my data clean and pretty). I have 3 custom columns all for the site's data - #siterating, #siteupdated, and #sitetags. The #siterating is for whatever rating that particular website uses (G, M, NC-17, etc) - when I actually read the story I put the rating I feel it should have in #myrating. This ensures a bit more consistency and I can adjust it up or down based on the author's prudishness or lack thereof without losing the original data - important for changes in rating on an in-progress story that I've already adjusted. The #siteupdated is just the date it was last updated on the site itself.

Those two columns work perfectly fine and aren't really a problem. The tags though, that's another story. It's not as much of a problem on ff.net, but sites like ao3 can go nuts on tags, which means I was either living with not getting anything new after I had updated/fixed the tags or I was putting up with five million tags that drove me crazy - so I created the new column to keep the two sets separate. Unfortunately I need to use the standard calibre tags column as the #mytags (if I had followed the schema above) for a few different reasons (metadata on the book jacket being in alphabetical order is simply one of them). So my problem became how to get the standard tags into the #sitetags column. For a new download it was fine if they both filled in, although I would have preferred only the #sitetags - I just couldn't figure out how to make that happen.

My cobbled together answer was to add this to my personal.ini:
extratags=>#sitetags,a
genre=>#sitetags,a
category=>#sitetags,a
characters=>#sitetags,a
ships=>#sitetags,a
status=>#sitetags,a

I also checked the "New only" for tags on the standard column tab and left the custom column tab 'Type of Data' for Site Tags empty.

For an updated download, whether it was for a new chapter or to update the metadata, this works fine. The information passed to the #sitetags column is complete and accurate (I blanked a few of them out and tested it to be sure). For a new download though, it fails to work the same. The only information sent to the #sitetags column is the "status" (ie Completed, In-Progress, etc). The tags column works fine in both instances. If I update that new story and ask for just metadata though, the #sitetags information is filled in just like any other update.

Is there a way to tell FFDL not to fill in a standard column and pass that information to a custom column on the initial download? Am I not redirecting the information correctly? Or is the problem simply what I'm trying to do and not how I'm doing it? I am getting by using other ways (search/replace is my savior), but I thought I'd ask.

Thanks in advance for any help or advice!

[JimmXinu]

Originally Posted by MidwestJen

...

My cobbled together answer was to add this to my personal.ini:

custom_columns_settings:
 extratags=>#sitetags,a
 genre=>#sitetags,a
 category=>#sitetags,a
 characters=>#sitetags,a
 ships=>#sitetags,a
 status=>#sitetags,a

I also checked the "New only" for tags on the standard column tab and left the custom column tab 'Type of Data' for Site Tags empty.

... For a new download though, it fails to work the same. The only information sent to the #sitetags column is the "status" (ie Completed, In-Progress, etc). The tags column works fine in both instances. If I update that new story and ask for just metadata though, the #sitetags information is filled in just like any other update.
...

Using multiple add('a') custom settings for the same column isn't something I anticipated. But it is a clever solution that almost works. And it doesn't work on new downloads because I didn't anticipate it in implementing new('n') custom column settings.

Attached is a new test version that should work as you are using it for both new downloads as well as updates.

[JimmXinu]

So you actually have how many different problems here??

Originally Posted by firefoxxy

I wanted to get tags like "a little bit fluff", or "angst and fluff" to become "fluff" but I do not now how to write that. I had used

category=>*.fluff=>fluff

which caused the previous error. I removed this line.

It's '.*' (dot then star), not '*.'. And if you want to match anything containing fluff, use both at front and back:

 category=>.*fluff.*=>fluff

Originally Posted by firefoxxy

I also tried to get the plugin to rename certain other tags (like Hurt/Comfort into Hurt-Comfort, or "Frigga (Marvel)" to "Frigga") which doesn't work.

This works fine for me:

 category=>Hurt/Comfort=>Hurt-Comfort

As for "Frigga (Marvel)" to "Frigga", this works fine for me:

 characters=>Frigga.*=>Frigga

Originally Posted by firefoxxy

Next, I noticed that the ship-tags like "Harry Potter/Severus Snape" are not converted to anymore, although it had been working. ((Edit: It does sometimes work, but sometimes not - I have yet to check which stories don't change the tag... Sorry))
...

At this point, I think you have so much complexity that parts are interfering with each either.

I'll try to help if you post specific problems with specific stories and the complete replace_metadata that applies, but it takes too much of my time to try and construct test cases for your problems without specifics.

Ultimately, if you want to control every piece of information, you have to control every piece of information.

[ILB]

Originally Posted by JimmXinu

The problem is being caused by some incorrect tag nesting in the chapter 27 text:

<em><br><center>You know this has to be
<br>we always were so free
<br>we promised we'd be
<p>
<strong>...perfect...</em></strong></center>

(This author (or their HTML editor) apparently also likes using <dd> tags. A lot. That's the left ident in most of this story.)

The HTML parser we use is somewhat forgiving of incorrect nesting, but that example takes it too far for the current parser configuration.

I can get it to parse that chapter by telling it to accept <p> tags as self-closing, but that change will effect all future chapters downloaded from fanfiction.net.

I don't think that will cause problems, but I can't promise it won't either. So this is only going into test versions for a while.

I'm sure cryzed is willing to remind me that there are newer parsers available, but they are not easy to include in all the different forms FFDL takes. I may take another look at them, but I'm more concerned right now about calibre 2.10 breaking keyboard shortcuts in FFDL.

I don´t think it is necessary to change it ultimate - especially when there might be problems in the future.
Can you tell me if there is an easy possibility how I can download the story regardless?
Otherwise I do without the story.

[JimmXinu]

Originally Posted by ILB

I don´t think it is necessary to change it ultimate - especially when there might be problems in the future.
Can you tell me if there is an easy possibility how I can download the story regardless?
Otherwise I do without the story.

Use the latest test version to download it. Then if we do decide to discard that change you at least have the story already. It's a completed story(12 years ago?!), so that should be end of it for that one.

[MidwestJen]

Originally Posted by JimmXinu

Attached is a new test version that should work as you are using it for both new downloads as well as updates.

It worked perfectly, thank you!

[ILB]

Originally Posted by JimmXinu

Use the latest test version to download it. Then if we do decide to discard that change you at least have the story already. It's a completed story(12 years ago?!), so that should be end of it for that one.

Thanks!

[Beraias]

Well not sure if anyone posted something like this or not.

I'm using the latest version of both Calibre (I can't find the version number) and Fanfiction downloader (2.0.9)

I have a friend who uses this to download froma place called Fimfiction.net. However when I am trying to download a mature story I get this error:


Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
Story requires confirmation of adult status: (http://www.fimfiction.net/story/119190/)
http://www.fimfiction.net/story/119190/


How do you even fix this? I have an account with the website. I am in fact 26 years old. I'd like to know whats going on as I've been wanting to read Fallout: Equestria for ages now

[JimmXinu]

Originally Posted by Beraias

...
I have a friend who uses this to download froma place called Fimfiction.net. However when I am trying to download a mature story I get this error:
...
Story requires confirmation of adult status: (http://www.fimfiction.net/story/119190/)
...

Apparently the popup 'Are you an Adult?' isn't working with Fimf right now.

In the mean time, it will work if you go into FFDL config, on the personal.ini tab and add:

[defaults]
is_adult:true

I believe most of us have that set, so we don't see (and therefore don't test) the popup confirmation very often.

[Beraias]

Unfortunatly I seem to see it every single time. But I've fiddled with the setting sin the personal.ini.. so maybe it will work now.

It works!

[JimmXinu]

Attached is (yet another) test version, with all of the changes previous, plus:

• Fix for FimF not working with the manual 'Are you an Adult' popup, and;
• Fix for the keyboard shortcuts issue with calibre 2.10.
• Still contains the self-closing <p> change for that one ffnet story--haven't decided yet if I should include that or not in the next release.