[GUI Plugin] FanFictionDownLoader - Page 193

JimmXinu · 06-09-2014, 07:45 AM

Quote:

Originally Posted by cryzed

@Jimm
I'm pretty sure that this is still a bug relating to merging the stories though. Why does an adapter need to provide "numWords"? There's a bug somewhere in "merge_meta_books" in the "ffdl_plugin.py" line 1992. Somehow the standard story metadata has a bunch of default keys, like "numWords" and "numChapters" with empty values. In the method it now runs into an error, because it expects both fields to have a valid value just because the keys are contained in the metadata. I think the easiest way would be to check "if book['all_metadata'][k] and b['all_metadata'][k]" before working with the values in the method -- your call! Also attached is a possible fix, it seems to work for me.

Can you provide a couple story URLs that demonstrate the problem? I'm not finding any where that pre-sets numWords to None. numChapters is required. If there's no chapters at all, there's no story.

cryzed · 06-09-2014, 07:51 AM

Choose any two from here: http://nocturnal-light.net/fanfiction/. You'll need to use one of my attached zip files.

And what about the extra metadata include feature? Does that exist currently?

JimmXinu · 06-09-2014, 07:52 AM

Quote:

Originally Posted by Amalthia

I figured out what was missing. I had to go to the tab for "Generate Cover" in the FFDL settings and select the box "Allow generate_cover_settings from personal.ini to override."

I'll add a note about that to defaults.ini in the next version. Sorry for the confusion.

JimmXinu · 06-09-2014, 08:33 AM

Quote:

Originally Posted by cryzed

Choose any two from here: http://nocturnal-light.net/fanfiction/. You'll need to use one of my attached zip files.

And what about the extra metadata include feature? Does that exist currently?

That's what was doing it. When no metadata value is found, story is returning getConfig("default_value_"+key), which defaults to "". I didn't even remember that feature was in there.

Attached is a test version with all of cryzed's committed changes and a fix for the anthology-without-wordcount problem.

JimmXinu · 06-09-2014, 11:00 AM

Quote:

Originally Posted by davidfor

You can use the same test in the replace_metadata more than once. You could try something like:

Code:

replace_metadata:
 fandoms=>Thor \(Movies\)=>Thor
 fandoms=>Thor \(Movies\)=>Marvel Cinematic Universe
 fandoms=>Captain America \(Movies\)=>Captain America 
 fandoms=>Captain America \(Movies\)=>Marvel Cinematic Universe

It doesn't work that way. Each replace line is applied sequentially.

In the case shown above, "Thor (Movies)" is changed into "Thor" and then doesn't match the next 'Thor \(Movies\)' pattern any more.

I expect davidfor is actually using the technique where you add a new metadata item with extra_valid_entries, populate it with a copy of an existing one, then change each separately and merge back together.

Quote:

Originally Posted by davidfor

The thing that I fell is missing for this is being able to supply more that one tag. There have been a couple of times where what I really wanted was to split the found tag into two. The solution was to put two lines in, but it does mean I have two copies of the same match string to maintain if things change.

I've looked at that a bit before and again now. I've found a way that appears to work.

I've arbitrarily chosen '{{,}}' as a separator to indicate to split the replacement string at that point(s) into a list and append the new entries to the end of the list. Because they go to the end, they will in turn have replace_metadata applied.

There's a risk of infinite recursion, that I haven't thought about very deeply yet.

If you're curious, here's my test case:

Spoiler:

JimmXinu · 06-09-2014, 07:39 PM

Quote:

Originally Posted by JimmXinu

I've arbitrarily chosen '{{,}}' as a separator to indicate to split the replacement string at that point(s) into a list and append the new entries to the end of the list. Because they go to the end, they will in turn have replace_metadata applied.

There's a risk of infinite recursion, that I haven't thought about very deeply yet.

If you're curious, here's my test case:

Spoiler:

I've been testing this and while introducing cycles doesn't cause infinite loops, the behavior isn't easily predicted.

But for straightforward cases, it seems to work as intended.

The attached version moves the in/exclude_metadata_post checks after the '{{,}}' splitting.

Amalthia · 06-09-2014, 11:49 PM

Quote:

Originally Posted by JimmXinu

I've been testing this and while introducing cycles doesn't cause infinite loops, the behavior isn't easily predicted.

But for straightforward cases, it seems to work as intended.

The attached version moves the in/exclude_metadata_post checks after the '{{,}}' splitting.

This is my original code:

Code:

replace_metadata:
 fandoms=>Marvel Cinematic Universe=>Marvel
 fandoms=>Thor \(Movies\)=>Thor
 fandoms=>Captain America \(Movies\)=>Captain America
 fandoms=>Captain America \(Movies\){{,}}Thor \(Movies\)=>Marvel

If I'm reading your directions accurately it looks like in order to add "Marvel" to any Thor and Captain America stories that do not list "Marvel Cinematic Universe" as a fandom, I would need to use this code instead:

Code:

replace_metadata:
 fandoms=>Marvel Cinematic Universe=>Marvel
 fandoms=>Thor \(Movies\)=>Thor
 fandoms=>Captain America \(Movies\)=>Captain America
 fandoms=>Thor \(Movies=>\)Thor{{,}}Marvel

I wasn't sure if I was supposed to leave in the word "category" that you had in your example?

Thanks for all your help on this,
Amalthia

JimmXinu · 06-10-2014, 12:13 AM

Quote:

Originally Posted by Amalthia

This is my original code:
...
I wasn't sure if I was supposed to leave in the word "category" that you had in your example?

'category' is what most sites use for the fandom. I believe the term was already in use by FFDL when I joined the project, but it's possible it's my fault.

Hmmm. Checking the code, the AO3 adapter populates both 'category' and 'fandoms' with the same list. So which ever you prefer to use. But 'category' is a "standard" metadata for FFDL and listed on the Custom Column tab etc.

You were trying to use {{,}} backwards. It goes in the output to indicate more than one list entry, not the pattern.

If I understand correctly, this will do what you want: change 'Marvel Cinematic Universe' to just 'Marvel' remove the ' (Movies)' part from Captain America and Thor, and make sure that Captain America and Thor movies also have 'Marvel'.

Code:

replace_metadata:
 fandoms=>Marvel Cinematic Universe=>Marvel
 fandoms=>Captain America \(Movies\)=>Captain America{{,}}Marvel
 fandoms=>Thor \(Movies\)=>Thor{{,}}Marvel

A more regexp version that also matches "The Avengers (Marvel Movies)" would be:

Code:

replace_metadata:
 fandoms=>Marvel Cinematic Universe=>Marvel
 fandoms=>(Captain America|Thor|The Avengers) \((Marvel )?Movies\)=>\1{{,}}Marvel

But one line per movie would be more readable if you're not a regexp lover.

Amalthia · 06-10-2014, 12:21 AM

Quote:

Originally Posted by JimmXinu

'category' is what most sites use for the fandom. I believe the term was already in use by FFDL when I joined the project, but it's possible it's my fault.

Code:

replace_metadata:
 fandoms=>Marvel Cinematic Universe=>Marvel
 fandoms=>Captain America \(Movies\)=>Captain America{{,}}Marvel
 fandoms=>Thor \(Movies\)=>Thor{{,}}Marvel

A more regexp version that also matches "The Avengers (Marvel Movies)" would be:

Code:

replace_metadata:
 fandoms=>Marvel Cinematic Universe=>Marvel
 fandoms=>(Captain America|Thor|The Avengers) \((Marvel )?Movies\)=>\1{{,}}Marvel

But one line per movie would be more readable if you're not a regexp lover.

This worked!

Thank you so much. Thanks for pointing out what I was doing wrong.

Best Regards,
Amalthia

davidfor · 06-10-2014, 12:45 AM

Quote:

Originally Posted by JimmXinu

It doesn't work that way. Each replace line is applied sequentially.

In the case shown above, "Thor (Movies)" is changed into "Thor" and then doesn't match the next 'Thor \(Movies\)' pattern any more.

I expect davidfor is actually using the technique where you add a new metadata item with extra_valid_entries, populate it with a copy of an existing one, then change each separately and merge back together.

I wish I could say I was being that clever, but I wasn't. I had assumed that it would test against each line. I just checked a couple of stories I use for test cases and of course it doesn't work. My test cases had combinations of tags that meant it looked like it was working, but wasn't really.

Quote:

I've looked at that a bit before and again now. I've found a way that appears to work.

I've arbitrarily chosen '{{,}}' as a separator to indicate to split the replacement string at that point(s) into a list and append the new entries to the end of the list. Because they go to the end, they will in turn have replace_metadata applied.

There's a risk of infinite recursion, that I haven't thought about very deeply yet.

If you're curious, here's my test case:

Spoiler:

Is that a typo? It looks like you have the "=>" in the wrong place. In any case, when I put it in the place I thought was correct, it worked. And worked well.

cryzed · 06-10-2014, 08:55 AM

@matteo65
Attached is a test version of the plugin with an adapter for http://fanfiction.csodaidok.hu/. Please let me know how it performs.

JimmXinu · 06-10-2014, 10:07 AM

Quote:

Originally Posted by davidfor

I wish I could say I was being that clever, but I wasn't. I had assumed that it would test against each line. I just checked a couple of stories I use for test cases and of course it doesn't work. My test cases had combinations of tags that meant it looked like it was working, but wasn't really.

I had to spend some time testing first to be sure I was right--and I wrote the darn thing.

Quote:

Originally Posted by davidfor

Is that a typo? It looks like you have the "=>" in the wrong place. In any case, when I put it in the place I thought was correct, it worked. And worked well.

Yes, that's a typo. My bad.

cryzed · 06-10-2014, 10:11 AM

Jimm, since you are already using "\s" for a "forced space", wouldn't it make sense to simply use "\," instead of "{{,}}"? Just a suggestion.

matteo65 · 06-10-2014, 02:56 PM

@cryzed

Thank you, its working great with most of the books.
Little request to set the language tag to Hungarian instead of English.

But I got an error message downloading this story:
http://fanfiction.csodaidok.hu/viewstory.php?sid=4146

calibre, version 1.39.0
HIBA: Ismeretlen hiba történt: <b>AttributeError</b>:'NoneType' object has no attribute 'replace'

calibre 1.39 [64bit] isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.5
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 607, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 614, in do_when_finished
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 1130, in start_download_job
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 2062, in escapehtml
AttributeError: 'NoneType' object has no attribute 'replace'

Quote:

Originally Posted by cryzed

@matteo65
Attached is a test version of the plugin with an adapter for http://fanfiction.csodaidok.hu/. Please let me know how it performs.

matteo65 · 06-10-2014, 03:07 PM

@cryzed

In the Summary I see some special Hungarian characters are missing and replaced by '?' marks. Maybe a character encoding issue?
The problematic characters are ő,Ő,ű,Ű

The summary as it looks now:
Mi történik, ha négy ?rültet szabadítanak rá a Roxfort mit sem sejt? népére?

And the correct version:
Mi történik, ha négy őrültet szabadítanak rá a Roxfort mit sem sejtő népére?

06-09-2014, 07:51 AM	#2882
cryzed Evangelist Posts: 408 Karma: 1050547 Join Date: Mar 2011 Device: Kindle Oasis 2	Choose any two from here: http://nocturnal-light.net/fanfiction/. You'll need to use one of my attached zip files. And what about the extra metadata include feature? Does that exist currently? Last edited by cryzed; 06-09-2014 at 07:54 AM.

06-10-2014, 08:55 AM	#2891
cryzed Evangelist Posts: 408 Karma: 1050547 Join Date: Mar 2011 Device: Kindle Oasis 2	@matteo65 Attached is a test version of the plugin with an adapter for http://fanfiction.csodaidok.hu/. Please let me know how it performs. Last edited by cryzed; 06-10-2014 at 04:08 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Count Pages	kiwidude	Plugins	1846	08-02-2025 10:44 PM
[GUI Plugin] Resize Cover	kiwidude	Plugins	100	07-10-2025 08:50 AM
[GUI Plugin] Find Duplicates	kiwidude	Plugins	1124	04-18-2025 09:19 AM
[GUI Plugin] Open With	kiwidude	Plugins	404	02-21-2025 05:42 AM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

06-10-2014, 10:11 AM	#2893
cryzed Evangelist Posts: 408 Karma: 1050547 Join Date: Mar 2011 Device: Kindle Oasis 2	Jimm, since you are already using "\s" for a "forced space", wouldn't it make sense to simply use "\," instead of "{{,}}"? Just a suggestion.

06-10-2014, 03:07 PM	#2895
matteo65 Member Posts: 13 Karma: 10 Join Date: May 2013 Location: Hungary Device: Kindle 3, Paperwhite	@cryzed In the Summary I see some special Hungarian characters are missing and replaced by '?' marks. Maybe a character encoding issue? The problematic characters are ő,Ő,ű,Ű The summary as it looks now: Mi történik, ha négy ?rültet szabadítanak rá a Roxfort mit sem sejt? népére? And the correct version: Mi történik, ha négy őrültet szabadítanak rá a Roxfort mit sem sejtő népére?