[GUI Plugin] FanFictionDownLoader - Page 201

rlh3 · 06-19-2014, 04:41 PM

Quote:

Originally Posted by davidfor

It's working here as well.

On a related note, the literotica pages have tags in the "keywords" of the page. You can see them in the source of the page, or Firefox will show them in the page info (right click on the page and select "View page info). At some time in the past I wrote some code to extract these and use them as tags. If anyone is interested, I can dig this out and add it. With an option to use or not use them.

Might be useful. I have not really looked a the tags that Literotica uses, but the option of more information is always a good thing.

JimmXinu · 06-19-2014, 07:21 PM

Quote:

Originally Posted by davidfor

On a related note, the literotica pages have tags in the "keywords" of the page. You can see them in the source of the page, or Firefox will show them in the page info (right click on the page and select "View page info). At some time in the past I wrote some code to extract these and use them as tags. If anyone is interested, I can dig this out and add it. With an option to use or not use them.

I don't read that site, so it doesn't make a lot of difference to me either way.

More metadata is generally good. But it also depends on how hard it is to get. I'd probably want to add them as a separate site-specific metadata unless there was a clear correlation with one of the existing metadata slots.

I'm not finding any 'keywords' doing some quick perusing of the site. Can you provide a link to a couple examples?

davidfor · 06-19-2014, 10:14 PM

Quote:

Originally Posted by rlh3

Might be useful. I have not really looked a the tags that Literotica uses, but the option of more information is always a good thing.

Quote:

Originally Posted by JimmXinu

More metadata is generally good.

I tend to be a "more is better" person as well.

Quote:

But it also depends on how hard it is to get.

They are easy to get:

Code:

soup.find("meta", {"name":"keywords"})['content'].split(', ')

The complications are that they usually include the story and author name. My code strips these out, but doesn't handle the case where the name has a comma in it.

Quote:

I'd probably want to add them as a separate site-specific metadata unless there was a clear correlation with one of the existing metadata slots.

I have just been putting them in the "category" list and they end up in the tags.

Quote:

I'm not finding any 'keywords' doing some quick perusing of the site. Can you provide a link to a couple examples?

As far as I can tell all the stories have them. All the stories on the author page from the other day do. That is: http://www.literotica.com/stories/me...ge=submissions

And I'm embarrassed to say that I have never noticed that at the end of the story there is a "Tags for this story" section. That has the same tags as in the keywords meta field. The keywords is a easier to handle and is available on the first page of a multi-page story.

JimmXinu · 06-19-2014, 10:51 PM

Quote:

Originally Posted by davidfor

And I'm embarrassed to say that I have never noticed that at the end of the story there is a "Tags for this story" section. That has the same tags as in the keywords meta field. The keywords is a easier to handle and is available on the first page of a multi-page story.

Well, I still don't see the keywords. I wonder if it's a skin issue.

But I found parsing the 'tags' to be very easy. Attached is a version that adds the site specific metadatum 'eroticatags' for literotica.com only.

davidfor · 06-19-2014, 11:53 PM

Quote:

Originally Posted by JimmXinu

Well, I still don't see the keywords. I wonder if it's a skin issue.

They aren't displayed. They are in the <head> section of the source as a attributes of a meta tag. Firefox will display this when you show the page info.

Quote:

But I found parsing the 'tags' to be very easy. Attached is a version that adds the site specific metadatum 'eroticatags' for literotica.com only.

That only works for a single page story. These tags are on the last page. See http://www.literotica.com/s/chandra as an example.

The full code I have is:

Code:

categories = soup.find("meta", {"name":"keywords"})['content'].split(', ')
categories = [c for c in categories if not self.story.getMetadata('title') in c]
if self.story.getMetadata('author') in categories:
    categories.remove(self.story.getMetadata('author'))
for category in categories:
    self.story.addToList('category', category.title())

That's a little bit clumsy and I should clean it up a bit.

davidfor · 06-20-2014, 01:38 AM

With the discussion on the Literotica adapter, I figured I should pull out the full version I wrote some time ago. I have been keeping it up to date. I sometimes replace the supplied adapter with this if I am getting a multi-chapter story.

Apart from using the keywords as we've been discussing, the main difference is how it handles chapters. Rather than just use the title of that part, it will use "Chapter nn" or "Part nn". It does this by parsing the chapter title against the story title and working out the number and whether to use "Chapter" or "Part". It also adds the chapter description at the top of each chapter. There are a couple of other changes to the code generated.

Overall, I think it produces a better looking epub. But, maybe others don't agree. If others do like it, I'll clean up the code and finally learn how to use the version control system that FFDL uses.

JimmXinu · 06-20-2014, 11:36 AM

Quote:

Originally Posted by davidfor

With the discussion on the Literotica adapter, I figured I should pull out the full version I wrote some time ago. I have been keeping it up to date.
...
Overall, I think it produces a better looking epub. But, maybe others don't agree. If others do like it, I'll clean up the code and finally learn how to use the version control system that FFDL uses.

I don't read that site and I didn't write the existing adapter, so I don't really care either way. If your adapter works better, I'd be happy to switch to it.

JimmXinu · 06-21-2014, 11:08 AM

Version 1.8.25 - 21 Jun 2014

New site: fictionmania.tv -- Thanks, cryzed!
Fix for site recognizer to handle with/without www. using https URLs.
Fix for some utf8 descriptions on fimf.
Site specific metadata 'eroticatags' for literotica.com.
Fix for 'a' flag on custom_columns_settings not working as intended.
Changes to '\,' to better implement split list feature to avoid infinite recursions.
Fix for calibre-injected series--don't treat series as a list, it isn't.
Auto-add http: to URLs starting with //.
Switch to transifex.com for translations.

JimmXinu · 06-21-2014, 11:25 AM

With Kovid's gracious help, I've switched over to using Transifex for translations for my plugins.

You can now help out by providing translations without having to install anything. All you need is to create an account on a web site and jump in to translating. The link to FFDL's translations specifically is here.

Thanks!

megme · 06-21-2014, 03:26 PM

Hi Jimm
I can't d/l any stories from fictionpad, here are some I tried today
https://fictionpad.com/author/StillD...504/Relentless
https://fictionpad.com/author/Dishie...-Sex-is-Divine
https://fictionpad.com/author/LovinR.../A-Bitter-Cold
https://fictionpad.com/author/tufano...A-Ilha-do-Amor

I get this message with all stories
http://www.solidfiles.com/d/f4174a09f6/fictionpad.jpg

Thanks for your help.

JimmXinu · 06-21-2014, 03:37 PM

Quote:

Originally Posted by megme

Hi Jimm
I can't d/l any stories from fictionpad, here are some I tried today
https://fictionpad.com/author/StillD...504/Relentless
https://fictionpad.com/author/Dishie...-Sex-is-Divine
https://fictionpad.com/author/LovinR.../A-Bitter-Cold
https://fictionpad.com/author/tufano...A-Ilha-do-Amor

I get this message with all stories
http://www.solidfiles.com/d/f4174a09f6/fictionpad.jpg

Thanks for your help.

The fictionpad.com adapter is unusual in that it access an API to get part of its data. Consequently, it actually enters int values into metadata, which the new metadata processing didn't like.

Attached is a version that force metadata values to be strings. It seems to work better.

FYI, I don't know what's wrong with your IMG tag, but I don't see it in the post, only in the reply text as a tag.

thecookiemommma · 06-21-2014, 09:44 PM

I don't know if this is FF.net or the new version, but I went to update a story I'd downloaded just two days ago, because I saw that it'd been updated, and got a message saying:
Bad
Unknown
Unknown
'NoneType' object is not iterable
https://www.fanfiction.net/s/9859066/1/

It's not reading the words after the /, I think? It should say: https://www.fanfiction.net/s/9859066...lt-in-the-Silt

JimmXinu · 06-21-2014, 10:15 PM

Quote:

Originally Posted by thecookiemommma

I don't know if this is FF.net or the new version, but I went to update a story I'd downloaded just two days ago, because I saw that it'd been updated, and got a message saying:
Bad
Unknown
Unknown
'NoneType' object is not iterable
https://www.fanfiction.net/s/9859066/1/

It's not reading the words after the /, I think? It should say: https://www.fanfiction.net/s/9859066...lt-in-the-Silt

That story downloads fine for me.

Can you run calibre in debug and post the output? Calibre now has an easy way to do that under the Preferences menu.

FYI, the title in the URL is superfluous and isn't saved.

JimmXinu · 06-21-2014, 11:06 PM

Quote:

Originally Posted by JimmXinu

That story downloads fine for me.

Can you run calibre in debug and post the output? Calibre now has an easy way to do that under the Preferences menu.

FYI, the title in the URL is superfluous and isn't saved.

Make sure you grab the latest test version posted above if you haven't. That's what I'm testing againt.

thecookiemommma · 06-21-2014, 11:50 PM

Ok, here's the file from it.

06-21-2014, 11:08 AM	#3008
JimmXinu Plugin Developer Posts: 6,988 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	New Version Version 1.8.25 - 21 Jun 2014 New site: fictionmania.tv -- Thanks, cryzed! Fix for site recognizer to handle with/without www. using https URLs. Fix for some utf8 descriptions on fimf. Site specific metadata 'eroticatags' for literotica.com. Fix for 'a' flag on custom_columns_settings not working as intended. Changes to '\,' to better implement split list feature to avoid infinite recursions. Fix for calibre-injected series--don't treat series as a list, it isn't. Auto-add http: to URLs starting with //. Switch to transifex.com for translations.

06-21-2014, 11:25 AM	#3009
JimmXinu Plugin Developer Posts: 6,988 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	Translations With Kovid's gracious help, I've switched over to using Transifex for translations for my plugins. You can now help out by providing translations without having to install anything. All you need is to create an account on a web site and jump in to translating. The link to FFDL's translations specifically is here. Thanks!

06-21-2014, 03:26 PM	#3010
megme Enthusiast Posts: 30 Karma: 100000 Join Date: Dec 2013 Device: Samsung	Hi Jimm I can't d/l any stories from fictionpad, here are some I tried today https://fictionpad.com/author/StillD...504/Relentless https://fictionpad.com/author/Dishie...-Sex-is-Divine https://fictionpad.com/author/LovinR.../A-Bitter-Cold https://fictionpad.com/author/tufano...A-Ilha-do-Amor I get this message with all stories http://www.solidfiles.com/d/f4174a09f6/fictionpad.jpg Thanks for your help. Last edited by megme; 06-21-2014 at 03:30 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Count Pages	kiwidude	Plugins	1846	08-02-2025 10:44 PM
[GUI Plugin] Resize Cover	kiwidude	Plugins	100	07-10-2025 08:50 AM
[GUI Plugin] Find Duplicates	kiwidude	Plugins	1124	04-18-2025 09:19 AM
[GUI Plugin] Open With	kiwidude	Plugins	404	02-21-2025 05:42 AM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

06-21-2014, 09:44 PM	#3012
thecookiemommma Enthusiast Posts: 30 Karma: 7286 Join Date: Mar 2011 Device: ZTE Grand Max + / HP Slate 7 HD	I don't know if this is FF.net or the new version, but I went to update a story I'd downloaded just two days ago, because I saw that it'd been updated, and got a message saying: Bad Unknown Unknown 'NoneType' object is not iterable https://www.fanfiction.net/s/9859066/1/ It's not reading the words after the /, I think? It should say: https://www.fanfiction.net/s/9859066...lt-in-the-Silt