[GUI Plugin] FanFictionDownLoader - Page 155

davidfor · 12-29-2013, 07:29 PM

Quote:

Originally Posted by JimmXinu

It's saddeningly common for sites to report one encoding while using the other more commonly. Or to use both on different stories.

I don't read storiesonline.net. If someone who does tells me that utf8 is correct more often than Windows-1252, I'm willing to change it.

A quick random click around and all are UTF-8. I would expect only the one encoding as all pages on the site is generated. And whatever is generating it is custom written for this and the related sites.

Quote:

I don't have the energy to look at this right now either. Maybe tomorrow.

Don't worry to much. I'll probably get bored in the next few days and have a go. If not, definitely sometime next week.

cryzed · 12-30-2013, 12:29 AM

This might also be interesting for corner-cases: In case the decoding fails, one could fallback to the chardet module, and try to detect the real encoding. And even if then there are still errors or the encoding couldn't be detected properly, just do a .decode(..., 'ignore'/'replace') possibly instead of crashing? I'm not sure what the module currently does.

Oilily Sepperl · 12-30-2013, 05:29 AM

Quote:

Originally Posted by JimmXinu

If you're trying to update that many stories at once, you're probably running into throttling on the server. Especially if they are fanfiction.net stories.

I've written before about why doing mass downloads/updates is a bad idea.

Thank you so much for your answer, JimmXinu. Should have searched first, but I didn't know which words to use.
I assumed something like that already - was surprised though, because it functioned quite a while for me -, so I'll simply change my personal updating system.
Thank you again!

davidfor · 12-30-2013, 06:10 AM

I have fixed the problems noted in the last few days plus a couple of other things. These are:

- Site uses UTF-8. As the base adapter uses this first, I have removed the setting from this adapter.

- If the story was in a series or universe, the summary was just that text and didn't include the story description.

- While looking at the above, I realised that when a story was in a universe, that was being lost. So, when there is a universe, the series is set to the universe. One problem with this is that with a universe, there isn't an order, so the series index is set to zero.

The other thing I mentioned the other day was the notice on some story index pages. This can be a disclaimer or some sort or some setup for the story. http://storiesonline.net/s/51245 is the story that started this round of changes, and it has a notice. Extracting the notice was easy. But, then I couldn't quickly work out how to display it in the story. It probably should be added after the story details, but, I don't immediately see how to do that. I have put the extract code in. If someone can tell me where to look, I can add it to the actual story.

Also, the finestories.com should have the change for the character set. I had a look at a few pages and they are being generated as UTF-8. I think it should have the universe/series change as well, but I didn't have a close look at that.

JimmXinu · 12-30-2013, 08:52 PM

Quote:

Originally Posted by davidfor

I have fixed the problems noted in the last few days plus a couple of other things. These are:

- Site uses UTF-8. As the base adapter uses this first, I have removed the setting from this adapter.
...

Okay.

Quote:

Originally Posted by davidfor

- While looking at the above, I realised that when a story was in a universe, that was being lost. So, when there is a universe, the series is set to the universe. One problem with this is that with a universe, there isn't an order, so the series index is set to zero.

That's not a good solution. I've changed it to use the site specific metadata universe and universeUrl. The datum universeHTML will be automatically created. I haven't tested it because I don't have any story URLs with universe. You'll need to add universeHTML to extra_titlepage_entries to see it.

Quote:

Originally Posted by davidfor

The other thing I mentioned the other day was the notice on some story index pages. This can be a disclaimer or some sort or some setup for the story. http://storiesonline.net/s/51245 is the story that started this round of changes, and it has a notice. Extracting the notice was easy. But, then I couldn't quickly work out how to display it in the story. It probably should be added after the story details, but, I don't immediately see how to do that. I have put the extract code in. If someone can tell me where to look, I can add it to the actual story.

Added in another new site specific metadatum, notice.

Quote:

Originally Posted by davidfor

Also, the finestories.com should have the change for the character set. I had a look at a few pages and they are being generated as UTF-8. I think it should have the universe/series change as well, but I didn't have a close look at that.

I've changed the encoding, but I'm not going to add the universe code with an example story URL to test against.

Attached is a test version with the changes discussed.

UPDATE Jan 4, 2013 - Remove obsolete beta versions

JimmXinu · 12-30-2013, 09:05 PM

Quote:

Originally Posted by cryzed

This might also be interesting for corner-cases: In case the decoding fails, one could fallback to the chardet module, and try to detect the real encoding. And even if then there are still errors or the encoding couldn't be detected properly, just do a .decode(..., 'ignore'/'replace') possibly instead of crashing? I'm not sure what the module currently does.

There's already a copy of chardet in FFDL--it can be invoked by putting 'auto' in the website_encodings parameter:

Code:

website_encodings: auto, utf8, Windows-1252

When I added it, chardet did a very poor job of differentiating utf8 from iso8859-1 or windows-1252. So while I left it in, I didn't use it much.

I should either update it so it works better or remove it outright, I suppose.

davidfor · 12-31-2013, 05:54 AM

Quote:

Originally Posted by JimmXinu

That's not a good solution. I've changed it to use the site specific metadata universe and universeUrl. The datum universeHTML will be automatically created. I haven't tested it because I don't have any story URLs with universe. You'll need to add universeHTML to extra_titlepage_entries to see it.

The story that started all this, http://storiesonline.net/s/51245, is in a universe, but it 42 chapters. I had a look through the list of universes and found the the "BOFH" universe - BOFH fanfic, so it is apt for this plugin and proof that there is fanfic for everything. One of the stories is http://storiesonline.net/s/64407.

I can see why, but, I'm not sure I agree. The site has both universes and series (and series can be ordered or be unordered). Universes can hold individual stories or series. But, there is no way to tell from a story in a series that the series is part of a universe. That means a story will only have one. For me, I suppose I want to get whichever the story has into the series column. Can I do this through the preferences or personal.ini?

For curiosity, I had a look at what is in one of the site generated epubs. For the BOFH story, http://storiesonline.net/s/64407, the OPF has:

Code:

<meta name="calibre:series" content="BOFH"/>

For a book that is in a series, it also has the series index. So, the site is at least partly treating the universe the same as a series.

Quote:

Added in another new site specific metadatum, notice.

Using http://storiesonline.net/s/51245 (which is also in a universe), I had an error that seems to be related to this. The traceback is:

Code:

Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.dialogs", line 590, in do_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1068, in prep_download_loop
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.story", line 542, in formatFileName
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.story", line 419, in getAllMetadata
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.story", line 330, in getMetadata
TypeError: sequence item 0: expected string or Unicode, Tag found

Looking at the code, the value for the notice needed to be converted to a string. That was my mistake.

Quote:

I've changed the encoding, but I'm not going to add the universe code with an example story URL to test against.

That's fair enough. I'll have a look and see if I can find something suitable for testing.

JimmXinu · 01-02-2014, 04:30 PM

davidfor,

So in summary, a storiesonline.net story can be a member of:

an ordered series;
an unordered series, or;
a universe.

Additionally, a series (ordered or not) can also be a member of a universe.

And you'd like 'universe' to be populated into the standard 'series' metadatum when a story is a direct member of a universe, since it can't be a member of a series then.

I suppose that's not unreasonable, but I'd also like to be able to populate 'universe' for stories that are members of series that are in turn members of a universe.

Have you found example stories of each of these cases? I haven't found any unordered series or series that are a member of a universe. I suspect the existing code will break on an unordered series.

Quote:

Originally Posted by davidfor

...
Looking at the code, the value for the notice needed to be converted to a string. That was my mistake.

Fixed for next version.

TrulyInnovative · 01-02-2014, 11:08 PM

Quote:

Originally Posted by JimmXinu

Have you found example stories of each of these cases? I haven't found any unordered series or series that are a member of a universe. I suspect the existing code will break on an unordered series.

Here are examples of series that are members of a universe:
http://storiesonline.net/library/universe.php?id=4
http://storiesonline.net/library/universe.php?id=659

Here are unordered series (presumably called collections):
http://storiesonline.net/library/show_series.php?id=177
http://storiesonline.net/library/show_series.php?id=604

Incidentally, it's possible that authors make a mistake in numbering a series. This series has three books all marked #1 and one book marked #2 in the series:
http://storiesonline.net/library/show_series.php?id=113

davidfor · 01-03-2014, 01:47 AM

Quote:

Originally Posted by JimmXinu

davidfor,

So in summary, a storiesonline.net story can be a member of:

an ordered series;
an unordered series, or;
a universe.

Additionally, a series (ordered or not) can also be a member of a universe.

Yes to all of that.

Quote:

And you'd like 'universe' to be populated into the standard 'series' metadatum when a story is a direct member of a universe, since it can't be a member of a series then.

I suppose that's not unreasonable, but I'd also like to be able to populate 'universe' for stories that are members of series that are in turn members of a universe.

Yes. I think the lowest level of universe/series membership should be in the series column. Populating the universe would also be good, but, there isn't a direct link. From what I can see, to get the universe you need to go back to the author page, choose the "Universes" link and then see if the story or the series it belongs to is there.

Actually that might not be that bad. The authors universe page has "&type=uni" added to the authors URL. Then it is simply a matter of finding an a tag that refers to the story page. I'll have a play later.

Quote:

Have you found example stories of each of these cases? I haven't found any unordered series or series that are a member of a universe. I suspect the existing code will break on an unordered series.

TrulyInnovative posted examples, but here are a couple that I found while clicking through the universes page. http://storiesonline.net/library/universe.php?id=19 is a universe with a series. The series looks to be ordered. And http://storiesonline.net/library/universe.php?id=377 has a series that doesn't seem to be ordered. Though from the names it should have been.

The unordered series is broken. I just tested to from stories from the above. The series name and link is put into the description and the description is lost. I'll have a play with that as well.

davidfor · 01-03-2014, 03:11 AM

Quote:

Originally Posted by davidfor

The unordered series is broken. I just tested to from stories from the above. The series name and link is put into the description and the description is lost. I'll have a play with that as well.

The following seems to work:

Code:

        try:
            a = lc4.find('a', href=re.compile(r"/library/show_series.php\?id=\d+"))
            if a:
                series_contents = a.parent.contents
                i = 0 if len(series_contents) == 1 else series_contents[1].strip(' ()')
                self.setSeries(stripHTML(a), i)
                self.story.setMetadata('seriesUrl','http://'+self.host+a['href'])
                desc = lc4.contents[2]
        except:
            pass

For unordered series, I'm not sure what the best series index is. Zero seems to be the safest. But, maybe it should be 1. I'm tempted to use zero for universes and 1 for series as they can be inside a universe. But, that might be complicating things.

aleyx · 01-03-2014, 05:21 AM

Maybe taking advantage of hierarchical series?

series = Series Name [0] (no universe, unordered series)
series = Series Name [n] (no universe, ordered series)
series = Universe [0] (universe, no series)
series = Universe.Series Name [0] (universe, unordered series)
series = Universe.Series Name [n] (universe, ordered series)

davidfor · 01-03-2014, 05:42 AM

That does make sense. I'm not sure if I am a fan of the "Universe.Series", but I can see that some people would like it. Maybe an option in the personal.ini?

aleyx · 01-03-2014, 05:58 AM

Well, it's the separator character for hierarchical items in the tag browser; I don't believe it's configurable. Obviously if the tag browser is not a consideration (along with all the search- and filter-related features), any string would do.

davidfor · 01-03-2014, 06:11 AM

No, I meant configure what gets put into the series field if the book is in series that is in a universe. I think I would be happy with just the series, but I can understand of someone wanted both.

12-30-2013, 12:29 AM	#2312
cryzed Evangelist Posts: 408 Karma: 1050547 Join Date: Mar 2011 Device: Kindle Oasis 2	This might also be interesting for corner-cases: In case the decoding fails, one could fallback to the chardet module, and try to detect the real encoding. And even if then there are still errors or the encoding couldn't be detected properly, just do a .decode(..., 'ignore'/'replace') possibly instead of crashing? I'm not sure what the module currently does. Last edited by cryzed; 12-30-2013 at 12:47 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Count Pages	kiwidude	Plugins	1849	08-30-2025 01:22 PM
[GUI Plugin] Resize Cover	kiwidude	Plugins	100	07-10-2025 08:50 AM
[GUI Plugin] Find Duplicates	kiwidude	Plugins	1124	04-18-2025 09:19 AM
[GUI Plugin] Open With	kiwidude	Plugins	404	02-21-2025 05:42 AM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

01-03-2014, 05:21 AM	#2322
aleyx Addict Posts: 250 Karma: 20386 Join Date: Sep 2010 Location: France Device: Bookeen Diva, Kobo Clara BW	Maybe taking advantage of hierarchical series? series = Series Name [0] (no universe, unordered series) series = Series Name [n] (no universe, ordered series) series = Universe [0] (universe, no series) series = Universe.Series Name [0] (universe, unordered series) series = Universe.Series Name [n] (universe, ordered series)

01-03-2014, 05:42 AM	#2323
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	That does make sense. I'm not sure if I am a fan of the "Universe.Series", but I can see that some people would like it. Maybe an option in the personal.ini?

01-03-2014, 05:58 AM	#2324
aleyx Addict Posts: 250 Karma: 20386 Join Date: Sep 2010 Location: France Device: Bookeen Diva, Kobo Clara BW	Well, it's the separator character for hierarchical items in the tag browser; I don't believe it's configurable. Obviously if the tag browser is not a consideration (along with all the search- and filter-related features), any string would do.

01-03-2014, 06:11 AM	#2325
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	No, I meant configure what gets put into the series field if the book is in series that is in a universe. I think I would be happy with just the series, but I can understand of someone wanted both.

Advert

Advert