Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 09-09-2013, 03:43 PM   #1846
Firedancer885
Occassional Beta Tester
Firedancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheese
 
Posts: 216
Karma: 1162
Join Date: Nov 2010
Location: Hungary
Device: Samsung Galaxy Tab
Quote:
Originally Posted by CyberCrisis View Post
Anybody having login problem with www.tthfanfic.org.
In FFDL? I had. Read back a few posts, Jimm already solved it.
Firedancer885 is offline   Reply With Quote
Old 09-09-2013, 03:49 PM   #1847
Firedancer885
Occassional Beta Tester
Firedancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheeseFiredancer885 can extract oil from cheese
 
Posts: 216
Karma: 1162
Join Date: Nov 2010
Location: Hungary
Device: Samsung Galaxy Tab
Quote:
Originally Posted by JimmXinu View Post
Okay, in that case, make it just:
Code:
[www.tthfanfic.org]
user_agent:FFDL/1.7
Thanks
Firedancer885 is offline   Reply With Quote
Old 09-10-2013, 03:59 AM   #1848
Feynor
Enthusiast
Feynor began at the beginning.
 
Posts: 26
Karma: 10
Join Date: Sep 2013
Device: Samsung Galaxy Tab 10.1 with Moon Reader+ and Google Books.
If only these sites would make good use of the HEAD command, it could cut down on their load tremendously.
The standard response header does have a Last-Modified field, for instance:
Last-Modified: Tue, 10 Sep 2013 07:43:03 GMT

However on FFNET and TTH, this seems to always be today's date, rendering it useless.

Are there any way to "suggest" that they start using the response header to provide simple metadata?

Why would that help them?
By using the HEAD command, the server "only" have to look up a few things about the given story id, it doesn't have to load all the data for the page, and render the HTML code or send the data, taxing their bandwidth.

Why would that help us?
Speed, and easier decoding. Pure and simple.

Last edited by Feynor; 09-10-2013 at 04:16 AM.
Feynor is offline   Reply With Quote
Old 09-10-2013, 04:35 AM   #1849
Sefiriot
Bujavid tekikin
Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.
 
Sefiriot's Avatar
 
Posts: 137
Karma: 40366
Join Date: Aug 2013
Location: Malaysia
Device: Samsung Galaxy Note N7000
Quote:
Originally Posted by Feynor View Post
If only these sites would make good use of the HEAD command, it could cut down on their load tremendously.
The standard response header does have a Last-Modified field, for instance:
Last-Modified: Tue, 10 Sep 2013 07:43:03 GMT

However on FFNET and TTH, this seems to always be today's date, rendering it useless.

Are there any way to "suggest" that they start using the response header to provide simple metadata?

Why would that help them?
By using the HEAD command, the server "only" have to look up a few things about the given story id, it doesn't have to load all the data for the page, and render the HTML code or send the data, taxing their bandwidth.

Why would that help us?
Speed, and easier decoding. Pure and simple.
TTH might listen, no idea about FFN. Then again since they're currently trying to upgrade their network, this might be of use to their current interests. They've got a Twitter you can suggest this to.
Sefiriot is offline   Reply With Quote
Old 09-10-2013, 04:47 AM   #1850
Feynor
Enthusiast
Feynor began at the beginning.
 
Posts: 26
Karma: 10
Join Date: Sep 2013
Device: Samsung Galaxy Tab 10.1 with Moon Reader+ and Google Books.
Quote:
Originally Posted by Sefiriot View Post
TTH might listen, no idea about FFN. Then again since they're currently trying to upgrade their network, this might be of use to their current interests. They've got a Twitter you can suggest this to.
Added the request to TtH: http://forum.tthfanfic.org/index.php?topic=8289.0
All I could find for FFN was their support email at http://www.fanfiction.net/support/
Twitter messages are not quite long enough to convey any meaningful technical message :P

Last edited by Feynor; 09-10-2013 at 04:54 AM.
Feynor is offline   Reply With Quote
Old 09-10-2013, 05:46 AM   #1851
Sefiriot
Bujavid tekikin
Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.
 
Sefiriot's Avatar
 
Posts: 137
Karma: 40366
Join Date: Aug 2013
Location: Malaysia
Device: Samsung Galaxy Note N7000
Quote:
Originally Posted by Feynor View Post
Added the request to TtH: http://forum.tthfanfic.org/index.php?topic=8289.0
All I could find for FFN was their support email at http://www.fanfiction.net/support/
Twitter messages are not quite long enough to convey any meaningful technical message :P
I forgot to mention that they've got a blog as well, and people have been leaving feedback there which the site devs actually respond to: http://blog.fictionpress.com. You might get a faster response there as opposed to emails. Their email inbox is pretty much swamped all the time.
Sefiriot is offline   Reply With Quote
Old 09-10-2013, 05:54 AM   #1852
Feynor
Enthusiast
Feynor began at the beginning.
 
Posts: 26
Karma: 10
Join Date: Sep 2013
Device: Samsung Galaxy Tab 10.1 with Moon Reader+ and Google Books.
Quote:
Originally Posted by Sefiriot View Post
I forgot to mention that they've got a blog as well, and people have been leaving feedback there which the site devs actually respond to: http://blog.fictionpress.com. You might get a faster response there as opposed to emails. Their email inbox is pretty much swamped all the time.
Thanks.
Feynor is offline   Reply With Quote
Old 09-10-2013, 09:26 AM   #1853
cklls
Zealot
cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.
 
Posts: 139
Karma: 1030058
Join Date: Jul 2010
Location: New England
Device: Nook, EB1150, Pandigital SuperNova Tablet, Samsung Galaxy Tab 2 10.1
Quote:
Originally Posted by Sefiriot View Post
TTH might listen, no idea about FFN. Then again since they're currently trying to upgrade their network, this might be of use to their current interests. They've got a Twitter you can suggest this to.
Yes, and they actually replied to my request to consider adding their own epub download and compilation function with questions about what I would like to see. It's clear that they've never given this any thought while their competitors are, at the very least, allowing compilation for printing that allows saving as a full document. Many sites, AO3 and harrypotterfanfiction.com to name just two that I use, have added the ability to download as epub or send directly to a Kindle. FFN is missing the boat and is far behind other fan-sites on this.

BTW, I suggested that they google ffdl plugin to understand what I was requesting. That might get them to recognize that we're not attacking them, just trying to access their product in a way that they've not accommodated.
cklls is offline   Reply With Quote
Old 09-10-2013, 12:03 PM   #1854
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,588
Karma: 411867
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by FaceDeer View Post
...
So I went into adapter_fimfictionnet.py and figured out how to fix my own problem so it wouldn't be a hassle. I felt kind of embarrassed about how my previous bug report turned out to be a bug with Calibre itself rather than FFDL.
...
Looks like I got to it first--attached is a version that incorporates my version of your changes.

I've included a diff so you can see how I did it. I did use dateutil.parser like you did (stupid 2nd/3rd/4th issue), but I also used the same loop for publishedDate and updatedDate.

Spoiler:
Code:
@@ -21,7 +21,8 @@
 import re
 import urllib2
 import cookielib as cl
-from datetime import datetime
+#from datetime import datetime
+import dateutil.parser as dparser
 import json
 
 from .. import BeautifulSoup as bs
@@ -185,6 +186,7 @@
         hrstr="<hr />"
         descdivstr = '<div class="description">'+descdivstr[descdivstr.index(hrstr)+len(hrstr):]
         self.setDescription(self.url,descdivstr)
-        
+
+        # Can't trust dates from API anymore I'm told.
         # Dates are in Unix time
         # Take the publish date from the first chapter posted
@@ -189,10 +191,25 @@
         # Dates are in Unix time
         # Take the publish date from the first chapter posted
-        rawDatePublished = storyMetadata["chapters"][0]["date_modified"]
-        self.story.setMetadata("datePublished", datetime.fromtimestamp(rawDatePublished))
-        rawDateUpdated = storyMetadata["date_modified"]
-        self.story.setMetadata("dateUpdated", datetime.fromtimestamp(rawDateUpdated))
-
+        # rawDatePublished = storyMetadata["chapters"][0]["date_modified"]
+        # self.story.setMetadata("datePublished", datetime.fromtimestamp(rawDatePublished))
+        # rawDateUpdated = storyMetadata["date_modified"]
+        # self.story.setMetadata("dateUpdated", datetime.fromtimestamp(rawDateUpdated))
+        
+        oldestChapter = None
+        newestChapter = None
+        # Scan all chapters to find the oldest and newest, on
+        # FiMFiction it's possible for authors to insert new chapters
+        # out-of-order or change the dates of earlier ones by editing
+        # them--That WILL break epub update.
+        for chapterDate in soup.findAll('span', {'class':'date'}):
+            chapterDate = dparser.parse(chapterDate.contents[1].strip())
+            if oldestChapter == None or chapterDate < oldestChapter:
+                oldestChapter = chapterDate
+            if newestChapter == None or chapterDate > newestChapter:
+                newestChapter = chapterDate
+        self.story.setMetadata("datePublished", oldestChapter)
+        self.story.setMetadata("dateUpdated", newestChapter)
+        
         chars = soup.find("div", {"class":"inner_data"})
         # fimfic stopped putting the char name on or around the char
         # icon now for some reason.  Pull it from the image name with
@@ -215,7 +232,11 @@
                 if not isinstance(value,basestring):
                     value = unicode(value)
                 self.story.setMetadata(metakey, value)
-        
+
+        rawGroupList = soup.find('ul', {'id':'story_group_list'})
+        if rawGroupList is not None:
+            for groupName in rawGroupList.findAll('a', {'href':re.compile('^/group/')}):
+                self.story.addToList("groups",stripHTML(groupName))
             
     def getChapterText(self, url):
         logger.debug('Getting chapter text from: %s' % url)


UPDATE Sep 14 2013 - Remove obsolete beta versions

Last edited by JimmXinu; 09-14-2013 at 06:10 PM. Reason: Remove obsolete beta versions
JimmXinu is offline   Reply With Quote
Old 09-10-2013, 03:14 PM   #1855
silvrock
Junior Member
silvrock began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Sep 2013
Device: none
A thankyou post to Jimmy

Hi Jimm,

This is a post to just THANK YOU! I had suddenly realized that the FFDL plugin in my calibre was not working with fanfiction.net stories and my heart plummeted to my stomach. I checked the plugin updates and found you had released the updated plugin for calibre. I updated the plugin, restarted calibre and the fanfiction stories were properly downloading again and only then my heart found its way back to its proper place. And in that moment I thought someone ought to thank you on behalf of all the people who live by fanfics.
So, thank you.

Sending across lots of love,hugs, kisses, wonders and delight. Hope you find it intact in the package of this comment.


A fanfic reader, Silver.

Last edited by silvrock; 09-10-2013 at 03:16 PM. Reason: thanking properly
silvrock is offline   Reply With Quote
Old 09-11-2013, 01:17 AM   #1856
FaceDeer
Connoisseur
FaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enoughFaceDeer will become famous soon enough
 
Posts: 68
Karma: 706
Join Date: Nov 2012
Device: Kobo Touch
Quote:
Originally Posted by JimmXinu View Post
Looks like I got to it first--attached is a version that incorporates my version of your changes.
Aww, you beat me by like half a day. Tonight I had free time. Guess I'll use it testing, then.

Seems to be working correctly, I reran a bunch of updates and got the same results as my version. This code does break when trying to collect metadata for a story with *no* chapters (for example, this one) but that's not unexpected. FFDL already choked on those, it's just giving a different error message with this new version (previously such fanfics would fail to update with an error message that simply read "'chapters'"). It never even occurred to me to report this as a bug before, I have no idea what the correct behaviour should be for a case like this. Just thought I'd mention it since this version changes the previous behavior slightly.

I've got my own solution to detecting updated stories where authors have edited or inserted chapters earlier on. I have separate columns for both FiMFiction's *reported* word count and the word count calculated locally via the word count plugin, and a third column that compares the two. When a metadata update results in a significant difference between the two I check it out and see if it needs a forced overwrite to include edits. It's very rare, coming up with a system like that was more for fun than real necessity.
FaceDeer is offline   Reply With Quote
Old 09-11-2013, 02:48 AM   #1857
seabream
Enthusiast
seabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheeseseabream can extract oil from cheese
 
Posts: 48
Karma: 1056
Join Date: Jul 2012
Location: Central Canada
Device: Sony PRS-T1, RIM Playbook & Z10, Samsung Note, MacOSX
Quote:
Originally Posted by cklls View Post
It's clear that they've never given this any thought while their competitors are, at the very least, allowing compilation for printing that allows saving as a full document. Many sites, AO3 and harrypotterfanfiction.com to name just two that I use, have added the ability to download as epub or send directly to a Kindle. FFN is missing the boat and is far behind other fan-sites on this.

That might get them to recognize that we're not attacking them, just trying to access their product in a way that they've not accommodated.
I wouldn't be so sure that they've not given any thought to it. An issue is that FFN is an advertisement funded site whereas AO3 is donation funded. Using FFDL or hypothetically providing an epub as a whole work as opposed having each chapter separately on FFN webpages, drives/would drive down their ad views, which essentially means that people who are actually reading on the website are subsidising the people using FFDL because they actually provide revenue, while FFDL users just add to costs.
seabream is offline   Reply With Quote
Old 09-11-2013, 09:10 AM   #1858
cklls
Zealot
cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.cklls ought to be getting tired of karma fortunes by now.
 
Posts: 139
Karma: 1030058
Join Date: Jul 2010
Location: New England
Device: Nook, EB1150, Pandigital SuperNova Tablet, Samsung Galaxy Tab 2 10.1
Quote:
Originally Posted by seabream View Post
I wouldn't be so sure that they've not given any thought to it. An issue is that FFN is an advertisement funded site whereas AO3 is donation funded. Using FFDL or hypothetically providing an epub as a whole work as opposed having each chapter separately on FFN webpages, drives/would drive down their ad views, which essentially means that people who are actually reading on the website are subsidising the people using FFDL because they actually provide revenue, while FFDL users just add to costs.
Based on the direct communication I've had with them over the past couple of days, they in fact have not previously considered this technology. They asked numerous questions, most of which were focused on what the reader would need, but the nature of those questions made it clear that this was a new concept for them. While you're certainly right about the page views driving their revenues, they were very concerned about the fact that I was moving away to different sites because they didn't provide this functionality.

FFN was one of the first in the game, and while they have undoubtedly made some improvements in their technology and interface, they are one of the few fanfic websites that doesn't even allow automatic compilation for the purposes of printing. They are way behind the curve on that, and I think they should be concerned that readers will abandon them for other sources. As a fanfic author, I have posted there, but now post my work at three other sites that allow the reader to fully download either for printing or ebook compilation.
cklls is offline   Reply With Quote
Old 09-11-2013, 09:48 AM   #1859
Feynor
Enthusiast
Feynor began at the beginning.
 
Posts: 26
Karma: 10
Join Date: Sep 2013
Device: Samsung Galaxy Tab 10.1 with Moon Reader+ and Google Books.
I could probably even live with a small (unobtrusive) ad at the top of chapters, if it meant that we could sustain the functionality. How to do this though, without adding to the overhead of the ebook size I can't tell, not all downloaders support in-lined graphics if I recall.
Feynor is offline   Reply With Quote
Old 09-11-2013, 11:02 AM   #1860
Sefiriot
Bujavid tekikin
Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.
 
Sefiriot's Avatar
 
Posts: 137
Karma: 40366
Join Date: Aug 2013
Location: Malaysia
Device: Samsung Galaxy Note N7000
Quote:
Originally Posted by Feynor View Post
I could probably even live with a small (unobtrusive) ad at the top of chapters, if it meant that we could sustain the functionality. How to do this though, without adding to the overhead of the ebook size I can't tell, not all downloaders support in-lined graphics if I recall.
Not all reader apps either (well, not all handle them gracefully)-- and if reading with e-ink, graphics might not be the greatest idea, but since I've never used an e-ink device, others will be better able to judge. Still, to hear that FFN are actually considering compilation-- that's amazing! I should note however that there are authors out there who are against having their fics available in one easy download by default from a site, because of plagiarism fears. I know a person who refused to join AO3 precisely because of the availability of downloads, for the above reasons.
Sefiriot is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Count Pages kiwidude Plugins 640 07-14-2014 04:06 AM
[GUI Plugin] Resize Cover kiwidude Plugins 34 07-13-2014 07:15 PM
[GUI Plugin] Find Duplicates kiwidude Plugins 472 07-11-2014 09:34 AM
[GUI Plugin] Open With kiwidude Plugins 223 07-09-2014 02:42 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 08:22 AM.


MobileRead.com is a privately owned, operated and funded community.