Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 06-09-2017, 06:20 PM   #2191
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,973
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
Quote:
Originally Posted by readerboy7 View Post
I'm not the person who asked this, but would that actually be that big of a deal? Assuming the stories are on wayback, would it be possible to use an old version of FFF (or even FFDL for the older stories) to download them? This also applies for FFNet fics, now that wayback is disobeying their robots.txt
You're welcome to try.

Making it work for a few specific stories would be a doable project if someone really wanted to do it. Making it work as a general solution would be a remarkable PITA.

Either way, it isn't really something I'm interested in doing. And I would need to be convinced to add it to the project even if somebody else did.
JimmXinu is offline   Reply With Quote
Old 06-09-2017, 07:41 PM   #2192
Atherton25
Zealot
Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.Atherton25 can teach chickens to fly.
 
Posts: 124
Karma: 3804
Join Date: May 2016
Device: Kindle Oasis 2
Quote:
Originally Posted by gcomyn View Post
I'll look at it tonight, after I finish working.

GComyn
Any news about webnovel.com? I noticed that github had a comment about accessing after the update.
Atherton25 is offline   Reply With Quote
Advert
Old 06-10-2017, 02:20 AM   #2193
gcomyn
Member
gcomyn began at the beginning.
 
Posts: 19
Karma: 22
Join Date: Dec 2016
Location: Glendale, AZ
Device: Laptop, Tablet (nothing specific)
Quote:
Originally Posted by Atherton25 View Post
Any news about webnovel.com? I noticed that github had a comment about accessing after the update.
I've been working on it, but I am having trouble with the cookies... I've never been able to figure them out...

I'll keep working on it.

GComyn
gcomyn is offline   Reply With Quote
Old 06-10-2017, 07:33 AM   #2194
ILB
Groupie
ILB can illuminate an eclipseILB can illuminate an eclipseILB can illuminate an eclipseILB can illuminate an eclipseILB can illuminate an eclipseILB can illuminate an eclipseILB can illuminate an eclipseILB can illuminate an eclipseILB can illuminate an eclipseILB can illuminate an eclipseILB can illuminate an eclipse
 
ILB's Avatar
 
Posts: 173
Karma: 8098
Join Date: Jun 2012
Location: Germany
Device: Kindle Voyage, Kindle Oasis
It seems fanfiktion.de changed also to https

I´ve got a downloaded story ( http://www.fanfiktion.de/s/505f126500016f9206d00fa2/1 ) but every new story I´d like to download has now https ( https://www.fanfiktion.de/s/505f1265...ag-auf-Schalke )
and I have to manually delete the "s" to download.
ILB is offline   Reply With Quote
Old 06-10-2017, 06:56 PM   #2195
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,973
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
Quote:
Originally Posted by ILB View Post
It seems fanfiktion.de changed also to https
Change is committed--it will be in the next test version.
JimmXinu is offline   Reply With Quote
Advert
Old 06-10-2017, 07:19 PM   #2196
bugstomper
Member
bugstomper began at the beginning.
 
Posts: 20
Karma: 10
Join Date: Jun 2017
Device: kindle
There's a story that I would like to convert to epub that is posted to tumblr http://docfuture.tumblr.com/post/111...-call-contents which I understand fanficfare can't support as there is no standard tumblr format for books.

I could easily write a script to scrape this book and put each chapter in a file. I could probably go on to work the book into a reasonable html book format and figure out the settings for Calibre to convert that to a reasonable epub, but would it be possible for fanficfare to have an adapter for some specified format on local disk to facilitate converting books like this that are not on a site that fanficfare directly supports but have a regular enough format that it can be scraped in a more manual process?

Or is there already an adapter for locally stored books in some defined format?
bugstomper is offline   Reply With Quote
Old 06-10-2017, 09:03 PM   #2197
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,973
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
Quote:
Originally Posted by bugstomper View Post
...
Or is there already an adapter for locally stored books in some defined format?
I had to stop and think about it for a while, but in point of fact, there is!

FanFicFare has a fake site, test1.com, that I use to generate stories with different attributes without needing to hit anyone's site.

The test1.com adapter has a teststory feature where you can construct whatever metadata you like with storyId >= 1000.

And at one point, for my own purposes, I extended that feature to also have a 'chapter_urls' setting that will pull a page from a URL for each chapter (which can be a file:// URL) and look for a <blockquote> tag inside it. The blockquote will be changed to a div (because reasons), and it and everything inside it will used for the chapter text.

personal.ini section:
Spoiler:
Code:
# make a new section and change 1000 to other 1000+ numbers for other stories.
[teststory:1000]
title:Your Story Name
author_list:Author Name
authorUrl_list:https://author.url
datePublished:2015-07-26
dateUpdated:2015-10-02
category_list:Harry Potter,Superman
genre_list:Action/Adventure,Drama
# leave numWords empty and, depending on other setting, FanFicFare
# will use Calibre's code to count words.
numWords:
description:<p>A HP x Superman story with a description!</p>
 <p>'Make using' line below isn't required, but I find it handy.</p>
 <p>Made using [teststory:{{storyId}}] in personal.ini.</p>
status:Completed

# file URL,chapter name
chapter_urls:
 file:///C:/Users/user/Desktop/storysrc/chap01.html,Chapter 1
 file:///C:/Users/user/Desktop/storysrc/chap02.html,Chapter 2


Example chap01.html (Use UTF8 if encoding needed.):
Spoiler:
Code:
<html>
  <body>
    <blockquote>
<div>
<h3 extra="value">Chapter title from site chapter 1</h3>
br breaks<br><br>
Puella Magi Madoka Magica/魔法少女まどか★マギカ
<p>
  Chapter 2 text...
</p>
br breaks<br><br>
Don't—e;ver—d;o—that—a;gain, 法 &#xE9;
<hr>
horizontal rules
<hr size=1 noshade>
<p>"Lorem ipsum dolor sit amet", consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore--et dolore magna aliqua. 'Ut enim ad minim veniam', quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
</div>
    </blockquote>
  </body>
</html>


Then, after setting personal.ini and creating the chapter files, give FanFicFare the story URL: http://test1.com?sid=1000

Warning:

This is an existing undocumented feature. I have no issues with someone who can write their own page scrape scripts using it--something bugstomper specifically mentioned being able to do.

But I'm not going to be interested in expending much energy on either enhancing or supporting it. It's a dev tool for my use.
JimmXinu is offline   Reply With Quote
Old 06-10-2017, 09:25 PM   #2198
Loba
Enthusiast
Loba began at the beginning.
 
Loba's Avatar
 
Posts: 29
Karma: 10
Join Date: Sep 2016
Device: PW2 Wifi-4GB
Quote:
Originally Posted by JimmXinu View Post
You're welcome to try.

Making it work for a few specific stories would be a doable project if someone really wanted to do it. Making it work as a general solution would be a remarkable PITA.

Either way, it isn't really something I'm interested in doing. And I would need to be convinced to add it to the project even if somebody else did.
I'll look into a work around, maybe I can Frankenstein something together with older versions. If I come up with a workable solution (at least partially) I'll post it here just for others to search up in the future.
Loba is offline   Reply With Quote
Old 06-11-2017, 04:13 AM   #2199
Tharos
Connoisseur
Tharos began at the beginning.
 
Posts: 66
Karma: 10
Join Date: Jul 2010
Device: Pocketbook InkPad Color 3, Kindle Oasis 2, Moon+ Reader pro
Quote:
Processing Warlock of the Magus World (巫界术士) by OMA (epub) Source
Processing Skyfire Avenue (天火大道) by Xiao Lai (epub) Source
fanficfare.appspot.com has held this status for over 12 hours.
Caliber does not create any file at all.
Tharos is offline   Reply With Quote
Old 06-11-2017, 12:46 PM   #2200
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,973
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
Quote:
Originally Posted by Tharos View Post
fanficfare.appspot.com has held this status for over 12 hours.
Caliber does not create any file at all.
Both of those stories have problems on the site. wuxiaworld.com/sfl-index/ has a bad URL for chapter 483 and wuxiaworld.com/wmw-index/ has the same chapter URL for every chapter from 649 on.

So those two aren't going to download correctly regardless.

More generally:

It's not uncommon for the web service to not work on stories with hundreds of chapters (those two have 700+ & 500+). There are limits and timeouts that apply. See FAQ.

"Caliber does not create any file at all" is not a terribly useful problem report. In future, include any error messages and a description of what specifically happened. Ideally, you would also run calibre in debug mode (under calibre Preferences dropdown) to see any errors.

With several hundred chapters, a download will often run for a really long time before finishing. Especially if you have a slow_down_sleep_time set.

It's also possible that the site has started slowing or even blocking your requests if you've been downloading many and/or large stories. In that case, a slow_down_sleep_time setting will often help by spreading your requests out more over time. I don't know enough about wuxiaworld.com to know for sure.
JimmXinu is offline   Reply With Quote
Old 06-11-2017, 01:58 PM   #2201
AndersW
Connoisseur
AndersW began at the beginning.
 
Posts: 51
Karma: 10
Join Date: Jun 2014
Device: none
I am trying to get prequelHTML to be added to a custom column, but I can't get it to work.
AndersW is offline   Reply With Quote
Old 06-11-2017, 02:27 PM   #2202
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,973
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
Quote:
Originally Posted by AndersW View Post
I am trying to get prequelHTML to be added to a custom column, but I can't get it to work.
Can you be more specific? It works fine for me as long as it's a Long Text custom column.

personal.ini:
Code:
[www.fimfiction.net]
add_to_custom_columns_settings:
 prequelHTML=>#long_text
JimmXinu is offline   Reply With Quote
Old 06-11-2017, 02:48 PM   #2203
bugstomper
Member
bugstomper began at the beginning.
 
Posts: 20
Karma: 10
Join Date: Jun 2017
Device: kindle
Quote:
Originally Posted by JimmXinu View Post
I had to stop and think about it for a while, but in point of fact, there is!
That is so cool! Even cooler, as I looked at the adapters I could see that it would be almost as easy to write my own adapter for the site based on the ones you have. The only question I am left with about it is that every adapter I have looked at assumes that the first URL it looks at contains somewhere a list of links to every chapter in the story. The story I am looking at has no such index page, only prev and next links on each chapter. Is there any example of an adapter that gets the chapter links as it iterates through the chapters or is it a requirement that a site has something that lists all chapters in one place?

I guess I can write something that generates all the chapter links and use it with your test site adapter or an adapter that I write. Or even override extractChapterAndMetadata to read all the chapters and then read all of them again in each call to getChapterText. Is that what would be necessary? Are there any adapters that do something similar whose code I can look at?

Last edited by bugstomper; 06-11-2017 at 02:55 PM.
bugstomper is offline   Reply With Quote
Old 06-11-2017, 03:03 PM   #2204
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,973
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
Generally speaking, if a site doesn't have chapter lists available, I declare it unsupportable for being too far outside the accepted paradigm.

I vaguely recall somebody writing an adapter for a site like that, but I don't remember if I allowed it to be included, let alone what site it was...

As for why, it's about updates. I assume that most users, like me, follow hundreds of in-progress stories. IMHO, updating existing ebooks with new chapters without downloading all the old ones again is just as important as downloading whole stories.
JimmXinu is offline   Reply With Quote
Old 06-11-2017, 03:24 PM   #2205
AndersW
Connoisseur
AndersW began at the beginning.
 
Posts: 51
Karma: 10
Join Date: Jun 2014
Device: none
Quote:
Originally Posted by JimmXinu View Post
Can you be more specific? It works fine for me as long as it's a Long Text custom column.

personal.ini:
Code:
[www.fimfiction.net]
add_to_custom_columns_settings:
 prequelHTML=>#long_text
In configure FFF under custom columns. When I select the column I want prequelHTML to go to, prequelHTML isn't an option that I can choose.
AndersW is offline   Reply With Quote
Reply

Tags
fanfiction


Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] KindleUnpack - The Plugin DiapDealer Plugins 523 07-15-2025 06:45 PM
[GUI Plugin] Open With kiwidude Plugins 404 02-21-2025 05:42 AM
[GUI Plugin] Marvin XD Philantrop Plugins 126 01-29-2017 12:48 PM
[GUI Plugin] KiNotes -axel- Plugins 0 07-14-2013 06:39 PM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 10:53 AM.


MobileRead.com is a privately owned, operated and funded community.