Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 11-27-2010, 06:23 AM   #1
siebert
Developer
siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.
 
Posts: 155
Karma: 280
Join Date: Nov 2010
Device: Kindle 3 (Keyboard) 3G / iPad 9 WiFi / Google Pixel 6a (Android)
Exclamation Calibre epub from recipe fails in Sigil and FBReader on Android

Hi,

I created a new recipe (included below) which seems to work fine as long as I open the generated epub file with the calibre viewer.

Using the same epub with FBReader on Android or Sigil reveals some issues though:

FBReader 0.7.17:
All pages show the same picture.
Most navigation links are broken.
Text in "Kurzkritik" section has same fontsize as "Kurzkritik" heading. Text in "Kritik" section is fine though.

Sigil 0.3.1:
All pages show the same picture.
Some navigation links are broken.

I opened the epub with winrar and noticed that all html and image files are stored using the same filename (index.html and img1.jpg) and the distinction is only made by the path in which the files are stored. Maybe this is the reason for the behaviour, as I don't experience these issues with epub files where the filenames are different.

Is this a known issue? Can I do anything within the recipe to prevent it?

Here is the recipe I'm using:
Code:
#!/usr/bin/env  python
# -*- coding: utf-8 mode: python -*-

__license__   = 'GPL v3'
__copyright__ = 'Steffen Siebert <calibre at steffensiebert.de>'
__version__   = '1.0'

""" http://film-dienst.de """

import re
import string
from calibre.web.feeds.recipes import BasicNewsRecipe
from calibre.ptempfile import PersistentTemporaryFile

class FilmDienst(BasicNewsRecipe):
    __author__ = 'Steffen Siebert'
    title = 'film-dienst'
    description = 'Filmmagazin'
    publisher ='Deutsche Zeitung GmbH, Bonn'
    category = 'Film, Germany'
    lang = 'de-DE'
    encoding = "windows-1252"
    language = 'de'
    publication_type = 'magazine'
    articles_are_obfuscated = True
    use_embedded_content = False
    no_stylesheets = True
    
    conversion_options = {'comments': description, 'tags': category, 'language': language,
                          'publisher': publisher}

    IMAGE_RE = re.compile('<img src="kritikenimages/([0-9]+)\.jpg"')
    REVIEW_RE = re.compile('<tr><td><table[^>]+><tr><td class="rubrikgross" align="left">([^<]+)</td><td align="right"><a[^>]+><img [^>]+></a></td></tr></table></td></tr>\s+<tr><td>(<img src="([^"]+)" align="right">)?(.*)</td></tr>\s+<tr><td align="right"><i>([^<]+)?</i></td></tr>', re.DOTALL)
    SHORT_REVIEW_RE = re.compile('<tr><td class="rubrikgross">[^<]+</td></tr>\s+<tr><td><img [^>]+>(.*)</td></tr>')

    """
    Calibre recipe for the film-dienst magazine.

    Only the reviews are fully available without a subscription, so we ignore the remaining articles.
    """

    def get_obfuscated_article(self, url):
        """
        The film-dienst pages are very hard to handle with BeautifulSoup.
        So we extract the desired content using regular expressions and create a simple
        html page with minimal formatting to convert into the ebook.
        """

        shortReview = None
        imageUrl = None
        fdNumber = None

        result = re.match("^http://film-dienst\.kim-info\.de/kritiken.php\?nr=([0-9]+)$", url)
        number = result.group(1)

        br = self.get_browser()

        # Fetch review text.
        reviewUrl = "http://film-dienst.kim-info.de/kritiken.php?nr=%s"        
        con = br.open(reviewUrl % number)
        output = con.read()
        match = self.IMAGE_RE.search(output)
        if match:
            fdNumber = match.group(1)
            imageUrl = "http://film-dienst.kim-info.de/kritikenimages/%s.jpg" % fdNumber
    
        match = self.REVIEW_RE.search(output)
        title = match.group(1)
        review = re.sub("</p>\n</p>", "<p/>\n", match.group(4))
        author = match.group(5)
        if author == None:
            author = "-"
        
        # Fetch short review text.
        shortReviewUrl = "http://film-dienst.kim-info.de/kritiken.php?pos=Kurz&nr=%s"
        con = br.open(shortReviewUrl % number)
        output = con.read()

        match = self.SHORT_REVIEW_RE.search(output)
        if match:
            shortReview = match.group(1)
    
        # Write content to new temporary html file.
        html = PersistentTemporaryFile('_fa.html')
        if fdNumber:
            html.write('<html>\n<head>\n<title>%s - fd %s</title>\n</head>\n' % (title, fdNumber))
            html.write("<body>\n<b>fd %s</b><h1>%s</h1>\n" % (fdNumber, title))
        else:
            html.write('<html>\n<head>\n<title>%s</title>\n</head>\n' % (title))
            html.write("<body>\n<h1>%s</h1>\n" % (title))
        if shortReview:
            html.write("<h2>Kurzkritik</h2>\n")
            html.write("%s\n" % shortReview)
        html.write("<h2>Kritik</h2>\n")
        if imageUrl:
            html.write('<img src="%s"><br>' % imageUrl)
        html.write("%s<br>\n" % review)
        html.write("<i>%s</i><br>\n" % author)
        html.write("</body>\n</html>\n")
        html.close()

        return html.name

    def parse_index(self):
        """
        Find all review links and group them by movie start date.
        Also get magazine cover and issue number.
        """

        feedName = None
        feeds = []
        articles = []

        # Find cover image.
        soup = self.index_to_soup("http://film-dienst.kim-info.de/")
        cover = soup.find('img', alt='Cover film-dienst')
        self.cover_url = 'http://film-dienst.kim-info.de/' + cover['src']

        # Find issue number.
        issue = soup.find('span', attrs={'class':'jahr'})
        self.timefmt = self.tag_to_string(issue)

        # Navigate to nested table containing the list of reviews.
        start = soup.find("td", "rubrikgross")
        table = start.parent.parent
        for row in table.findAll("tr"):
            if row.table:
                # We found the right table.
                # Now handle all table rows.
                for row in row.table.findAll("tr"):
                    # Movie start date is enclosed in a bold tag.
                    b = row.find("b")
                    if b:
                        # If it's not the first section, append previous section to list of feeds.
                        if feedName != None:
                            feeds.append((feedName, articles))
                        articles = []
                        feedName = self.tag_to_string(b)
                        continue
                    # Find reviews via the link tag.
                    link = row.find("a")
                    if link:
                        url = "http://film-dienst.kim-info.de/" + link['href']
                        articles.append({'title': self.tag_to_string(link), 'url': url, 'date': ''})
                        
                # Append last section to list of feeds.
                if feedName:
                    feeds.append((feedName, articles))

                return feeds
siebert is offline   Reply With Quote
Old 11-27-2010, 07:57 AM   #2
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Typically these questions would be discussed in the Calibre Recipes sub-forum.
DoctorOhh is offline   Reply With Quote
Old 11-27-2010, 08:54 AM   #3
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by dwanthny View Post
Typically these questions would be discussed in the Calibre Recipes sub-forum.
This one is borderline, even though he posted a recipe. His real problem seems to be that the EPUB displays differently in two different programs. That shouldn't happen. He'll probably need to post a sample EPUB and the tester will need FBReader or Sigil.

I doubt the problem has anything to do with the recipe default structure (which is used in all recipes). If it did, no recipe-created EPUBs would work with Sigil. If that is related to the problem, then he should be asking in FBReader or Sigil forums, as the normal recipe-created EPUB structure works fine in all my browsers (as html) and in all my EPUB readers.

Normally, I'd think his problem is in the site/recipe, but he says it works in the Calibre viewer, which rules that out.
Starson17 is offline   Reply With Quote
Old 11-27-2010, 09:01 AM   #4
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by Starson17 View Post
This one is borderline, even though he posted a recipe. His real problem seems to be that the EPUB displays differently in two different programs. That shouldn't happen. He'll probably need to post a sample EPUB and the tester will need FBReader or Sigil.
He also stated in another post that masthead had quit working with this update. In my ignorance I wonder if there isn't some difference between Python 2.6 and 2.7 rearing it's ugly head?
DoctorOhh is offline   Reply With Quote
Old 11-27-2010, 09:09 AM   #5
siebert
Developer
siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.
 
Posts: 155
Karma: 280
Join Date: Nov 2010
Device: Kindle 3 (Keyboard) 3G / iPad 9 WiFi / Google Pixel 6a (Android)
Hi,

I would rather not like to post a sample EPUB, as it contains copyrighted material. But I see no reason why it should be necessary as everyone can create the EPUB with my recipe.

The issue isn't related with Python 2.7 as it also happens with calibre versions before 0.7.30.

Ciao,
Steffen
siebert is offline   Reply With Quote
Old 11-27-2010, 09:34 AM   #6
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by siebert View Post
I would rather not like to post a sample EPUB, as it contains copyrighted material. But I see no reason why it should be necessary as everyone can create the EPUB with my recipe.
That's your decision, but notice that to help you, the helper must install and run your recipe, then hope he has created an EPUB like yours to show the error, (perhaps it isn't the same and your system is special) then have one of the programs, FBReader or Sigil to do the test.

Posting a completed recipe that only contains publicly available text and images for the purpose of debugging a problem is almost certainly fair use. You would be free to delete the book after the problem is solved.
Starson17 is offline   Reply With Quote
Old 11-27-2010, 01:55 PM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
the epub generated from that recipe has links that work in the calibre reader, Adobe digital editions and on my SONY PRS 650. Further running flightcrew on it to check its validity pointed to no errors (beyond the usual XHTML guff).

So if the epub isn't working in fbreader, sigil or aldiko you need to open bug reports for those programs.

The cause most likely is the common filenames, as you point out. But a failure to handle files with the same name in different directories is most definitely a bug in the viewer program.
kovidgoyal is offline   Reply With Quote
Old 11-28-2010, 08:46 AM   #8
siebert
Developer
siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.
 
Posts: 155
Karma: 280
Join Date: Nov 2010
Device: Kindle 3 (Keyboard) 3G / iPad 9 WiFi / Google Pixel 6a (Android)
Quote:
Originally Posted by kovidgoyal View Post
So if the epub isn't working in fbreader, sigil or aldiko you need to open bug reports for those programs.
Is this a known issue? I can't belive that I'm the first and only one to use calibre generated EPUB files with fbreader or sigil?

I agree it would be best to fix the failing programs, but as there are several affected (including closed source programs), it seems more promising and easy to me to make calibre work around the issue.

I first tried to use unique file names, which fixed the pictures, but navigation was still broken. So I changed calibre to use unique filenames without subdirectories and the result works fine for me

So it would be great if you could apply the following change to calibre.

Ciao,
Steffen

Code:
# Bazaar merge directive format 2 (Bazaar 0.90)
# revision_id: siebert@steffensiebert.de-20101128132414-\
#   i7ak3h1gflu2m11b
# target_branch: http://bazaar.launchpad.net/~kovid/calibre/trunk/
# testament_sha1: cf755f77a363fc0310b3d12cb2487c4ed531d91f
# timestamp: 2010-11-28 14:25:50 +0100
# base_revision_id: kovid@kovidgoyal.net-20101128023305-\
#   0ew07r4bzia4bb0t
# 
# Begin patch
=== modified file 'src/calibre/web/feeds/__init__.py'
--- src/calibre/web/feeds/__init__.py	2010-09-13 16:15:35 +0000
+++ src/calibre/web/feeds/__init__.py	2010-11-28 13:24:14 +0000
@@ -14,6 +14,11 @@
 from calibre import entity_to_unicode, strftime
 from calibre.utils.date import dt_factory, utcnow, local_tz
 
+FEED_NAME = 'feed%d.html'
+''' Template for the feed index file. '''
+ARTICLE_NAME = 'feed%d_article%d.html'
+''' Template for the article file. '''
+
 class Article(object):
 
     def __init__(self, id, title, url, author, summary, published, content):

=== modified file 'src/calibre/web/feeds/news.py'
--- src/calibre/web/feeds/news.py	2010-11-04 22:26:10 +0000
+++ src/calibre/web/feeds/news.py	2010-11-28 13:24:14 +0000
@@ -21,7 +21,7 @@
 from calibre.web import Recipe
 from calibre.ebooks.metadata.toc import TOC
 from calibre.ebooks.metadata import MetaInformation
-from calibre.web.feeds import feed_from_xml, templates, feeds_from_index, Feed
+from calibre.web.feeds import feed_from_xml, templates, feeds_from_index, Feed, FEED_NAME, ARTICLE_NAME
 from calibre.web.fetch.simple import option_parser as web2disk_option_parser
 from calibre.web.fetch.simple import RecursiveFetcher
 from calibre.utils.threadpool import WorkRequest, ThreadPool, NoResultsPending
@@ -912,16 +912,10 @@
 
         self.feed_objects = feeds
         for f, feed in enumerate(feeds):
-            feed_dir = os.path.join(self.output_dir, 'feed_%d'%f)
-            if not os.path.isdir(feed_dir):
-                os.makedirs(feed_dir)
 
             for a, article in enumerate(feed):
                 if a >= self.max_articles_per_feed:
                     break
-                art_dir = os.path.join(feed_dir, 'article_%d'%a)
-                if not os.path.isdir(art_dir):
-                    os.makedirs(art_dir)
                 try:
                     url = self.print_version(article.url)
                 except NotImplementedError:
@@ -934,12 +928,12 @@
                 func, arg = (self.fetch_embedded_article, article) if self.use_embedded_content else \
                             ((self.fetch_obfuscated_article if self.articles_are_obfuscated \
                               else self.fetch_article), url)
-                req = WorkRequest(func, (arg, art_dir, f, a, len(feed)),
+                req = WorkRequest(func, (arg, self.output_dir, f, a, len(feed)),
                                       {}, (f, a), self.article_downloaded,
                                       self.error_in_article_download)
                 req.feed = feed
                 req.article = article
-                req.feed_dir = feed_dir
+                req.feed_dir = self.output_dir
                 self.jobs.append(req)
 
 
@@ -961,8 +955,7 @@
 
         for f, feed in enumerate(feeds):
             html = self.feed2index(f,feeds)
-            feed_dir = os.path.join(self.output_dir, 'feed_%d'%f)
-            with open(os.path.join(feed_dir, 'index.html'), 'wb') as fi:
+            with open(os.path.join(self.output_dir, FEED_NAME%f), 'wb') as fi:
                 fi.write(html)
         self.create_opf(feeds)
         self.report_progress(1, _('Feeds downloaded to %s')%index)
@@ -1148,9 +1141,7 @@
             ref.title = 'Masthead Image'
             opf.guide.append(ref)
 
-        manifest = [os.path.join(dir, 'feed_%d'%i) for i in range(len(feeds))]
-        manifest.append(os.path.join(dir, 'index.html'))
-        manifest.append(os.path.join(dir, 'index.ncx'))
+        manifest = [dir, os.path.join(dir, 'index.html'), os.path.join(dir, 'index.ncx')]
 
         # Get cover
         cpath = getattr(self, 'cover_path', None)
@@ -1183,7 +1174,6 @@
             f = feeds[num]
             for j, a in enumerate(f):
                 if getattr(a, 'downloaded', False):
-                    adir = 'feed_%d/article_%d/'%(num, j)
                     auth = a.author
                     if not auth:
                         auth = None
@@ -1192,14 +1182,15 @@
                         desc = None
                     else:
                         desc = self.description_limiter(desc)
-                    entries.append('%sindex.html'%adir)
+                    indexname = ARTICLE_NAME%(num, j)
+                    entries.append(indexname)
                     po = self.play_order_map.get(entries[-1], None)
                     if po is None:
                         self.play_order_counter += 1
                         po = self.play_order_counter
-                    parent.add_item('%sindex.html'%adir, None, a.title if a.title else _('Untitled Article'),
+                    parent.add_item(indexname, None, a.title if a.title else _('Untitled Article'),
                                     play_order=po, author=auth, description=desc)
-                    last = os.path.join(self.output_dir, ('%sindex.html'%adir).replace('/', os.sep))
+                    last = os.path.join(self.output_dir, (indexname).replace('/', os.sep))
                     for sp in a.sub_pages:
                         prefix = os.path.commonprefix([opf_path, sp])
                         relp = sp[len(prefix):]
@@ -1226,7 +1217,7 @@
 
         if len(feeds) > 1:
             for i, f in enumerate(feeds):
-                entries.append('feed_%d/index.html'%i)
+                entries.append(FEED_NAME%i)
                 po = self.play_order_map.get(entries[-1], None)
                 if po is None:
                     self.play_order_counter += 1
@@ -1237,11 +1228,11 @@
                 desc = getattr(f, 'description', None)
                 if not desc:
                     desc = None
-                feed_index(i, toc.add_item('feed_%d/index.html'%i, None,
+                feed_index(i, toc.add_item(FEED_NAME%i, None,
                     f.title, play_order=po, description=desc, author=auth))
 
         else:
-            entries.append('feed_%d/index.html'%0)
+            entries.append(FEED_NAME%0)
             feed_index(0, toc)
 
         for i, p in enumerate(entries):
@@ -1253,7 +1244,7 @@
             opf.render(opf_file, ncx_file)
 
     def article_downloaded(self, request, result):
-        index = os.path.join(os.path.dirname(result[0]), 'index.html')
+        index = os.path.join(os.path.dirname(result[0]), ARTICLE_NAME%request.requestID)
         if index != result[0]:
             if os.path.exists(index):
                 os.remove(index)
@@ -1263,7 +1254,7 @@
         article = request.article
         self.log.debug('Downloaded article:', article.title, 'from', article.url)
         article.orig_url = article.url
-        article.url = 'article_%d/index.html'%a
+        article.url = ARTICLE_NAME%request.requestID
         article.downloaded = True
         article.sub_pages  = result[1][1:]
         self.jobs_done += 1

=== modified file 'src/calibre/web/feeds/templates.py'
--- src/calibre/web/feeds/templates.py	2010-08-29 18:39:20 +0000
+++ src/calibre/web/feeds/templates.py	2010-11-28 13:24:14 +0000
@@ -12,6 +12,7 @@
         TABLE, TD, TR
 
 from calibre import preferred_encoding, strftime, isbytestring
+from calibre.web.feeds import FEED_NAME, ARTICLE_NAME
 
 def CLASS(*args, **kwargs): # class is a reserved word in Python
     kwargs['class'] = ' '.join(args)
@@ -92,7 +93,7 @@
         for i, feed in enumerate(feeds):
             if feed:
                 li = LI(A(feed.title, CLASS('feed', 'calibre_rescale_120',
-                    href='feed_%d/index.html'%i)), id='feed_%d'%i)
+                    href=FEED_NAME%i)), id='feed_%d'%i)
                 ul.append(li)
         div = DIV(
                 PT(IMG(src=masthead,alt="masthead"),style='text-align:center'),
@@ -115,14 +116,14 @@
             hr.tail = '| '
 
         if f+1 < len(feeds):
-            link = A('Next section', href='../feed_%d/index.html'%(f+1))
+            link = A('Next section', href=FEED_NAME%(f+1))
             link.tail = ' | '
             navbar.append(link)
-        link = A('Main menu', href="../index.html")
+        link = A('Main menu', href="index.html")
         link.tail = ' | '
         navbar.append(link)
         if f > 0:
-            link = A('Previous section', href='../feed_%d/index.html'%(f-1))
+            link = A('Previous section', href=FEED_NAME%(f-1))
             link.tail = ' |'
             navbar.append(link)
         if top:
@@ -203,20 +204,19 @@
                 navbar.append(BR())
             navbar.append(BR())
         else:
-            next = 'feed_%d'%(feed+1) if art == number_of_articles_in_feed - 1 \
-                    else 'article_%d'%(art+1)
-            up = '../..' if art == number_of_articles_in_feed - 1 else '..'
-            href = '%s%s/%s/index.html'%(prefix, up, next)
+            next = FEED_NAME%(feed+1) if art == number_of_articles_in_feed - 1 \
+                    else ARTICLE_NAME%(feed, art+1)
+            href = next
             navbar.text = '| '
             navbar.append(A('Next', href=href))
-        href = '%s../index.html#article_%d'%(prefix, art)
+        href = FEED_NAME%feed + '#article_%d'%art
         navbar.iterchildren(reversed=True).next().tail = ' | '
         navbar.append(A('Section Menu', href=href))
-        href = '%s../../index.html#feed_%d'%(prefix, feed)
+        href = 'index.html#feed_%d'%feed
         navbar.iterchildren(reversed=True).next().tail = ' | '
         navbar.append(A('Main Menu', href=href))
         if art > 0 and not bottom:
-            href = '%s../article_%d/index.html'%(prefix, art-1)
+            href = ARTICLE_NAME%(feed, art-1)
             navbar.iterchildren(reversed=True).next().tail = ' | '
             navbar.append(A('Previous', href=href))
         navbar.iterchildren(reversed=True).next().tail = ' | '

=== modified file 'src/calibre/web/fetch/simple.py'
--- src/calibre/web/fetch/simple.py	2010-11-04 19:35:23 +0000
+++ src/calibre/web/fetch/simple.py	2010-11-28 13:24:14 +0000
@@ -7,7 +7,7 @@
 Fetch a webpage and its links recursively. The webpages are saved to disk in
 UTF-8 encoding with any charset declarations removed.
 '''
-import sys, socket, os, urlparse, re, time, copy, urllib2, threading, traceback
+import sys, socket, os, urlparse, re, time, copy, urllib2, threading, traceback, hashlib
 from urllib import url2pathname, quote
 from httplib import responses
 from PIL import Image
@@ -334,7 +334,7 @@
                 self.log.exception('Could not fetch image ', iurl)
                 continue
             c += 1
-            fname = ascii_filename('img'+str(c))
+            fname = ascii_filename(hashlib.sha1(data).hexdigest())
             if isinstance(fname, unicode):
                 fname = fname.encode('ascii', 'replace')
             imgpath = os.path.join(diskpath, fname+'.jpg')

# Begin bundle
IyBCYXphYXIgcmV2aXNpb24gYnVuZGxlIHY0CiMKQlpoOTFBWSZTWSOcC/AABbZ/gARUQABa7//3
f+dWjr////BgCY76Ct7nFApBKDhsmnr6hPb1zu7iwklE9AKntMUyntE9JplPDVHqHpNpk1G1ADIH
qEao1NPUxNDQyaGgGgBiA00aAAAFU/JTYoGQNNNA0aBiAGRpkGgBoAkRE0EE9JtJpqQ8mpvU1NNA
0aDRoAZqNAIqENTKNpqPSaYmhNG0j1M1APFA0AAAJIiYgEyKeJhGmkNpE09NINAA0AaDhKOuuqvJ
v3UnQyZlo0z+RxdcjB6Hxez/j/B5jPR1oIWPhaPVMeRnUNDr1X1ufPeBDEjjWEiAwPLUhyr6mg0Y
SjR4u0HajypZOlGectwgQ4C7vQdFsRhyO4PWGQNJNtobE2L2fsIVz7Kz5oXTpnO5k9MaOGr9IL2W
Y7QpZ12u1YQY4xbErF7xrFWY2vkpMUdMKWwum7BzN5Z2dLrm4cWhVH2n6fwn+v85z4RSf9nm+S5F
0KOREOkB2nOhpauLbI8dfAxrUfkjKf7nY0oz8s1Q8qmBQ7+3w+EPt6CblA9KhGuEbcpVEBogSL4E
WHYhOdnHjybICovbhqDz6efLr1AFzSPfnsSxKgsAUGQzIBW9i01mFGNg8Rq24ngeLSaGCz+YDfvR
GUsEXFT2TmaD0SnAZlBHdihQBjkMBLwgy4sXROGSMeRcpPYKzzpIcdEsag0VZSDmBsgMvyIPLbfO
jRGq7fC9V7scJxqp/lfb6I4Vd8tNwL2q/JkqwXDME1QJgXrBdko9LvXWUp1UwTCVH9vga9xiNJD4
JZhelz2RDhqIvyKY9JVcZDQNN9Q7PPvCO4iyDxOjXU0UaQSwHIC8sVYKkcgFJ6CJSCRB3+jCaUpi
qq4IPpBWPgU31M309Bd/q6PoKD5wBXpXL43gu0nEzMQVCMdC9NS1CFG95MXuaQ9suc8PsTpRaa8w
WSQaVyOw7uBzrxL0BBkwK27wU5loTaVy+htwLJJaqUxtx46bdlfTS4Fq1tzmCqjgi912/LuocTUZ
dFtMypU4V2rZdMOfsdxI/PfLZm83vBPaE7wUpHSPpi7v68MfMCet57pU5QUgm6NJznxJA996hptX
jN2vKamWPCdxXOcU8fgVTZTIExEoGc+FKU1D47ISUvkmW1pQleuYCUCZiUPmBW0ZQwnWPSN67njQ
oMCaCmQcDrLzxF2Twh8oBVJ1GzIz0JMTQvFUyJoajs7gU1u6AzNbo4cFIBbWiZ49rol0Aqgnqzjs
3ksib2OlpNJfB3KjQi0vzol1LVM1jG/zYY24yW+GlXApVgKayGJMahiuJI5JOQNmIK7jqKyF4Kkr
rjVMsSmCtMzQ5jUcIzKr89b0jfOTYUeWghOZVDJnhHZp0ixKywFQppGKy4nLFni90cxgVChKQ04A
sIe1spYb9Dit0cAUBy/s0Z6yO2C7bNm7MlwYKSiFnILiX2Y6cdshOwFjSmfvcTQ3Gfj6zE6Mdrb0
3UmaVoMeMdd0NqhEVZvdujgci4FJYZmsc121YnLdbs3eV5ntcLrme3draU0pgpxRNxOXVE6GsFZK
/B+lHBJelrI8/HeJjJgfCPjCkDuou7mG1LM2qyWQlevaCP4g2sXu1ym2Db9Pk9giELwBFfbIL2gs
PADDqJ/BjhwZjYkLb4e4DcQB+QI+77wREXmyJA7fcBU91YMGevnp4impajvbTTExl64Hbbx0hru+
L0UotSbrSnq1KByXB3bEBC094Exa5J8YLfVsC02FCnBesFCC2m459mgu9t/fDIC4mFRGGFL2dAKv
kldVXSCxxQQTLheBbG+GQOr8/t58ymGYseLrBIK59d9Px5X+s6WQwLkw++Ff9BZVoHZjUazSRkBI
y1gtZO6qBCUoCzZUXvQgu2AtX74EcgXb0ImjIzPSSV8DM6zKlZhoeeuJw2Gw71u081iN7FhtOpWY
HJnNSoR+QKTlXuBbxkXDHJxNDViLbgLrLUxsUsgMDJqpoBd+wFidUToCgNkcB4mAK5LuR8I95cMv
hwr0Aq93M2nTu6OyO3kQh9MnPNQNTM0a+oFITjA5YNJbXiYcOLMfOJ7VkIDAR5TMatoVBHCylZ6h
FCWzSepJbN4JenaVVqu15sBqNjoC828XQeTJBZts2vSIERLf2R4Nay0B8c6H2jBp72GW3QJRwF2L
38VjEG1kRsQImE6s9g6RiICWdOQ2EAuiaYu4CMdxYCOIAdSgrwOTA5+F6CvqC8+T1b2yDDjzOvvH
LA6es75M2VpqLTsxHsLg2R1UaYmpCVgvFJwxTgFp/oMeQMF7cwXhxgj3SJD5c5W7AKPvBcOSXjyX
WEB8QO9LUpZhqbctJJ3XwhN19yacQrzhvVmb1A8uTW8OQVhmwDPOhd4xZYwWbVSTzJOHHhAKnAzO
CHJpTD1rmpkbelz8bZiqcbR0ctVzct+OwSKxbDnanVMN+7wcyUVuXJmjRlBjnqBpootXhXctNopJ
dYWJYuHjtYWd3YDuOQTuDMENLkJBT2FeALoCuXX5x9+Xi1o9fMFtIkfeCgZW0VViWTBV7JNOlH24
2tZGoV4vjygsmUbZAm79lKysAmG/Zooa1CtjJbzpwY5nLXZIYA1olnX9JDTBdMj6vVGKnq8UTPhZ
EI9x6X9zpmfmUyWNtPO4s2cHKOlghDFiq0B2C865LdnJZ0dUQiOmwZphuYWSBlW81NgCzCsFXWZM
ixyuKcUV1ppxSpDiChhEbTSDgVgrfceI5TX+WjwuRXjj2uqzSmW3KO4Ja6TcBYKEyQSZ+rUTYE1M
/OZRCB/TWF44FHaMZ5/zsAuHCKGXHM/kNUd9VjYq2YNbYAkNol5GxDBS6yWhZhtr4Rv3W2A3lsF8
4slolGPlvqjkJMeL6Yj0ZkoDiUCTiTmzJTCQUAnKic9RF3iJTLNRqryoFQF0ZkRRV0HhwwShiR9h
tK6LiiZilKpWhHKAaAVzirbAF9Hlbs/sXohMe6wFHRAvNIXxDqrYiyzAQg7dfprxg7xTLZAmmWC9
2ABtnChkDF1KOrlVRWjsUv/F3JFOFCQI5wL8AA==
siebert is offline   Reply With Quote
Old 11-28-2010, 09:39 AM   #9
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by siebert View Post
Is this a known issue? I can't belive that I'm the first and only one to use calibre generated EPUB files with fbreader or sigil?
You are the first I've seen report this. Most likely because we send our recipe epubs straight to read our recipes on our device or view them in calibre and have no reason to view them in Sigil.

Quote:
Originally Posted by siebert View Post
I agree it would be best to fix the failing programs, but as there are several affected (including closed source programs), it seems more promising and easy to me to make calibre work around the issue.
I used your recipe and saw the same results in Sigil. I also tried a local newspaper with the same results in Sigil.

That said, I opened your recipe with the calibre viewer, FBReader, Adobe Digital Editions and my PRS-505 ereader and they all work fine, the images are as expected and the links work as advertised.

This appears to be just a Sigil problem.

I'm using Windows XP sp3 32bit, FBReader 0.12.10, calibre 0.7.31, Sigil 3.2, Adobe Digital Editions 1.7.2.1131.

Of course if your code works as advertised without any unintended consequences for other readers or devices then ...

Last edited by DoctorOhh; 11-28-2010 at 10:24 AM. Reason: Clarifying language first paragraph
DoctorOhh is offline   Reply With Quote
Old 11-28-2010, 10:10 AM   #10
siebert
Developer
siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.
 
Posts: 155
Karma: 280
Join Date: Nov 2010
Device: Kindle 3 (Keyboard) 3G / iPad 9 WiFi / Google Pixel 6a (Android)
Quote:
Originally Posted by dwanthny View Post
You are the first I've seen report this. Most likely because we send our recipe epubs straight to our device or view them in calibre.
Does the resulting EPUB look different if it is send straight to a device? What do I have to do to try it?

Quote:
That said, I opened your recipe with the calibre viewer, FBReader, Adobe Digital Editions and my PRS-505 ereader and they all work fine, the images are as expected and the links work as advertised.

This appears to be just a Sigil problem.
Please note that I was refering to FBReader for Android devices, not the PC version. On Android FBReader shows this issue.

Ciao,
Steffen
siebert is offline   Reply With Quote
Old 11-28-2010, 10:19 AM   #11
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by siebert View Post
Does the resulting EPUB look different if it is send straight to a device? What do I have to do to try it?
It is identical. I didn't mean to confuse you with my imprecise language. The epub is never sent straight to the device. It is always created and added to the library just like you experienced. Then it is transferred to the device if you have auto transfer on.

Last edited by DoctorOhh; 11-28-2010 at 10:25 AM.
DoctorOhh is offline   Reply With Quote
Old 11-28-2010, 10:30 AM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I don't want to obfuscate filenames in recipe downloads as I often look at the downloaded files to debug problems and the nice folder structure makes it easy to find files.

And I tried FBReader on my computer, it works fine. Only Sigil has issues and I'm sure if you point them out to Valloric on the Sigil forum they will be quickly fixed.

That said, I would be willing to accept the patch if you can make it optional. That way when developing recipes it would use the current structure, but for normal downloads, it would not.
kovidgoyal is offline   Reply With Quote
Old 11-28-2010, 11:52 AM   #13
siebert
Developer
siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.
 
Posts: 155
Karma: 280
Join Date: Nov 2010
Device: Kindle 3 (Keyboard) 3G / iPad 9 WiFi / Google Pixel 6a (Android)
Quote:
Originally Posted by kovidgoyal View Post
I don't want to obfuscate filenames in recipe downloads as I often look at the downloaded files to debug problems and the nice folder structure makes it easy to find files.
They are not obfuscated, only flattened.

Instead of

feed_0/index.html
feed_0/article_0/index.html

the files are named

feed0.html
feed0_article0.html

Finding files should be as easy as with the current scheme. If you don't like the names, you could change them via the two new constants.

Quote:
And I tried FBReader on my computer, it works fine. Only Sigil has issues and I'm sure if you point them out to Valloric on the Sigil forum they will be quickly fixed.
As said above, the issue is in FBReader for Android, not the PC version.

Quote:
That said, I would be willing to accept the patch if you can make it optional. That way when developing recipes it would use the current structure, but for normal downloads, it would not.
I tried to extract the code into a class so it would be possible to switch between different implementations, but it turned out to be rather hard, as two separate modules are affected.

I really think that the new name scheme is as usable as the current one, so please reconsider accepting the patch as is.

Ciao,
Steffen
siebert is offline   Reply With Quote
Old 11-28-2010, 11:58 AM   #14
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
open a a ticket for it, I dont have the time right now and I'll forget otherwise.
kovidgoyal is offline   Reply With Quote
Old 12-04-2010, 06:10 AM   #15
siebert
Developer
siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.siebert has a complete set of Star Wars action figures.
 
Posts: 155
Karma: 280
Join Date: Nov 2010
Device: Kindle 3 (Keyboard) 3G / iPad 9 WiFi / Google Pixel 6a (Android)
Quote:
Originally Posted by kovidgoyal View Post
open a a ticket for it, I dont have the time right now and I'll forget otherwise.
Done (http://bugs.calibre-ebook.com/ticket/7788)

I also updated my branch to launchpad and requested a merge (I hope I did everything right, I'm new to bazaar and launchpad, I'm more a git and cvs guy):

https://code.launchpad.net/~steffens.../calibre/trunk

The branch contains also the change to enable recipes downloading unmodified EPUBs (http://bugs.calibre-ebook.com/ticket/7789), it would be great if you would include that change, too.

Ciao,
Steffen

PS: Does bazaar support something like cherry-pick in git (merging single commits from another branch) so it is ok to make all my changes on a single branch or is it recommended to create a separate branch for each feature/fix I implement?
siebert is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Recipe works when mocked up as Python file, fails when converted to Recipe ode Recipes 7 09-04-2011 04:57 AM
Calibre Content Server d/l fails on Android Mogrith Android Devices 9 11-01-2010 06:04 PM
Calibre ePub TOC and Sigil salasnet Calibre 3 09-26-2010 05:34 PM
ePub-Sigil-Calibre-Sony PRS505 dicknskip Calibre 20 06-10-2010 03:58 PM
NY Times Recipe in Calibre 6.36 Fails keyrunner Calibre 1 01-28-2010 11:56 AM


All times are GMT -4. The time now is 01:38 AM.


MobileRead.com is a privately owned, operated and funded community.