Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 09-23-2014, 11:45 AM   #16
ichbindasauge
Connoisseur
ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.ichbindasauge ought to be getting tired of karma fortunes by now.
 
Posts: 88
Karma: 203522
Join Date: Oct 2009
Device: Kindle 2 US, iPad, Samsung Win 8 Tablet
Thanks, it works fine now!
ichbindasauge is offline   Reply With Quote
Old 08-15-2016, 04:40 AM   #17
keba
Junior Member
keba began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Aug 2012
Device: Kobo Aura H2O
Hi

I'm currently trying to use this plugin. It doesn't work in Calibre at all, so I decided to go to the console instead and run it in python. I noticed a few things:
  • The files do not have the ending ".yves" any more (fixed)
  • The "manifest.yves" is now called "_version_" (fixed)
  • It does not run on Python 3 (fixed)
  • UTF-8 does not work (main issue)

So I started translating the file yvesDecode.py to Python 3 and I fixed the filename issues, but I'm stuck with the conversion: Since I want to work with different languages, I wanted to use UTF-8 but the conversion of the .yves-files does not seem to work. For example, I get "La création" instead of "La création".

How can I get this to run properly?

So far, this is my code (I changed it to only work on Gen 1 for testing, as running the entire bible would take too long):
Code:
import sys, os
#import json # JSON module did not work
import rson

def readFile(paramFile):
	file = open(paramFile, 'rb')
	arrayOfByte = file.read()
	file.close()
	return(readString(arrayOfByte))

def readString(arrayOfByte):
	i = 0
	byteArray = []
	while (i < len(arrayOfByte)):
		if (len(arrayOfByte) > i + 1):
			j = ((int('0xFF',16) & arrayOfByte[(i + 1)]) >> 5 | (int('0xFF',16) & arrayOfByte[(i + 1)]) << 3)
			byteArray.append(j)
			byteArray.append(((int('0xFF',16) & arrayOfByte[i]) >> 5 | (int('0xFF',16) & arrayOfByte[i]) << 3))
		else:
			byteArray.append(((int('0xFF',16) & arrayOfByte[i]) >> 5 | (int('0xFF',16) & arrayOfByte[i]) << 3))
		i += 2;
	return(''.join([ chr(x & 0xFF) for x in byteArray]))

def loadJson(string):
    #return json.loads(string)
    return rson.loads(string)

def yvesDir2HTML(yvesfile, yves_temp_directory):
	file = open('out.txt', 'w', encoding="utf8") #for testing, only output part of the bible
	yvesfile = 'GEN.1' #for testing, only output part of the bible
	content = readFile(yvesfile)
	file.write(content)
	file.close()
	return("done")

if __name__ == '__main__':
    if len(sys.argv[1:]) > 0:
        for file in sys.argv[1:]:
            print(loadJson(readFile(file)))
    else:
        print(yvesDir2HTML("_version_","."))
Of course, I'll put everything back together when it works.
keba is offline   Reply With Quote
Old 08-15-2016, 05:12 AM   #18
keba
Junior Member
keba began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Aug 2012
Device: Kobo Aura H2O
Sometimes you find the solution right after asking

This works!
Code:
#!/usr/bin/env python

import sys, os
import json
import rson

def readFile(paramFile): ## for the manifest
	file = open(paramFile, 'rb')
	arrayOfByte = file.read()
	file.close()
	return(readString(arrayOfByte))

def readString(arrayOfByte): ## for the manifest
	return(arrayOfByte.decode("utf-8"))
	
def readFile2(paramFile):
	file = open(paramFile, 'rb')
	arrayOfByte = file.read()
	file.close()
	return(readString2(arrayOfByte))

def readString2(arrayOfByte):
	i = 0
	byteArray = []
	while (i < len(arrayOfByte)):
		if (len(arrayOfByte) > i + 1):
			j = ((int('0xFF',16) & arrayOfByte[(i + 1)]) >> 5 | (int('0xFF',16) & arrayOfByte[(i + 1)]) << 3)
			byteArray.append(j)
			byteArray.append(((int('0xFF',16) & arrayOfByte[i]) >> 5 | (int('0xFF',16) & arrayOfByte[i]) << 3))
		else:
			byteArray.append(((int('0xFF',16) & arrayOfByte[i]) >> 5 | (int('0xFF',16) & arrayOfByte[i]) << 3))
		i += 2;
	byteArray2 = bytearray((x & 0xFF) for x in byteArray)
	return(byteArray2.decode("utf8"))

def loadJson(string):
    return rson.loads(string)

def yvesDir2HTML(yvesfile, yves_temp_directory):
	# for iOS bibles, everything is stored in a plist, eventually use something like this:
	# xmltodict.parse(plistDecode.plistFromString(base64.decodestring(xmltodict.parse(plistDecode.plistFromFile('tv.lifechurch.bible.plist','xml1'))['plist']['dict']['dict'][0]['data'][0]),'xml1'))
	bibleMetaData = loadJson(readFile(yvesfile))
	
	yvesDir = os.path.dirname(yvesfile)
	
	bibleName = bibleMetaData['abbreviation'] + ".html"
	
	DEST = open(os.path.join(yves_temp_directory, bibleName), 'w', encoding="utf8")
	DEST.write( '<html dir="' + bibleMetaData['language']['text_direction'] + '"><head><title>' )
	DEST.write( bibleMetaData['local_title'] )
	DEST.write( '</title>\n')
	if( ('publisher' in bibleMetaData) and bibleMetaData['publisher'] and ('name' in bibleMetaData['publisher']) and bibleMetaData['publisher']['name'] ):
		DEST.write( '<meta name="Publisher" content="' + bibleMetaData['publisher']['name'] + '">\n')
	if( ('copyright_long' in bibleMetaData) and ('text' in bibleMetaData) and bibleMetaData['copyright_long']['text'] ):
		DEST.write( '<meta name="Copyright" content="' + bibleMetaData['copyright_long']['text'] + '">\n')
	if( ('language' in bibleMetaData) and ('iso_639_1' in bibleMetaData['language']) and bibleMetaData['language']['iso_639_1'] ):
		DEST.write( '<meta name="DC.language" content="' + bibleMetaData['language']['iso_639_1'] + '">\n')
	elif( ('language' in bibleMetaData) and ('iso_639_3' in bibleMetaData['language']) and bibleMetaData['language']['iso_639_3'] ):
		DEST.write( '<meta name="DC.language" content="' + bibleMetaData['language']['iso_639_3'] + '">\n')
	DEST.write( '<meta name="Source" content="YouVersion">\n')
	DEST.write( '<meta charset="utf-8" />\n')
	DEST.write( '<style type="text/css">' )
	DEST.write( '</style>' )
	DEST.write( '</head><body>\n' )
	#DEBUG
	file = open('out2.txt', 'w', encoding="utf8")
	for book in bibleMetaData['books']:
		DEST.write('<div class="book">\n<div class="bookTitle">')
		DEST.write(book['human_long'])
		DEST.write('</div>\n')
		for chapter in book['chapters']:
			##chapterFile = chapter['usfm'][len(book['usfm'])+1:]
			chapterFile = chapter['usfm']
			# DEST.write('<a href="')
			# DEST.write(book['usfm'])
			# DEST.write("/")
			# DEST.write(chapterFile)
			# DEST.write('.html">')
			# DEST.write(book['abbreviation'].encode('utf8'))
			# DEST.write(' ')
			# DEST.write(chapter['human'])
			# DEST.write('</a>')
			# DEST.write('\n<br />\n')
			print(yvesDir,chapterFile) # DEBUG
			bibleData = loadJson(readFile2(os.path.join(yvesDir,chapterFile)))
			if( ('content' in bibleData) and bibleData['content'] ):
				## COMMENT: Some bibles need JSON parsing first
				chapterLines = bibleData['content'].splitlines()
				DEST.writelines( chapterLines[2:len(chapterLines)-2] )
			else: ## COMMENT: Some bibles have direct HTML content
				DEST.write(readFile2(os.path.join(yvesDir,chapterFile)))
		DEST.write('</div>\n')
	DEST.write( '</body></html>\n' )
	DEST.close()
	file.close()
	return bibleName

if __name__ == '__main__':
    if len(sys.argv[1:]) > 0:
        for file in sys.argv[1:]:
            print(loadJson(readFile(file)))
    else:
        print(yvesDir2HTML("_version_","."))
keba is offline   Reply With Quote
Old 08-15-2016, 11:19 PM   #19
ClashTheBunny
Member
ClashTheBunny began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Sep 2014
Device: Kindle Paperwhite 2
Glad this was helpful!

I'm happy to hear that it wasn't too tough to update.

One thing that is interesting is the desire to run in Python 3. As Calibre doesn't work with Python 3, do your updates support both 2 and 3? Or would your updates prevent the plug-in from working in Calibre?
ClashTheBunny is offline   Reply With Quote
Old 08-15-2016, 11:24 PM   #20
ClashTheBunny
Member
ClashTheBunny began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Sep 2014
Device: Kindle Paperwhite 2
Another question. UTF-8 seemed to be working for me when I tested it last. Did you try UTF-8 with Python 2, or were you just having problems with how python changed handling strings between python 2 and 3?
ClashTheBunny is offline   Reply With Quote
Old 08-16-2016, 07:37 AM   #21
keba
Junior Member
keba began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Aug 2012
Device: Kobo Aura H2O
To answer your 2 questions:

The Calibre plugin didn't work, and since I have no clue how to create a calibre plugin, I decided to simply run the script from the command line. There, I hit the problem of not actually having Python 2 installed, so I was forced to translate your script. Also, I'm not very familiar with Python 2, so some of the encoding/decoding to and from byte arrays were a bit of a mystery to me.

As you can see in the script, I had to duplicate the 2 functions readFile() and readString() because the manifest (now called _version_) uses a different encoding (plain UTF-8 text) than the actual content of the bibles (the stuff actually in need of decoding).

UTF-8:
In readString2(), which reads the actual .yves-files, the problem was that the chr() function didn't provide the required results, so I had to create a byte-array before being able to decode it into UTF-8. I assume this may also be a Python 3 specific problem though:

Code:
byteArray2 = bytearray((x & 0xFF) for x in byteArray)
return(byteArray2.decode("utf8"))
And finally, it appears that some .yves-files contain html embedded in JSON, while others provide plain html. So I added a check for JSON, and if that is found it extracts the html, otherwise it uses the plain decoded html.

Does that answer all your questions?
keba is offline   Reply With Quote
Old 08-16-2016, 07:39 AM   #22
keba
Junior Member
keba began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Aug 2012
Device: Kobo Aura H2O
You could probably port that back to Python 2 if you just look at the changes (I can't), and make it into an updated Calibre addon, I guess. In my case, I was unable to import the files even after renaming _version_ to manifest.yves and the content files to *.yves...
keba is offline   Reply With Quote
Old 08-16-2016, 07:48 AM   #23
keba
Junior Member
keba began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Aug 2012
Device: Kobo Aura H2O
One more interesting note: The bibles can be downloaded directly, without dumping the content of the smartphone. It works like this:
  • Go to bible.com and open a bible of your choice. The link in your browser's toolbar will be something like https://www.bible.com/de/bible/157/deu.32 (This is Schlachter 2000 in German)
  • The number after */bible/ is the key, copy it
  • Paste it into the following link, between */offline/ and -1.zip, e.g. http://d1ibykdbqak8v3.cloudfront.net...line/157-1.zip
  • Download this zip file and unpack. The result is a folder containing 1 folder per book (e.g. Genesis, Deuteronomy, etc) with the *.yves files in them.
Using this, it should be possible to create a sophisticated Calibre addon that shows a list of bibles, and upon selection downloads imports and converts the chosen bible to epub.
keba is offline   Reply With Quote
Old 08-16-2016, 07:53 AM   #24
keba
Junior Member
keba began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Aug 2012
Device: Kobo Aura H2O
And finally, when porting to Python 3, JSON didn't work, so I used rson instead.
keba is offline   Reply With Quote
Old 07-21-2018, 11:11 AM   #25
JPG
Junior Member
JPG began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Feb 2014
Location: South Africa
Device: Kobo Touch
Hi
Is there a description of the yves format available?
I do not see manifest or _version_ file in downloaded zip files?
JPG is offline   Reply With Quote
Old 10-13-2019, 12:32 AM   #26
ClashTheBunny
Member
ClashTheBunny began at the beginning.
 
Posts: 12
Karma: 10
Join Date: Sep 2014
Device: Kindle Paperwhite 2
I updated the plugin to work with bible_v3 from the latest versions of YouVersion for Android. You need to rename the _version_ file to something ending in ".yves" for it to import, but it seems to do a fair job.
ClashTheBunny is offline   Reply With Quote
Reply

Tags
bible, calibre, input format

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] KindleUnpack - The Plugin DiapDealer Plugins 412 04-24-2018 07:07 PM
Plugin not customizable: Plugin: HTML Output does not need customization flyingfoxlee Conversion 2 02-24-2012 03:24 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 01:27 PM
Zip Filetype Plugin? rsingley Plugins 7 02-11-2011 06:11 PM
New Plugin Type Idea: Library Plugin cgranade Plugins 3 09-15-2010 01:11 PM


All times are GMT -4. The time now is 07:58 AM.


MobileRead.com is a privately owned, operated and funded community.