[FileType Plugin] YVES Bible Plugin - Page 2

ichbindasauge · 09-23-2014, 10:45 AM

Thanks, it works fine now!

keba · 08-15-2016, 03:40 AM

Hi

I'm currently trying to use this plugin. It doesn't work in Calibre at all, so I decided to go to the console instead and run it in python. I noticed a few things:

The files do not have the ending ".yves" any more (fixed)
The "manifest.yves" is now called "_version_" (fixed)
It does not run on Python 3 (fixed)
UTF-8 does not work (main issue)

So I started translating the file yvesDecode.py to Python 3 and I fixed the filename issues, but I'm stuck with the conversion: Since I want to work with different languages, I wanted to use UTF-8 but the conversion of the .yves-files does not seem to work. For example, I get "La crÃ©ation" instead of "La création".

How can I get this to run properly?

So far, this is my code (I changed it to only work on Gen 1 for testing, as running the entire bible would take too long):

Code:

import sys, os
#import json # JSON module did not work
import rson

def readFile(paramFile):
	file = open(paramFile, 'rb')
	arrayOfByte = file.read()
	file.close()
	return(readString(arrayOfByte))

def readString(arrayOfByte):
	i = 0
	byteArray = []
	while (i < len(arrayOfByte)):
		if (len(arrayOfByte) > i + 1):
			j = ((int('0xFF',16) & arrayOfByte[(i + 1)]) >> 5 | (int('0xFF',16) & arrayOfByte[(i + 1)]) << 3)
			byteArray.append(j)
			byteArray.append(((int('0xFF',16) & arrayOfByte[i]) >> 5 | (int('0xFF',16) & arrayOfByte[i]) << 3))
		else:
			byteArray.append(((int('0xFF',16) & arrayOfByte[i]) >> 5 | (int('0xFF',16) & arrayOfByte[i]) << 3))
		i += 2;
	return(''.join([ chr(x & 0xFF) for x in byteArray]))

def loadJson(string):
    #return json.loads(string)
    return rson.loads(string)

def yvesDir2HTML(yvesfile, yves_temp_directory):
	file = open('out.txt', 'w', encoding="utf8") #for testing, only output part of the bible
	yvesfile = 'GEN.1' #for testing, only output part of the bible
	content = readFile(yvesfile)
	file.write(content)
	file.close()
	return("done")

if __name__ == '__main__':
    if len(sys.argv[1:]) > 0:
        for file in sys.argv[1:]:
            print(loadJson(readFile(file)))
    else:
        print(yvesDir2HTML("_version_","."))

Of course, I'll put everything back together when it works.

keba · 08-15-2016, 04:12 AM

Sometimes you find the solution right after asking

This works!

Code:

#!/usr/bin/env python

import sys, os
import json
import rson

def readFile(paramFile): ## for the manifest
	file = open(paramFile, 'rb')
	arrayOfByte = file.read()
	file.close()
	return(readString(arrayOfByte))

def readString(arrayOfByte): ## for the manifest
	return(arrayOfByte.decode("utf-8"))
	
def readFile2(paramFile):
	file = open(paramFile, 'rb')
	arrayOfByte = file.read()
	file.close()
	return(readString2(arrayOfByte))

def readString2(arrayOfByte):
	i = 0
	byteArray = []
	while (i < len(arrayOfByte)):
		if (len(arrayOfByte) > i + 1):
			j = ((int('0xFF',16) & arrayOfByte[(i + 1)]) >> 5 | (int('0xFF',16) & arrayOfByte[(i + 1)]) << 3)
			byteArray.append(j)
			byteArray.append(((int('0xFF',16) & arrayOfByte[i]) >> 5 | (int('0xFF',16) & arrayOfByte[i]) << 3))
		else:
			byteArray.append(((int('0xFF',16) & arrayOfByte[i]) >> 5 | (int('0xFF',16) & arrayOfByte[i]) << 3))
		i += 2;
	byteArray2 = bytearray((x & 0xFF) for x in byteArray)
	return(byteArray2.decode("utf8"))

def loadJson(string):
    return rson.loads(string)

def yvesDir2HTML(yvesfile, yves_temp_directory):
	# for iOS bibles, everything is stored in a plist, eventually use something like this:
	# xmltodict.parse(plistDecode.plistFromString(base64.decodestring(xmltodict.parse(plistDecode.plistFromFile('tv.lifechurch.bible.plist','xml1'))['plist']['dict']['dict'][0]['data'][0]),'xml1'))
	bibleMetaData = loadJson(readFile(yvesfile))
	
	yvesDir = os.path.dirname(yvesfile)
	
	bibleName = bibleMetaData['abbreviation'] + ".html"
	
	DEST = open(os.path.join(yves_temp_directory, bibleName), 'w', encoding="utf8")
	DEST.write( '<html dir="' + bibleMetaData['language']['text_direction'] + '"><head><title>' )
	DEST.write( bibleMetaData['local_title'] )
	DEST.write( '</title>\n')
	if( ('publisher' in bibleMetaData) and bibleMetaData['publisher'] and ('name' in bibleMetaData['publisher']) and bibleMetaData['publisher']['name'] ):
		DEST.write( '<meta name="Publisher" content="' + bibleMetaData['publisher']['name'] + '">\n')
	if( ('copyright_long' in bibleMetaData) and ('text' in bibleMetaData) and bibleMetaData['copyright_long']['text'] ):
		DEST.write( '<meta name="Copyright" content="' + bibleMetaData['copyright_long']['text'] + '">\n')
	if( ('language' in bibleMetaData) and ('iso_639_1' in bibleMetaData['language']) and bibleMetaData['language']['iso_639_1'] ):
		DEST.write( '<meta name="DC.language" content="' + bibleMetaData['language']['iso_639_1'] + '">\n')
	elif( ('language' in bibleMetaData) and ('iso_639_3' in bibleMetaData['language']) and bibleMetaData['language']['iso_639_3'] ):
		DEST.write( '<meta name="DC.language" content="' + bibleMetaData['language']['iso_639_3'] + '">\n')
	DEST.write( '<meta name="Source" content="YouVersion">\n')
	DEST.write( '<meta charset="utf-8" />\n')
	DEST.write( '<style type="text/css">' )
	DEST.write( '</style>' )
	DEST.write( '</head><body>\n' )
	#DEBUG
	file = open('out2.txt', 'w', encoding="utf8")
	for book in bibleMetaData['books']:
		DEST.write('<div class="book">\n<div class="bookTitle">')
		DEST.write(book['human_long'])
		DEST.write('</div>\n')
		for chapter in book['chapters']:
			##chapterFile = chapter['usfm'][len(book['usfm'])+1:]
			chapterFile = chapter['usfm']
			# DEST.write('<a href="')
			# DEST.write(book['usfm'])
			# DEST.write("/")
			# DEST.write(chapterFile)
			# DEST.write('.html">')
			# DEST.write(book['abbreviation'].encode('utf8'))
			# DEST.write(' ')
			# DEST.write(chapter['human'])
			# DEST.write('</a>')
			# DEST.write('\n<br />\n')
			print(yvesDir,chapterFile) # DEBUG
			bibleData = loadJson(readFile2(os.path.join(yvesDir,chapterFile)))
			if( ('content' in bibleData) and bibleData['content'] ):
				## COMMENT: Some bibles need JSON parsing first
				chapterLines = bibleData['content'].splitlines()
				DEST.writelines( chapterLines[2:len(chapterLines)-2] )
			else: ## COMMENT: Some bibles have direct HTML content
				DEST.write(readFile2(os.path.join(yvesDir,chapterFile)))
		DEST.write('</div>\n')
	DEST.write( '</body></html>\n' )
	DEST.close()
	file.close()
	return bibleName

if __name__ == '__main__':
    if len(sys.argv[1:]) > 0:
        for file in sys.argv[1:]:
            print(loadJson(readFile(file)))
    else:
        print(yvesDir2HTML("_version_","."))

ClashTheBunny · 08-15-2016, 10:19 PM

Glad this was helpful!

I'm happy to hear that it wasn't too tough to update.

One thing that is interesting is the desire to run in Python 3. As Calibre doesn't work with Python 3, do your updates support both 2 and 3? Or would your updates prevent the plug-in from working in Calibre?

ClashTheBunny · 08-15-2016, 10:24 PM

Another question. UTF-8 seemed to be working for me when I tested it last. Did you try UTF-8 with Python 2, or were you just having problems with how python changed handling strings between python 2 and 3?

keba · 08-16-2016, 06:37 AM

To answer your 2 questions:

The Calibre plugin didn't work, and since I have no clue how to create a calibre plugin, I decided to simply run the script from the command line. There, I hit the problem of not actually having Python 2 installed, so I was forced to translate your script. Also, I'm not very familiar with Python 2, so some of the encoding/decoding to and from byte arrays were a bit of a mystery to me.

As you can see in the script, I had to duplicate the 2 functions readFile() and readString() because the manifest (now called _version_) uses a different encoding (plain UTF-8 text) than the actual content of the bibles (the stuff actually in need of decoding).

UTF-8:
In readString2(), which reads the actual .yves-files, the problem was that the chr() function didn't provide the required results, so I had to create a byte-array before being able to decode it into UTF-8. I assume this may also be a Python 3 specific problem though:

Code:

byteArray2 = bytearray((x & 0xFF) for x in byteArray)
return(byteArray2.decode("utf8"))

And finally, it appears that some .yves-files contain html embedded in JSON, while others provide plain html. So I added a check for JSON, and if that is found it extracts the html, otherwise it uses the plain decoded html.

Does that answer all your questions?

keba · 08-16-2016, 06:39 AM

You could probably port that back to Python 2 if you just look at the changes (I can't), and make it into an updated Calibre addon, I guess. In my case, I was unable to import the files even after renaming _version_ to manifest.yves and the content files to *.yves...

keba · 08-16-2016, 06:48 AM

One more interesting note: The bibles can be downloaded directly, without dumping the content of the smartphone. It works like this:

Go to bible.com and open a bible of your choice. The link in your browser's toolbar will be something like https://www.bible.com/de/bible/157/deu.32 (This is Schlachter 2000 in German)
The number after */bible/ is the key, copy it
Paste it into the following link, between */offline/ and -1.zip, e.g. http://d1ibykdbqak8v3.cloudfront.net...line/157-1.zip
Download this zip file and unpack. The result is a folder containing 1 folder per book (e.g. Genesis, Deuteronomy, etc) with the *.yves files in them.

Using this, it should be possible to create a sophisticated Calibre addon that shows a list of bibles, and upon selection downloads imports and converts the chosen bible to epub.

keba · 08-16-2016, 06:53 AM

And finally, when porting to Python 3, JSON didn't work, so I used rson instead.

JPG · 07-21-2018, 10:11 AM

Hi
Is there a description of the yves format available?
I do not see manifest or _version_ file in downloaded zip files?

ClashTheBunny · 10-12-2019, 11:32 PM

I updated the plugin to work with bible_v3 from the latest versions of YouVersion for Android. You need to rename the _version_ file to something ending in ".yves" for it to import, but it seems to do a fair job.

winstonma · 07-31-2022, 03:40 AM

I am using Bible App on a e-reader (only wifi connection) and would like to know if it is possible to download undownloadable bible version.

For example https://bible.youversionapi.com/3.2/version.json?id=463 is the API call that for New American Bible, revised edition. The offline.allow_redownload flag in the API response should control if the bible can be downloaded.

In the API response there is offline.url for downloading the whole bible offline, but the Bible App wouldn't take it - I guess they need to do some process after download, which I am not sure where should I place the zip file to activate this process.

Does anyone have any idea how to get the offline file processed by the Bible Android App so I could read it offline.

Thanks in advance

JPG · 01-16-2023, 01:25 AM

It seems the files downloaded directly on pc does not contain a manifest file anymore?
any solution to this?

08-16-2016, 06:37 AM	#21
keba Junior Member Posts: 8 Karma: 10 Join Date: Aug 2012 Device: Kobo Aura H2O	To answer your 2 questions: The Calibre plugin didn't work, and since I have no clue how to create a calibre plugin, I decided to simply run the script from the command line. There, I hit the problem of not actually having Python 2 installed, so I was forced to translate your script. Also, I'm not very familiar with Python 2, so some of the encoding/decoding to and from byte arrays were a bit of a mystery to me. As you can see in the script, I had to duplicate the 2 functions readFile() and readString() because the manifest (now called _version_) uses a different encoding (plain UTF-8 text) than the actual content of the bibles (the stuff actually in need of decoding). UTF-8: In readString2(), which reads the actual .yves-files, the problem was that the chr() function didn't provide the required results, so I had to create a byte-array before being able to decode it into UTF-8. I assume this may also be a Python 3 specific problem though: Code: byteArray2 = bytearray((x & 0xFF) for x in byteArray) return(byteArray2.decode("utf8")) And finally, it appears that some .yves-files contain html embedded in JSON, while others provide plain html. So I added a check for JSON, and if that is found it extracts the html, otherwise it uses the plain decoded html. Does that answer all your questions?

08-16-2016, 06:48 AM	#23
keba Junior Member Posts: 8 Karma: 10 Join Date: Aug 2012 Device: Kobo Aura H2O	One more interesting note: The bibles can be downloaded directly, without dumping the content of the smartphone. It works like this: Go to bible.com and open a bible of your choice. The link in your browser's toolbar will be something like https://www.bible.com/de/bible/157/deu.32 (This is Schlachter 2000 in German) The number after /bible/ is the key, copy it Paste it into the following link, between /offline/ and -1.zip, e.g. http://d1ibykdbqak8v3.cloudfront.net...line/157-1.zip Download this zip file and unpack. The result is a folder containing 1 folder per book (e.g. Genesis, Deuteronomy, etc) with the *.yves files in them. Using this, it should be possible to create a sophisticated Calibre addon that shows a list of bibles, and upon selection downloads imports and converts the chosen bible to epub.

07-31-2022, 03:40 AM	#27
winstonma Zealot Posts: 108 Karma: 120918 Join Date: Nov 2013 Device: Onyx Boox Nova	I am using Bible App on a e-reader (only wifi connection) and would like to know if it is possible to download undownloadable bible version. For example https://bible.youversionapi.com/3.2/version.json?id=463 is the API call that for New American Bible, revised edition. The offline.allow_redownload flag in the API response should control if the bible can be downloaded. In the API response there is offline.url for downloading the whole bible offline, but the Bible App wouldn't take it - I guess they need to do some process after download, which I am not sure where should I place the zip file to activate this process. Does anyone have any idea how to get the offline file processed by the Bible Android App so I could read it offline. Thanks in advance Last edited by winstonma; 07-31-2022 at 10:05 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] KindleUnpack - The Plugin	DiapDealer	Plugins	492	10-25-2022 08:13 AM
Plugin not customizable: Plugin: HTML Output does not need customization	flyingfoxlee	Conversion	2	02-24-2012 02:24 AM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM
Zip Filetype Plugin?	rsingley	Plugins	7	02-11-2011 05:11 PM
New Plugin Type Idea: Library Plugin	cgranade	Plugins	3	09-15-2010 12:11 PM

09-23-2014, 10:45 AM	#16
ichbindasauge Connoisseur Posts: 89 Karma: 203522 Join Date: Oct 2009 Device: Kindle 2 US, iPad, Samsung Win 8 Tablet	Thanks, it works fine now!

08-15-2016, 10:19 PM	#19
ClashTheBunny Member Posts: 12 Karma: 10 Join Date: Sep 2014 Device: Kindle Paperwhite 2	Glad this was helpful! I'm happy to hear that it wasn't too tough to update. One thing that is interesting is the desire to run in Python 3. As Calibre doesn't work with Python 3, do your updates support both 2 and 3? Or would your updates prevent the plug-in from working in Calibre?

08-15-2016, 10:24 PM	#20
ClashTheBunny Member Posts: 12 Karma: 10 Join Date: Sep 2014 Device: Kindle Paperwhite 2	Another question. UTF-8 seemed to be working for me when I tested it last. Did you try UTF-8 with Python 2, or were you just having problems with how python changed handling strings between python 2 and 3?

08-16-2016, 06:39 AM	#22
keba Junior Member Posts: 8 Karma: 10 Join Date: Aug 2012 Device: Kobo Aura H2O	You could probably port that back to Python 2 if you just look at the changes (I can't), and make it into an updated Calibre addon, I guess. In my case, I was unable to import the files even after renaming _version_ to manifest.yves and the content files to *.yves...

08-16-2016, 06:53 AM	#24
keba Junior Member Posts: 8 Karma: 10 Join Date: Aug 2012 Device: Kobo Aura H2O	And finally, when porting to Python 3, JSON didn't work, so I used rson instead.

07-21-2018, 10:11 AM	#25
JPG Junior Member Posts: 9 Karma: 10 Join Date: Feb 2014 Location: South Africa Device: Kobo Touch	Hi Is there a description of the yves format available? I do not see manifest or _version_ file in downloaded zip files?

10-12-2019, 11:32 PM	#26
ClashTheBunny Member Posts: 12 Karma: 10 Join Date: Sep 2014 Device: Kindle Paperwhite 2	I updated the plugin to work with bible_v3 from the latest versions of YouVersion for Android. You need to rename the _version_ file to something ending in ".yves" for it to import, but it seems to do a fair job.

01-16-2023, 01:25 AM	#28
JPG Junior Member Posts: 9 Karma: 10 Join Date: Feb 2014 Location: South Africa Device: Kobo Touch	It seems the files downloaded directly on pc does not contain a manifest file anymore? any solution to this?